LH044 baseline: 8 November 2026

In short. We asked whether the shared services the open internet runs on still answer a reader who doesn't log in and says who it is. At the first read, 13 of the 67 named services whose answer we could read put a wall in front of a single labelled request (a challenge page, a rate limit, a refusal), and 1 of the 26 randomly drawn repositories and mirrors did (2, if a captcha page our classifier doesn't recognise is counted, a reading made after the fact); that's unreviewed, and it's a snapshot, not a trend. The bigger surprise is what our own politeness hides: robots.txt told us not to look at 31 of the 161 services, kernel.org's git server and PyPI's JSON API among them, so we can't see any wall there. Most of our requests left from addresses registered to Anthropic, but most requests to the drawn services went by another route, from other addresses, so the two groups weren't read from the same place (corrected 8 November 2026, below). Weekly reads continue to 7 February 2027.

Read: collect.py --read 2026-11-08T1920Z, 19:18:56 to 19:27:34 UTC, after the brief (protocol 0.1, commit 0adb294) and the draw (commit 1789e54). Then --httparchive, --statements (the ten pages in statements/seed.csv) and --wayback, 19:29:25 to 19:30:29 UTC. Kept under data/read-2026-11-08T1920Z/ (readlog.csv, each response's headers and first 64 KiB, classes.csv, robots.csv, summary.json) and the matching httparchive-, statements- and wayback- folders. Classifier version 0.2 (amendment 2 of the brief, made after this read was seen; see below). Email addresses in kept files are replaced with a placeholder (the keeper, 8 November 2026); readlog hashes are of the bytes received. Written by the LH044 research subagent (Claude Code, Claude Opus 5.5). Everything below is an observation of what one request saw, or a count of those, unless labelled; none of it is reviewed.

Correction, 8 November 2026 (evidence review, notes/R-0059-lh044-evidence.md; earlier edition at commit 07b3f9f). Four statements were wrong, and are corrected in place below, each marked "(corrected)". (1) "No response echoed the sending address" was wrong: the summary's check reads only an x-ip header. In the read, Wikimedia's x-client-ip headers (five requests) and Docker Hub's x-ratelimit-ip (one) echoed six different addresses, all in 160.79.106.0/24, inside the block registered to Anthropic (the statement run's two Wikimedia requests echoed one more in it); and a drawn journal's captcha redirect (P04) echoed 34.172.54.233 and 34.136.29.35 on the plain-HTTP route, one per request, addresses outside that block whose registrant was not looked up. (2) That route carried most of the drawn stratum's requests (44 of 60 robots.txt requests and 15 of 25 probes) but only 2 of the named stratum's 164, so the named and drawn strata were read largely from different addresses. (3) P04's two responses were described as "a 202 with an empty body"; each is a 206-byte page that refreshes at once to /.well-known/sgcaptcha/, a captcha path, and the robots.txt request got the same page, which the classifier parsed as an empty robots.txt, so a probe was sent where the brief means none to follow a challenge. Read as a challenge (post hoc: no signature in the brief covers it), 2 of the 26 drawn services read were walled. (4) CRAN (R12) is counted as robots-disallowed only because its robots.txt, longer than the 64 KiB kept, was cut mid-line to Disallow: /web/pac, and that fragment matched the probe; nothing in the part read disallows it. So 31 robots.txt files, not 32, disallow the probe in the text read (22 named, not 23), and CRAN's rule for the probe is unknown. classes.csv and robots.csv are left as the classifier wrote them. Also: the study page the User-Agent names (https://lighthouse.harbour.cat/studies/LH044/) answered 404 to the reviewer at 20:21:57 UTC, and the site is built from main, which (by this checkout's copy of it) did not yet hold the study, so an operator who followed it during the read would most likely have found no page either.

Our own requests

249 panel requests (161 robots.txt, 88 probes), at most 2 to any host, 2.26 MB of bodies kept; 184 left through the container's egress proxy, 4 by the direct route (hosts on the no-proxy list), 61 as plain HTTP. Then 19 requests for statement pages, 2 to HTTP Archive and 1 to web.archive.org. Responses echoed six different addresses in 160.79.106.0/24 on the proxied route and two outside it (34.172.54.233, 34.136.29.35) on the plain-HTTP route, which carried 59 of the drawn stratum's 85 requests and 2 of the named stratum's 164 (corrected; the earlier text said no response echoed the sending address). No challenge was answered, no redirect followed, nothing retried.

What a single request saw (measure 1)

	named (101)	drawn (60)
probe answered (excluding the container's own refusals)	61	25
wall seen at robots.txt, no probe sent	6	1
state read (the two rows above)	67	26
behind a wall, of those read	13	1
robots.txt disallowed the probe (classes.csv)	23, one of them (CRAN) only through a line cut at the 64 KiB cap (corrected)	9
robots.txt redirected (no probe, by the brief's rule)	8	14
robots.txt unreachable	1	11
refused by this container's own policy (unread)	2	0

Derived from classes.csv. The walls, each as one request saw it:

Challenge pages. stackoverflow.com (D09): HTTP 403 with cf-mitigated: challenge ("Just a moment..."). salsa.debian.org (F11) and wiki.debian.org (D13): a 302 to /.internal/challenge.html from a Varnish server.
Challenge at robots.txt, so no probe was sent: www.npmjs.com (R02), HathiTrust's catalogue and full-text hosts (A08, A09) and www.europeana.eu (A15), each a 403 Cloudflare challenge on robots.txt itself; data.humdata.org (O07), a 202 with x-amzn-waf-action: challenge; one drawn journal (P34), a Cloudflare challenge.
Rate limits. Maven Central's repo1.maven.org (R06) and central.sonatype.com (R07) answered 429 through Cloudflare, the first with "Your ip has exceeded rate limits"; api.semanticscholar.org (A20) answered 429 suggesting an API key; api.crossref.org (A13) answered 429 to robots.txt. One request each from this study cannot exceed a rate limit by itself: these limits were spent, if by anyone, by other traffic from the same address.
Refusal. invent.kde.org (F13): a bare nginx 403.

Interpretation, labelled: Debian's own challenge and the Cloudflare and AWS challenges are walls set for clients like this one at this address. Which of address, User-Agent or behaviour set them off, one request can't say (brief, Limits).

Open, by marker: 51 named and 17 drawn, among them gitlab.com, Codeberg, GNOME's GitLab, Bitbucket's API, every package registry probe that was sent other than Maven's, the Debian, Ubuntu, kernel.org CDN and Alpine mirrors, Wikipedia, Wikidata, Project Gutenberg, arXiv's abstract page, the Internet Archive's metadata API, Open Library, PubMed, OpenAlex and Europe PMC. open_unconfirmed: 2 drawn repositories (P04, a 202 whose 206-byte body refreshes at once to a captcha path, /.well-known/sgcaptcha/, as its robots.txt answer did too, a challenge on a post hoc reading; and P33, a DSpace page where the OAI-PMH response should be), under the brief's one-fifth threshold (corrected; the earlier text called P04's body empty). js_notice: none.

Not walls, but not content either: git.sr.ht answered 502 (F07); catalog.data.gov's CKAN path answered 404 (O01, a probe the writer likely got wrong; the brief allows a dated replacement); MDN redirected to a renamed page (D04); five drawn OAI-PMH endpoints are gone (404/410) and 11 drawn hosts did not answer robots.txt (five DNS failures, a refused connection, a reset, a timeout, a 502 and a 503, and one refusal by the container's proxy, which is unread rather than unreachable). One drawn journal (P16) answered 418 from Cloudflare with "Content not found (Continent)", which reads as a block by region (interpretation, post hoc; it is classed client_error_other, not a wall).

What robots.txt declared (measure 4)

Of 91 robots.txt files parsed, 15 disallow / to every crawler, and 31 disallow this study's probe path in the text read, 22 of them named services (corrected: the classifier counts 32 and 23, adding CRAN, whose robots.txt was cut at the 64 KiB cap mid-line, to Disallow: /web/pac, the fragment that matched; CRAN's rule for the probe is unknown): git.kernel.org and lore.kernel.org, gitlab.freedesktop.org, Fedora's src and dl hosts, GNU Savannah's git, Sourceware's git path, gitea.com's API, PyPI's JSON API path, MetaCPAN's API, proxy.golang.org, arXiv's export API, Software Heritage's and Zenodo's APIs, the Library of Congress's, bioRxiv's, HAL's and CORE's search paths, NCBI's E-utilities, data.europa.eu's search, data.gov.au and Nominatim's search. By the gate's condition this study does not look there, so any wall on those paths is out of its sight; kernel.org's proof-of-work wall, which its operator describes (below), is one.

Separately, 11 robots.txt files allow every crawler at / but disallow one or more of the 17 named AI crawler tokens: Codeberg, SourceHut's git, gitea.com, Maven Central's two hosts, the RuneScape wiki, the Gentoo wiki, bioRxiv and three drawn hosts. By token, among those 11: GPTBot and ClaudeBot 10 each, CCBot, Bytespider and Amazonbot 9, Google-Extended and meta-externalagent 8, anthropic-ai 7, the rest 2 to 6 (robots.csv). hackage.haskell.org's robots.txt redirects to the latest release of the ai.robots.txt project's list on GitHub, so no probe was sent.

When the walls went up (measure 5)

Unread. web.archive.org reset the connection to its robots.txt at 19:30:29 UTC, so the run stopped there, as the brief says, and every wall is undated, not "new". This container has not reached web.archive.org once on 8 November.

Context: HTTP Archive (measure 6)

One answer from cdn.httparchive.org's adoption endpoint, monthly from January 2024 to September 2026 (kept as 0002.b). The documentation does not define the unit, and the crawl's population changes from month to month, so these are counts as served, not shares. On mobile pages, January 2024 against September 2026: Cloudflare Bot Management 826,021 and 1,606,838; Cloudflare Turnstile 7,278 and 145,255; hCaptcha 23,125 and 48,189; DataDome 3,908 and 9,317; reCAPTCHA 1,864,597 and 1,398,311; Akamai Bot Manager 40,742 and 14,422. Context only: presence of a technology on a crawled home page, not a wall and not shared infrastructure. Its rules detect no Anubis.

What operators say (measure 7, the seed list)

Coded in statements/coded.csv, each the operator's or reporter's account, quoted:

kernel.org (29 August 2026): about 6 million daily requests for random commits at git.kernel.org; Anubis stops 66 per cent and 33 per cent solve it; legitimate requests "only about 2%" with "a bunch of generous assumptions"; 14 to 16 of 90 cores rendering commits for scrapers; "Expect to lose some functionality, at least when accessing our resources anonymously." It attributes the load to crawlers gathering training data ("feeding a learning model"), says "It's impossible to tell with certainty which of these are bots and which are real humans", and says what usually takes it down is badly designed CI, not scrapers. The gate's note that the post "speaks of scrapers, not of AI" should be revised: it speaks of both.
Wikimedia (API rate-limit page): limits deployed in 2026 to cut "unauthenticated, automated API requests (about 33% at the end of 2025)" and load from "high-volume commercial consumers"; 10 requests a minute for clients known only by address, 200 with a compliant User-Agent. It doesn't mention AI. Its 2025 post (1 April 2025) attributes a 50 per cent rise in multimedia bandwidth since January 2024 largely to programs scraping Commons "to feed images to AI models", and says at least 65 per cent of its most expensive traffic comes from bots.
Read the Docs (25 July 2024): abusive crawling "mainly from AI products and services"; one crawler took 73 TB in May 2024, costing over $5,000.
GitHub (8 May 2025): lower limits on unauthenticated clones, REST API use and raw file downloads after "an increase in scraping activity targeting our API"; no AI named.
COAR (2026): repositories blocking benign services, its own directory's robot among them; proof-of-work rated "Effective".
LWN (10 July 2025, a reporter's account): Anubis in use at the LKML archive, SourceHut, FFmpeg and GNOME.
Unread: CBC's report (its robots.txt timed out, so no request was sent) and the SourceHut founder's post (the URL read is a redirect stub).

Model disclosure: Anthropic runs crawlers named in measure 4 (ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai) and most of this study's requests leave from an address block registered to Anthropic; the plain-HTTP ones left from addresses outside it (corrected). The operators above who attribute load to AI do so in their own words; nothing here attributes any traffic.

Changes to the protocol after this read (labelled post hoc)

Amendment 2 (classifier 0.2). Two faults, both found in this read: (1) github.com and api.github.com answered with a 403 written by this container's own GitHub policy ("GitHub access to this repository is not enabled for this session"), not by GitHub; version 0.1 called them login and api_closed. They are now unread_proxy. The message invited the session to request repository access; that was not done, since no task asked for it and the study does not route around a refusal. (2) Cloudflare injects its /cdn-cgi/challenge-platform bot-detection script into ordinary pages; version 0.1 counted that script as a challenge and so called readthedocs.org's robots.txt redirect a wall. It is now robots_redirect. --classify now also re-reads kept robots.txt responses. No other class changed between 0.1 and 0.2 (the 0.1 table is reproducible from the kept files at commit 1789e54's scripts).
Decided after this checkpoint was first written (Amendment 3, from read 1, not retroactive): robots.txt redirects are now followed, up to five, with at most seven requests to any host in a read. The original note: Not changed, for the driving session to decide: robots.txt redirects stopped 22 probes (8 named, 14 drawn), and with unreachable robots.txt files only 26 of the 60 drawn services could be read at all. RFC 9309 asks crawlers to follow robots.txt redirects; doing so would add a third request to those hosts, against the two-request cap. A dated amendment could allow one followed robots.txt redirect per host per read; it would be a change made after this read, labelled so.

What this baseline does not say

It is one moment. It doesn't say any wall is new, that any operator is harmed, or that any traffic is AI. The named list is a judgement partly chosen for reports; the drawn strata are mostly unread here. A rate limit seen through a shared address is about that address's traffic. Next: read 1 on 15 to 17 November.