Lighthouse

Lighthouse watches the internet for what AI systems do there and what their activity leaves behind, and publishes what it finds, with the evidence and its limits.

Studies LH044

LH044: for the driving session

Written 8 November 2026 by the LH044 subagent. Nothing here has been put into a shared file.

Amendment 5, the keeper's robots.txt rule (added 8 November 2026, about 21:00 UTC)

Done in this directory and in the shared scripts/robots_rules.py and scripts/robots_tokens.json (new), with notes/robots-tokens-2026-11-08.md. Brief, Amendments: Amendment 5 replaces the open question. From read 1 the collector obeys groups for Lighthouse-research or the running provider's tokens, else *, with the strictest winning. The provider comes from LIGHTHOUSE_PROVIDER (anthropic or openai), and unset means all providers. Under this rule, eight baseline probes would not have been sent (D15, D11, F06, F07, R06, R07 named; P18, Q07 drawn), whatever the provider setting, and no others. scripts/test_lh044.py checks this offline and passes (0 failed), as does python3 -I scripts/robots_rules.py.

For shared files:

  • schedule/due.csv: in the task text of rows LH044-read-01 to LH044-read-13, after "python3 -I studies/LH044/scripts/collect.py --read <UTC label>", add "with LIGHTHOUSE_PROVIDER set to the run's provider (anthropic under Claude Code, openai under Codex; brief, Amendment 5)". Add a row for the sweep (the driving session may reword or re-window it):
robots-sweep-1,method,"Bring every collector with open schedule rows under the keeper's robots.txt rule (AGENTS.md, Rules that matter): call scripts/robots_rules.py with LIGHTHOUSE_PROVIDER before each request, by a dated amendment in each brief, from the list in notes/robots-tokens-2026-11-08.md; first LH036 and LH038, whose kernel.org URLs the retained robots.txt files disallow, then LH030 to LH034, LH037, LH039 and LH040 before their reads of 9 November",2026-11-08,2026-11-09,open,2026-11-08,,,notes/robots-tokens-2026-11-08.md
  • schedule/runner.md (a suggestion): a scheduled run exports LIGHTHOUSE_PROVIDER for its provider before any collector runs. Unset is safe, because it obeys every provider's tokens, but it is stricter than the rule needs.
  • AGENTS.md, Rules that matter (a suggestion, one clause): after "The tokens are kept in scripts/robots_tokens.json", add "and applied by scripts/robots_rules.py; the provider is named in LIGHTHOUSE_PROVIDER".
  • Investigation page, Where the evidence stands, under the baseline line: "From read 1 (15 November) the closing web also obeys robots.txt groups addressed to the agents of the provider running Lighthouse (the keeper, 8 November 2026). Eight services the baseline probed are kept out that way, six of them named, and two of those walled at the baseline (Maven Central's two hosts) (brief, Amendment 5)."
  • programme.md, Next, under the LH044 entry: "Amendment 5 (the keeper's decision, 8 November 2026): from read 1 the collector obeys groups for Lighthouse's own token or the running provider's agent tokens, else *, the strictest winning, through the shared scripts/robots_rules.py. Eight baseline probes would not have been sent under it. A sweep of the other collectors is listed in notes/robots-tokens-2026-11-08.md: most check no robots.txt, and LH036 and LH038 fetch kernel.org pages that kernel.org's robots.txt, as read on 8 November, disallows to every crawler."
  • For the keeper or the driving session to confirm: (1) names that robots.txt files use for a provider's agents but no provider page lists (Claude-Code, anthropic-ai, Claude-Web, ChatGPT, OpenAI) are applied, the stricter choice; (2) as written, * applies only when no applicable token is named, so a file that shuts * out but lets ClaudeBot in by name would let Lighthouse in as anthropic. No baseline file does this. Both choices are recorded in notes/robots-tokens-2026-11-08.md and in robots_rules.py's self-test.

schedule/due.csv

Replace the P-0013-brief row with (status done):

P-0013-brief,P-0013,"Write and fix a brief for proposals.md P-0013 (the closing web), the first study of investigations/ai-cost-and-spread.md, under the gate's conditions there: the panel drawn by a rule before any status is read, Lighthouse's own requests labelled, identified, within robots.txt and with an end date; then a baseline read",2026-11-09,,done,2026-11-08,"driving session 2026-11-08, LH044 subagent (Claude Code, Anthropic cloud)",studies/LH044/brief.md (protocol 0.1); studies/LH044/checkpoints/2026-11-08.md,"brief fixed and committed before any panel status was read; panel drawn by its rule after that commit; baseline read 2026-11-08T1920Z; web.archive.org unreachable from the container, so no wall is dated yet"

Add (status open, by empty):

LH044-read-01,LH044,"Weekly read 1 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-11-15.md",2026-11-15,2026-11-17,open,2026-11-08,,,
LH044-read-02,LH044,"Weekly read 2 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-11-22.md",2026-11-22,2026-11-24,open,2026-11-08,,,
LH044-read-03,LH044,"Weekly read 3 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-11-29.md",2026-11-29,2026-12-01,open,2026-11-08,,,
LH044-read-04,LH044,"Weekly read 4 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-12-06.md",2026-12-06,2026-12-08,open,2026-11-08,,,
LH044-checkpoint-w04,LH044,"Interim checkpoint after read 4 (brief, Reads): measures 2 to 4 and 7 to date, HTTP Archive context read (collect.py --httparchive <label>), statement searches for every service walled at read 4; interim record version 0.1",2026-12-06,2026-12-10,open,2026-11-08,,,
LH044-read-05,LH044,"Weekly read 5 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-12-13.md",2026-12-13,2026-12-15,open,2026-11-08,,,
LH044-read-06,LH044,"Weekly read 6 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-12-20.md",2026-12-20,2026-12-22,open,2026-11-08,,,
LH044-read-07,LH044,"Weekly read 7 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2026-12-27.md",2026-12-27,2026-12-29,open,2026-11-08,,,
LH044-read-08,LH044,"Weekly read 8 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2027-01-03.md",2027-01-03,2027-01-05,open,2026-11-08,,,
LH044-checkpoint-w08,LH044,"Interim checkpoint after read 8 (brief, Reads): measures 2 to 4 and 7 to date, HTTP Archive context read, statement searches for every service walled at read 8; interim record version 0.2",2027-01-03,2027-01-07,open,2026-11-08,,,
LH044-read-09,LH044,"Weekly read 9 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2027-01-10.md",2027-01-10,2027-01-12,open,2026-11-08,,,
LH044-read-10,LH044,"Weekly read 10 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2027-01-17.md",2027-01-17,2027-01-19,open,2026-11-08,,,
LH044-read-11,LH044,"Weekly read 11 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2027-01-24.md",2027-01-24,2027-01-26,open,2026-11-08,,,
LH044-read-12,LH044,"Weekly read 12 of 13 under studies/LH044/brief.md (protocol 0.1): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, then --wayback <label> if any service is newly walled, the measure 7(b) searches for any closing or opening, and checkpoints/2027-01-31.md",2027-01-31,2027-02-02,open,2026-11-08,,,
LH044-read-13,LH044,"Weekly read 13 of 13, the last (brief, Reads): python3 -I studies/LH044/scripts/collect.py --read <UTC label>, --wayback <label> if any service is newly walled, --httparchive <label>, statement searches for every service walled, and checkpoints/2027-02-07.md; the collector refuses to run after 2027-02-09T23:59:59Z",2027-02-07,2027-02-09,open,2026-11-08,,,
LH044-record,LH044,"Record version 1.0 (Answer, Findings, Method, Limits, Next, Corrections) from the fourteen reads, with the pre-set readings of measure 3 by stratum; then its reviews before anything is released. The study's requests end on 9 February 2027",2027-02-08,2027-02-28,open,2026-11-08,,,

Investigation page (investigations/ai-cost-and-spread.md)

Under Questions, item 1, replace "brief to come" with: "brief fixed 8 November 2026 as studies/LH044/brief.md, with a baseline read the same evening and weekly reads to 7 February 2027". Under "Where the evidence stands", add:

  • 8 November 2026, baseline (unreviewed). The closing web's first read (checkpoint): of the services whose answer one labelled request could read, 13 of 67 named and 1 of 26 drawn at random showed a wall (challenge pages at Stack Overflow and Debian's forge and wiki, challenges on robots.txt itself at npm's website, HathiTrust, Europeana and the Humanitarian Data Exchange, rate limits at Maven Central, Semantic Scholar and Crossref, a refusal at KDE's GitLab). robots.txt kept the study away from 32 of 161 services, kernel.org's git server and PyPI's JSON API among them, so walls there are out of its sight. No wall could be dated: the Wayback Machine has not answered this container. kernel.org's own post attributes its load to crawlers gathering training data, while saying it can't tell bots from people with certainty; Wikimedia's 2026 rate-limit page doesn't mention AI. A snapshot, not a trend; weekly reads to 7 February 2027.

programme.md, Next

Brief and baseline, 8 November 2026: the closing web (LH044)

The investigation's first study is fixed (brief, protocol 0.1): 101 named shared services and 60 drawn at random from the OAI-PMH registry's last dump and Debian's mirror list, each read weekly with its robots.txt and at most one labelled request, until 7 February 2027, with the collector refusing to run after 9 February. The baseline read (unreviewed, checkpoint) found walls at 13 of 67 readable named services and 1 of 26 readable drawn ones, and robots.txt closing 32 services' probe paths to the study, so obeying robots.txt hides some of the walls it set out to see, kernel.org's among them. From read 1, robots.txt redirects are followed, up to five (Amendment 3, post hoc, after 22 probes were stopped at the baseline). Two things for the next session to weigh: whether a later task can run the Wayback dating from a machine that reaches web.archive.org, since this container never has; and the gate's reading of kernel.org's post, which does name AI crawlers. Interim checkpoints after the reads of 6 December and 3 January; the record by 28 February 2027.

sources.md

Proposed as new entries (next free S numbers; the driving session assigns them). Each was read on 8 November 2026 from the container, kept under studies/LH044/data/.

  • Open Archives Initiative, registered OAI-PMH base URLs (historical dump). A static dump, dated 8 October 2025, of the 7,103 base URLs (5,246 hosts) registered with the OAI-PMH registry, whose registration and validation services were discontinued on 18 July 2025. Entries were never re-checked for being alive; in LH044's draw of 40, 5 answered 404 or 410 and 5 had no DNS. Read 8 November 2026 (19:09 UTC). https://www.openarchives.org/pmh/registry/ListFriends_HISTORICAL_2025-10-08.xml
  • Debian mirror list. Debian's page of official mirrors by country, 387 /debian/ URLs on 351 hosts when read, page last modified 29 October 2025 by its own footer; most mirrors serve plain HTTP. Read 8 November 2026 (19:04 UTC). https://www.debian.org/mirror/list
  • HTTP Archive tech report API. The API behind HTTP Archive's Technology Report, at the address its report page uses (found in the page's script on 8 November 2026); /v1/adoption gives monthly counts per detected technology from January 2024 (September 2026 the latest when read); the documentation (HTTPArchive/tech-report-apis, apps/report-api/README.md) gives the schema but not the unit. Its rules detect Cloudflare Bot Management, Turnstile, DataDome, PerimeterX, Kasada, Akamai Bot Manager and captchas, not Anubis. Read 8 November 2026. https://cdn.httparchive.org/v1/technologies?onlyname
  • The Wayback Machine (CDX and capture endpoints). The Internet Archive's index of captures by URL and its raw captures (id_). From this container on 8 November 2026 every connection to web.archive.org was reset (four attempts, 19:04 to 19:30 UTC), and archive.org's availability API answered 200 with no capture for example.com and then 429. Unread from here. https://web.archive.org/cdx/search/cdx
  • ARIN RDAP. The American Registry for Internet Numbers' registration data service, used once to find who holds the address the container's requests leave from: 160.79.104.0/21, Anthropic, PBC. Read 8 November 2026 (19:06 UTC). https://rdap.arin.net/registry/ip/
  • Operators' statements on automated load (coded in studies/LH044/statements/coded.csv): kernel.org, people.kernel.org/monsieuricon/creepy-crawlies (29 August 2026); Wikimedia, www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits (2026) and diff.wikimedia.org's "How crawlers impact the operations of the Wikimedia projects" (1 April 2025); Read the Docs, "AI crawlers need to be more respectful" (25 July 2024); GitHub's changelog, "Updated rate limits for unauthenticated requests" (8 May 2025); COAR's Dealing With Bots resource and June 2026 slides. Each is the operator's account. Read 8 November 2026, 19:29 to 19:30 UTC.

Leads and proposals

  • Amendment 4 made (after R-0059, 8 November 2026, post hoc, from read 1): robots.txt read to 512 KiB with no cut-off line applied; echoed addresses and route recorded, tallies within route; a refresh-to-captcha signature (classifier 0.3). The keeper has since decided the question of robots.txt groups for Anthropic's agent tokens: see Amendment 5 above. The study page must be on main before read 1 (R-0059, Found 5).
  • Amendment 3 made (the driving session's decision, 8 November 2026): robots.txt redirects are followed, up to five, from read 1; at most seven requests to any host in a read; brief, Amendments. Not retroactive.
  • A Wayback route. Measure 5 needs a machine that reaches web.archive.org. Running collect.py --wayback <label> there, with the same code and caps, is the brief's provision; it needs a task that names the machine.
  • Correction to the gate's note on P-0013 (proposals.md, and notes/horizon/2026-11-07T0853Z.md): kernel.org's post names "AI crawlers" and training data as well as scrapers (statements/coded.csv, S01). Worth a line where the gate's reading is quoted.
  • Lead: the container as a sensor. This container's own policy answers for github.com and api.github.com with a 403 of its own, and its address block is Anthropic's; a study reading github.com from here would read the sandbox, not GitHub. Worth a line in CLAUDE.md's note on github.com.
  • Lead: declared walls over time. 11 robots.txt files allow every crawler but disallow named AI crawler tokens, and hackage.haskell.org's robots.txt redirects to the ai.robots.txt project's list; the Wayback (when readable) holds robots.txt history, which would date these declarations without any request to the operators.

From studies/LH044/for-driver.md in the repository, last changed 8 November 2026.