Lighthouse

Lighthouse watches the internet for what AI systems do there and what their activity leaves behind, and publishes what it finds, with the evidence and its limits.

Studies LH044

LH044: Which shared services of public infrastructure close their doors to an anonymous, identified reader, when, and what do their operators say about why? A weekly panel, 8 November 2026 to 9 February 2027

In short (added 8 November 2026; not part of the protocol). Operators of shared public services, kernel.org and Wikimedia among them, have said this year that automated traffic is overwhelming them and that they are putting up walls: proof-of-work checks, rate limits, logins, closed APIs. We'll watch a fixed panel of 161 such services (forges, package registries, mirrors, documentation, archives and open data portals, plus repositories and mirrors drawn at random) once a week for thirteen weeks, sending each its robots.txt request and at most one labelled request, and record what that request sees and what the operators say. Nothing has been measured yet; the baseline read follows this brief. Our own requests leave from an address block registered to Anthropic, which runs crawlers, so what we see is what a client from that block sees. From the first weekly read we also obey what each site's robots.txt asks of the agents of the AI company whose models run Lighthouse (Anthropic now), which keeps us away from eight services the baseline probed. From the first weekly read we also note whether each robots.txt names __usage_ai-training, a shared token four AI companies announced on 30 October 2026 for saying no to AI training (added 10 November 2026). That's a site's stated wish, not a sign that any crawler obeys it, and our panel isn't the web.

Protocol 0.1, fixed 8 November 2026, before any panel service's probe URL or robots.txt was requested by this study, and before the panel was drawn. Writer: a research subagent in Claude Code (Claude Opus 5.5) for the driving session, from proposals.md P-0013, which the keeper agreed on 7 November 2026 with the gate's conditions (notes/horizon/2026-11-07T0853Z.md, Gate, P-0013); first study of What AI activity costs the shared internet, question 1. Before this brief, the writer read the scout file (notes/horizon/2026-11-07T0853Z/scout-7-wildcard.md), the gate, the investigation page, and the sampling frames and instrument tests kept under data/prebrief-2026-11-08T1915Z/ with their script; no operator statement page was opened. The classifier, collector, shared HTTP code, draw and named list are committed with this brief. One slip to disclose: while checking files, the writer ran draw_panel.py once before this commit (its --help is not implemented, so it drew); the panel/drawn.csv it wrote was deleted unopened, and the draw is deterministic, so the committed draw will be the same file. The keeper's caution of 8 November 2026 binds every step: a wall is the observation, and nothing here solves, evades or retries around one.

Question

From one or two labelled requests a week, over thirteen weeks: which services in a fixed panel of shared public infrastructure answer an anonymous, self-identified reader with a wall rather than their content; which of them change, in either direction, and between which reads; what does the Wayback Machine's record say about when each wall went up, where it can be read; what does each robots.txt declare to named AI crawlers; and what do the operators themselves publish about what they withdrew and why? The unit is the service at a read: one host, one fixed probe URL, one UTC read.

What the question is not. It does not measure load, does not attribute any traffic to AI, and does not say that any operator is harmed. An operator's statement that AI scrapers caused its load is recorded as the operator's, with what it rests on, never as Lighthouse's finding. A wall is a fact about what one request saw; it says nothing by itself about why it is there.

What is already known of the outcome

  • Operators' statements, as scout 7 quoted them on 7 November 2026 (reported, read through a fetch tool; not reopened before this brief): kernel.org (29 August 2026) reports about six million daily random-commit requests to git.kernel.org, an Anubis proof-of-work wall in front, legitimate traffic near 2 per cent, and warns "Expect to lose some functionality, at least when accessing our resources anonymously"; it names scrapers, not AI. Wikimedia's API rate-limit page (edited 15 October 2026) says limits are being deployed in 2026, with 10 requests a minute for clients known only by address. CBC (5 November 2026) reports academic libraries knocked offline by scraping bots and quotes Project Gutenberg's director. COAR (June 2026) reports repositories blocking its directory's checker and, in places, OAI-PMH. Titles only: Weird Gloop on scrapers and wikis (20 May 2026), mirrors.wikimedia.org's sunset (17 June 2026).
  • Statuses seen before this brief. Wikimedia's pageview API answered 429 to scout 7's first request on 7 November 2026 at 09:00 UTC (the same API family as panel service D02, a different URL, sent without this study's User-Agent). Today, before the brief: archive.org's Wayback availability API answered 200 (19:05:07 UTC, with no capture listed even for example.com) and then 429 (19:05:34; archive.org hosts panel service A01), and web.archive.org reset the connection on all three requests made to it (19:04:55 to 19:05:48), as index.commoncrawl.org did once (19:06:24). The study's collector, tried on neutral hosts in a scratch directory, also had web.archive.org reset its robots.txt request at 19:15:18. The frame and context hosts answered 200 (www.openarchives.org, www.debian.org, raw.githubusercontent.com, httparchive.org, cdn.httparchive.org, rdap.arin.net); none is a panel host. Lighthouse's own records say github.com's issue and release pages returned 403 to this container on 28 September 2026 (CLAUDE.md), and other studies read pypi.org (LH040) and Packagist (LH039) without refusal on 7 and 8 November.
  • The writer's general knowledge (recalled from training, not verified, and to be looked for by the statement step): during 2025 several open-source forges and archives reported automated load and put proof-of-work or similar walls in front of web views (among them, as the writer recalls, GNOME's GitLab, SourceHut, Fedora's Pagure and others); Wikimedia described crawler load in 2025; Docker Hub and GitHub tightened anonymous limits in 2025. So several named services may already be walled at the baseline. That is why the baseline is a description, not a reading of change, and why the drawn strata exist.

Lighthouse's own requests

This study adds Lighthouse's automated requests to services whose operators report being overwhelmed. It keeps them few, labelled, polite and finite (the gate's conditions):

  • Identified. Every request names Lighthouse and this study's page: Lighthouse-research/1.0 (+https://lighthouse.harbour.cat/studies/LH044/) for robots.txt, APIs and files, and Mozilla/5.0 (compatible; Lighthouse-research/1.0; +https://lighthouse.harbour.cat/studies/LH044/) for page probes. The second is the form self-identifying crawlers use on web pages; it claims to be no browser, and its Mozilla prefix means a wall set for browsers is shown to it rather than skipped, which is what a page probe exists to see. No email address, cookie, credential, key or token is sent; no JavaScript is run; no form is submitted; no account is made.
  • Few. In a read, a panel host gets at most two requests: its robots.txt, then at most one probe. No two panel services share a host. No retries. A redirect is recorded and not followed. A probe that robots.txt disallows is not sent. Requests are sequential, paced 1.5 seconds apart.
  • robots.txt obeyed, RFC 9309. The product token is Lighthouse-research; its group applies if present, else *; longest match wins, Allow wins a tie, * and $ are honoured (classify.py, robots_allowed). A robots.txt that answers 5xx, fails to connect or redirects means no probe that read (robots_unreachable, robots_redirect); 4xx other than 429 means no rules (RFC 9309, "unavailable"). A robots.txt answered with a challenge or a 429 is itself the observation (wall_at_robots) and no probe follows.
  • Left alone when asked. A probe answered with a Retry-After that runs past the next read's start skips that service at that read (skipped_retry_after).
  • Finite. Fourteen reads (the baseline and thirteen weekly ones). The collector refuses to run after 2027-02-09T23:59:59Z (END in collect.py).
  • Labelled in the record. Every request is a row in a read's readlog.csv (URL, UTC time, status, bytes kept, sha256, User-Agent form, route), and each read's summary gives the request count, bytes retained and the most requests sent to any one host, beside the readings (charter: account for our own presence, measure our overhead).
  • Where they come from. HTTPS requests leave this container through its egress proxy. Before this brief, rdap.arin.net (read 19:06:22 UTC) gave the address archive.org echoed back, 160.79.106.30, as part of NET-160-79-104-0-1 (160.79.104.0 to 160.79.111.255), registered to Anthropic, PBC. Hosts on the container's no-proxy list (pypi.org, registry.npmjs.org, proxy.golang.org among the panel) and plain-HTTP probes leave by a direct route whose address was not read. Each read records any address a response echoes (egress_ip_echoed).

The panel

Population. Shared services of public infrastructure: services run for the public by an organisation, used anonymously by many people and programs, in six named classes: code forges and development infrastructure; package registries; mirrors and download services; documentation and wikis; archives, libraries and scholarly services; open data portals. Two further strata are drawn at random from public frames: open repositories and journals that speak OAI-PMH, and Debian mirrors.

The named stratum (panel/named.csv, 101 services, fixed in this commit before any status is read). Rule: for each class, the services named in the scout file and gate as candidates or as having reported walls, and the largest shared public services of that class the writer could name, as the writer listed them on 8 November 2026 before any status read, one probe per host. It is a judgement list, not a sample, and it is partly chosen for its reports: column prior marks reported (an operator statement quoted in the scout file: git.kernel.org, Wikimedia's API, Project Gutenberg), title (named in a report title only: Weird Gloop's wiki), candidate (proposed as a panel member by scout 7), else empty. Classes: forge 19, registry 21, mirror 11, docs 15, archive 22, opendata 13.

The drawn strata (scripts/draw_panel.py; panel/drawn.csv, written after this commit and before any status is read). Frames, both retained before this brief and read only from the retained files: the Open Archives Initiative's static dump of registered OAI-PMH base URLs at the registry's discontinuation (ListFriends_HISTORICAL_2025-10-08.xml, 7,103 entries, read 19:09:09 UTC), and Debian's mirror list page (351 distinct /debian/ URLs, read 19:04:53 UTC). One URL per host (the lexicographically first), hosts in the named list and the frames' own hosts dropped, hosts sorted, then random.Random(seed).sample with seed the SHA-256 of LH044 panel, fixed 8 November 2026: 40 OAI-PMH hosts and 20 Debian mirror hosts. They are the comparison for the named stratum: chosen by no one for their outcome. The OAI frame is about a year old and its entries were never checked by anyone for being alive; a dead endpoint is unreachable or gone, not a wall.

Probes. Each service has one probe URL and a surface: page (a human-facing page; compatible User-Agent; Accept HTML), api (an anonymous API; plain User-Agent; Accept JSON or XML) or file (a static file on a mirror; plain User-Agent). Each has an optional content marker, a string the real content contains (for OAI-PMH, <Identify>; for Debian mirrors' README, Debian). Markers were written from the writer's knowledge, not from any response, so some may be wrong; a missing marker is a flag, not a wall.

Fixed. No service is added or replaced after the baseline, except by a dated amendment, labelled post hoc, for a probe that answers gone at the baseline (a probe the writer got wrong), with the replacement's first read as its baseline.

Measures

  1. What a single request saw (classify.py, version 0.1), from the status, headers and first 64 KiB of the probe's response, in this order: a network failure is unreachable (or unread_proxy when the container's own proxy refused, which is never a wall); a challenge signature (Cloudflare challenge or Turnstile, Anubis, go-away, AWS WAF, DataDome, HUMAN, Imperva, Sucuri, Vercel, Kasada) is challenge unless the response is a 2xx carrying the marker; 429, or 503 with Retry-After, or 403/503 with rate-limit wording, is rate_limited; 401 is api_closed on an API, login elsewhere; a redirect to a login path is login, to a challenge path challenge, otherwise redirect_other; 403 or 451 with a vendor's block page is refused (with the vendor), with authentication wording api_closed or login, otherwise refused; 404/410 gone; other 4xx client_error_other; 5xx server_error; a 2xx with the marker open, with a captcha widget and no marker challenge, with a password field on a page and no marker login, with no marker set open, with JavaScript-required wording and no marker js_notice, otherwise open_unconfirmed. A wall is challenge, login, rate_limited, refused, api_closed or wall_at_robots. robots_disallowed is not a wall: it is a declaration, counted under measure 4. js_notice and open_unconfirmed are not walls; each checkpoint lists them with what the retained body shows, and any reading of one as a wall is labelled post hoc.
  2. The tally: services by class at each read, by stratum and by class of service, with the share behind a wall among those with a probe answered.
  3. Changes: a closing is a service open at one read and walled at the next; it holds if it is still walled at the read after (a closing at the final read is unconfirmed). An opening is the reverse. Each closing and opening is listed with the two reads that bracket it and the classes either side.
  4. Declared walls: from each robots.txt read, whether it allows this study's probe, whether * may fetch / and the probe, and whether each of 17 named AI crawler tokens (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, CCBot, Google-Extended, PerplexityBot, Perplexity-User, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai, Diffbot) may fetch /. Counted at each read; changes listed.
  5. When the wall went up (collect.py --wayback): for each service walled at the baseline or at its first walled read, the Wayback Machine's CDX list of captures of the probe URL since 1 January 2024 (deduplicated by digest), then archived captures read raw (id_) and classified with the same classifier, first the latest and the earliest and then by bisection, at most eight per service, giving the last capture not walled and the first walled. The bisection assumes a wall, once up, stayed up; a capture shows what the Wayback's crawler saw, with its own address and User-Agent, not what this study's request would have seen. A fixed check set (every tenth open service by id) gets a CDX listing only, as a check of the archive's coverage. At most 60 requests to web.archive.org per run, 4 seconds apart, after its robots.txt; the run stops at the first 429 or two network failures in a row, and whatever is unread is undated, never "no wall before". From this container web.archive.org reset every connection on 8 November; if that holds, measure 5 is unread here, and the script may be run under a later task from any machine that reaches it, with the same code and caps, its outputs kept the same way.
  6. Context: HTTP Archive (collect.py --httparchive): one request to HTTP Archive's tech report API (cdn.httparchive.org/v1/adoption, the address its own report page uses, found on 8 November) for the monthly counts of origins on which its crawler detects each of: Cloudflare Bot Management, Cloudflare Turnstile, Akamai Bot Manager, DataDome, PerimeterX, Kasada, AntiBot.Cloud, AWS WAF Captcha, hCaptcha, reCAPTCHA and Friendly Captcha, from January 2024, all geographies and ranks. Read at the baseline, week 4, week 8 and the end. It counts a technology's presence on crawled home pages, not walls and not shared infrastructure; its rules detect no Anubis (scout 7, and its technology list read 8 November). Context only; nothing in it is compared with the panel.
  7. What operators say. (a) At the baseline: the statement pages in statements/seed.csv, fetched once each after their robots.txt (collect.py --statements). (b) At each weekly read: for each closing or opening, and at weeks 4, 8 and 13 for each service walled at that read, a bounded search for the operator's own statement (at most two web searches per service, for the operator's name with "bots" and with "anonymous access" or the wall's kind, and at most two result pages opened per service), the queries, results and pages kept. Each statement is coded in statements/coded.csv: operator, URL, date published, read time, what was withdrawn or limited, the mechanism named, the reason stated and the words the operator uses for the traffic (for example "AI scrapers", "crawlers", "bots", "abuse"), quoted. The attribution is the operator's. A press report is coded as the reporter's account of the operator. Nobody is contacted.
  8. Our own load: requests, bytes and the most requests to one host, per read (measure 1's summary).

The comparison that matters

The named stratum against the drawn strata. The named services are the ones people talk about and were partly chosen because their operators reported walls; the drawn repositories and mirrors were chosen by a seed. If walls and closings are common in the named stratum and rare in the drawn ones, that is what selection on reports alone would also produce, and the record says so; if the drawn strata show them too, the closing reaches past the services that make the news, within the limits of two frames. The unit is the service; per-stratum counts are reported separately and never pooled into a rate for "the web".

Pre-set readings, over the thirteen weekly reads. For each stratum and for the panel: "the panel closed" if closings that hold number at least three and exceed openings that hold by at least three; "the panel opened" in the mirror case; otherwise "held". With about 160 services and thirteen reads, this can register only a few changes; "held" is not "nothing is happening", and a rate-limited service can flap from read to read with the load on a shared address.

Missing evidence

  • A read that does not happen in its window is missing, not filled.
  • unreachable, unread_proxy, robots_unreachable and robots_redirect are unknown for that read; unknown is never open and never walled. A closing or opening needs both bracketing reads known.
  • A Wayback run that is cut short leaves the rest undated. A statement not found by the bounded search is "none found", not "none made".
  • An operator's silence says nothing about its reasons.

Limits stated now

  • One request, one address, one client. A wall may be set for data-centre addresses, for Anthropic's address block in particular, for unknown clients, for this User-Agent, or for everyone; one request cannot tell them apart, and the direct and proxied routes may leave from different addresses. A page that walls browsers but passes tools, or the reverse, shows the wall only on the matching surface.
  • The named stratum is the writer's list and is partly selected on reports; the drawn strata come from two frames, one a year old. Neither is a census of anything.
  • The probe is one URL per service. A wall on another path, at another time of day, or after more requests than one is not seen.
  • Thirteen weeks is short against walls that went up over 2025 and 2026; the Wayback is the only reach back, and it may be unreadable from here.
  • Markers and signatures can miss a new kind of wall (open_unconfirmed, js_notice) or a changed page.

Overlap with earlier studies

No earlier Lighthouse study reads any panel service's access. LH040 reads pypi.org's JSON and ClickPy, LH039 reads Packagist and W3Techs, and LH006 recorded Docker Hub's anonymous limit through a shared address (S24); their requests are part of Lighthouse's traffic to those hosts and are listed in their own readlogs. Scout 7 suggested the panel's shape and quoted the statements above; this study reads them itself.

Model disclosure

Lighthouse runs on Anthropic's models; Anthropic runs web crawlers (ClaudeBot among the tokens in measure 4) and its address block is where this study's requests come from. Any reading that attributes load to AI crawlers carries this disclosure, and reviews are asked to look for wording easier on AI companies, Anthropic included, than the sources allow, in either direction. A wall against Anthropic's addresses would be both a finding about the closing web and a limit on this instrument; the record says which it can tell apart, which is little.

Resource ceiling

Per read: at most 322 panel requests (two per service), retained bodies at most 64 KiB each. Statement fetches: at most two requests per statement host per run. Wayback: at most 61 requests per run. HTTP Archive: two requests per context read. Fourteen reads in all, so at most about 4,500 panel requests over the study. No account, key, credential, BigQuery or paid service; no contact with anyone. Model spend: about $3 a read on Sonnet or Opus for collection and the checkpoint, about $20 for each of the two interim checkpoints and $30 for the record, so about $110 at list rates before reviews.

Reads, schedule and stopping

  • Baseline, 8 November 2026, as soon as this brief and then the drawn panel are committed: python3 -I studies/LH044/scripts/draw_panel.py (commit panel/drawn.csv), then python3 -I studies/LH044/scripts/collect.py --read <UTC label>, --httparchive <label>, --statements <label> studies/LH044/statements/seed.csv, --wayback <label>; a descriptive checkpoint in checkpoints/2026-11-08.md. The baseline is not a record version.
  • Weekly reads on Sundays 15, 22 and 29 November; 6, 13, 20 and 27 December 2026; 3, 10, 17, 24 and 31 January and 7 February 2027, each in a window of that day and the two after, each with --read, --wayback if any service is newly walled, the measure 7(b) searches for any change, and a short checkpoint in checkpoints/<date>.md.
  • Interim checkpoints at week 4 (after the 6 December read, by 10 December) and week 8 (after the 3 January read, by 7 January): measures 2 to 4 and 7 to date, HTTP Archive's context read, an interim record version 0.1 and then 0.2.
  • Record, version 1.0, by 28 February 2027, then its reviews before anything is released.
  • Stopping: after the 7 February 2027 read, or earlier if the keeper stops it or an operator asks Lighthouse to stop by any route that reaches it; a robots.txt that disallows this study's token stops its requests to that service at once, by the rule above. The collector refuses to run after 9 February 2027. The panel, probes, classifier, measures and pre-set readings do not change after any status is read; any later change is a dated amendment, labelled post hoc if made after statuses were seen.

What would leave it inconclusive

  • The container's egress changes between reads (a different address block echoed): classes either side are not compared; the record says so.
  • More than a fifth of services unreachable, unread_proxy or robots_unreachable in a read: that read's tally is reported and kept out of measure 3.
  • More than a fifth of answered probes open_unconfirmed or js_notice at the baseline: the classifier is not separating walls from content, and measures 2 and 3 are reported as the classifier's with that caveat.
  • Measure 5 unread throughout: walls are undated beyond the reads themselves.

It cannot say, in any case, how much automated traffic any service receives, who sends it, whether any of it is AI, or whether any operator is harmed; those are operators' accounts, recorded as theirs.

Amendments

  • 8 November 2026, before any panel status was read (after the draw, which reads only the retained frames). The Debian frame holds 387 /debian/ URLs on 351 hosts, not "351 distinct URLs" as written above; the draw used hosts, as the rule says, so nothing changes. The OAI frame holds 7,103 URLs on 5,246 hosts. One drawn OAI-PMH endpoint (P01) is a bare address on port 81: it is in the panel because its operator registered it for harvesting, and it gets the same two requests as any other.
  • Amendment 2, 8 November 2026, after the baseline read was seen (post hoc). Classifier version 0.2: (1) a 403 whose body is this container's own GitHub policy message ("access to this repository is not enabled for this session") is unread_proxy, not a wall; version 0.1 called github.com and api.github.com login and api_closed; (2) Cloudflare's /cdn-cgi/challenge-platform script, which it injects into ordinary pages, no longer counts by itself as a challenge; version 0.1 had called readthedocs.org's robots.txt redirect a wall. --classify now re-reads kept robots.txt responses too. Collector: email addresses in kept headers and text bodies are replaced with a placeholder before saving (the keeper, 8 November 2026; scripts/redact_emails.py); readlog hashes stay those of the bytes received. No probe, service, measure or pre-set reading changed. A proposed amendment that was not made, following robots.txt redirects, is set out in checkpoints/2026-11-08.md for the driving session.
  • Amendment 3, 8 November 2026, made after the baseline read was seen (post hoc); in force from read 1 (15 to 17 November 2026), not retroactive: the baseline stays as read. The driving session decided it on the evidence of the baseline, where robots.txt redirects stopped 22 probes (8 named, 14 drawn). RFC 9309 (section 2.3.1.2) says crawlers should follow at least five consecutive redirects when fetching robots.txt, so treating a redirect as unknown both departed from the standard this brief cites and set aside operators' rules. From read 1:
    • robots.txt only: the collector follows up to five redirects, same-site or cross-site, resolving a relative Location against the URL that gave it. Each hop is a request, logged in readlog.csv as robots-redirect-<k> and counted in the budget. The rules of the final robots.txt apply to the probe host, read by the same RFC 9309 matching. A chain still redirecting after the fifth hop is robots_redirect and no probe is sent. A hop that ends in a network failure, a 5xx, or a refusal by this container's own policy is robots_unreachable. A hop answered with a challenge or a 429 is wall_at_robots. 4xx other than 429 means no rules. In each case no probe is sent unless the final robots.txt allows it.
    • Probes still follow no redirect.
    • The new ceiling: at most seven requests to any one host in a read, enforced by the collector across all services (HOST_CAP). For one service that is its robots.txt, up to five redirects and one probe. A request that would pass the ceiling is not sent: the service is robots_unreachable (host cap) or not_probed_host_cap, both unknown and never walls.
    • Resource ceiling, replacing "at most 322 panel requests" a read: at most seven requests per service, so 1,127 a read in the worst case. On the baseline's pattern, one hop each for 22 services, about 271 is expected.
    • Elsewhere: the statement and HTTP Archive runs fetch robots.txt the same way. The Wayback run's single robots.txt request is unchanged and stays within its 60-request budget.
    • Unchanged: the panel, probes, classifier version 0.2, measures, pre-set readings, the thirteen weekly reads and the end date (no request after 2027-02-09T23:59:59Z). Offline checks are in scripts/test_lh044.py, which makes no network request.
    • Comparability: at read 1 some services will have a probe for the first time. A closing or opening needs both bracketing reads known (brief, Missing evidence), so these services enter measure 3 from read 1 onwards, and the record lists them.
  • Amendment 4, 8 November 2026, made after the baseline read and its evidence review (notes/R-0059) were seen (post hoc). In force from read 1 (15 to 17 November 2026), not retroactive: reads labelled before 9 November 2026 are classified as before (classifier 0.2), and the baseline stays as read. It fixes three faults in the instrument that the review found:
    1. robots.txt read limit. robots.txt (and each redirect hop under amendment 3) is read up to 512 KiB. RFC 9309 section 2.5 sets a parsing limit of at least 500 KiB; the baseline read only 64 KiB. A cut-off last line is never applied as a rule: a robots.txt still cut at 512 KiB is robots_unreachable for that read, so no probe is sent, and for measure 4 only its complete lines are parsed. At the baseline, CRAN's robots.txt (R12) was the only one cut. The only rule that matched its probe was the fragment Disallow: /web/pac, so the baseline's "32 disallow the probe" is 31 in the text read, with CRAN's rule unknown (as the checkpoint's correction says). CRAN is read afresh at read 1.
    2. Echoed addresses and route. Each read's summary now records every address a response echoes back, per route. That covers headers named for the purpose (x-ip, x-client-ip, x-ratelimit-ip, x-real-ip, x-forwarded-for, cf-connecting-ip, true-client-ip and similar), any header whose whole value is an IPv4 address, and an address following an address-like key in a kept body (for example ipr:). Each echo goes to echoes.csv (request, service, route, where, name, address), and summary.json gets egress_ip_echoed and echoed_blocks_16 keyed by route. classes.csv gains each service's route (the probe's, else the final robots.txt request's), and summary.json gains by_stratum_route, so named and drawn tallies can be reported within route as well as by stratum. The inconclusive rule ("the container's egress changes between reads") is applied within each route, by registered block. These are facts about the read rather than rules, so the baseline's derived tables were regenerated with them; no class changed (scripts/test_lh044.py checks this). At the baseline the proxied route echoed six addresses in 160.79.106.0/24, inside the block registered to Anthropic, PBC. P04's pages on the plain-HTTP route echoed 34.172.54.233 and 34.136.29.35, whose registrant was not looked up. The drawn stratum went mostly by plain HTTP (44 of its 60 services by route of last request) and the named stratum mostly through the proxy (97 of 101). So at the baseline the named-against-drawn comparison is also largely a comparison of address blocks, and the record says so.
    3. Refresh-to-captcha pages (post hoc, from P04). Classifier 0.3 adds one challenge signature, captcha-refresh: a page that sends the client at once to a captcha path, by meta refresh or a script assignment, or that names /.well-known/sgcaptcha/. It applies to probes and robots.txt alike, so such a robots.txt is wall_at_robots and no probe follows. Under it, the baseline's P04 would have read as wall_at_robots. The probe sent there under 0.2 followed a robots.txt answered with a captcha page, which the brief's rules meant not to happen. The signature was written after P04 was seen.
    • Unchanged: the panel, probes, cadence, the thirteen weekly reads and the end date. The baseline's study-page 404 (R-0059, Found 5) is for the driving session, by putting the study on main before read 1. It is not a change to the collector.
  • Amendment 5, 8 November 2026: the keeper's decision on whose robots.txt rules Lighthouse obeys. In force from read 1 (15 to 17 November 2026) for every request the study makes; not retroactive, so the baseline stays as read. It replaces the open question noted here earlier the same day (should Lighthouse also honour groups for Anthropic's agent tokens?), which the keeper answered: "Lighthouse should obey the rules, and is both it's own agent, currently using this harness, model, etc. when we use OpenAI we'll follow those relevant rules, etc." (proposals.md, From the keeper; AGENTS.md, Rules that matter). From read 1:
    • Which groups apply. A robots.txt group applies if it names Lighthouse's own token, Lighthouse-research, or an agent token of the provider running the session; if the file names none of those, * applies. Where several apply, the strictest wins: the probe is withheld if any applicable group disallows it. This replaces "the product token is Lighthouse-research; its group applies if present, else *" under Lighthouse's own requests, and "this study's token" under Stopping. Matching is otherwise RFC 9309 as before: product tokens case-insensitive (the leading run of letters, _ and - in a user-agent line's value), groups naming the same token combined, longest match, Allow on a tie, * and $, complete lines only, up to 512 KiB (amendment 4), redirects as amendment 3.
    • Whose tokens. The provider comes from the environment variable LIGHTHOUSE_PROVIDER (anthropic or openai). Unset, or any other value, means the collector cannot tell, and every listed provider's tokens apply. The tokens are kept in the shared scripts/robots_tokens.json, from the providers' own pages read on 8 November 2026 (Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, and Claude-Agent for its hosted browsers; OpenAI: GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User). It also lists names that robots.txt files in this baseline use for those providers' agents and that no provider page read lists, marked as seen in the wild: Claude-Code, anthropic-ai and Claude-Web; ChatGPT (also how ChatGPT Agent is read) and OpenAI. notes/robots-tokens-2026-11-08.md gives the sources.
    • The code. Every request the collector makes (--read, --wayback, --httparchive, --statements) is decided by the shared scripts/robots_rules.py. services.json and summary.json record the provider (all when unset). A withheld probe is robots_disallowed with the deciding group in its detail ("by ClaudeBot: Disallow: / (line 94; provider anthropic)"). For reads labelled from 9 November 2026, measure 4's columns are read with the same matcher, and robots.csv gains obeyed_probe_allowed, obeyed_decided_by and obeyed_provider. On every robots.txt parsed at the baseline, both matchers give the same measure 4. The baseline's derived tables are unchanged. scripts/test_lh044.py checks all of this offline, including a read run through the collector with the network replaced by the retained responses.
    • What it would have changed at the baseline (derived from the retained robots.txt files; checked by scripts/test_lh044.py and by scripts/robots_rules.py's self-test). Of the baseline's 88 probes, eight would not have been sent, whether the provider is set to anthropic, set to openai or left unset: wiki.gentoo.org (D15), runescape.wiki (D11), codeberg.org (F06), git.sr.ht (F07), repo1.maven.org (R06) and central.sonatype.com (R07), all in the named stratum, and jurnal.itg.ac.id (P18) and ftp.eq.uc.pt (Q07), drawn. Every one of the eight files disallows / to ClaudeBot, which decides as anthropic or unset, and to GPTBot, which decides as openai. These are the eight the evidence review listed. No other baseline probe would have been withheld, and no baseline file would let through a probe that protocol 0.1 refused. The eight keep their baseline classes as read: open (F06, D11, D15, P18, Q07), rate_limited (R06, R07) and server_error (F07). The 29 probes whose robots.txt answered 4xx had no rules under either reading. For the 41 services whose robots.txt redirected (22), was unreachable (12) or was walled (7) at the baseline, no rules were read, so whether this rule would have changed anything there is unknown.
    • What it does to the measures. While their files say the same, these eight services get only their robots.txt request from read 1 and read as robots_disallowed: a declaration, not a wall and not open. They leave measures 1 to 3, because a closing or opening needs a known class on both sides, and they stay in measures 4 and 7. The record lists them. That takes six services out of the named stratum's probes and two out of the drawn strata's. Two of the six (R06, R07) were walled at the baseline, so those two walls will not be seen again by this instrument. If a file later stops disallowing, the next probe is that service's first since the baseline, and the same rule about both bracketing reads applies.
    • A reading for the driving session to confirm. As the rule is written, * applies only when no applicable token is named. So a file that disallows * but allows a named provider token would let through a probe that protocol 0.1 refused. No baseline file does this (above). The collector follows the rule as written, and robots_rules.py's self-test records it, so a change of reading would show as a change to that test.
    • Unchanged: the panel, probes, User-Agent strings, classifier 0.3, measures, pre-set readings, the cadence of thirteen weekly reads and the end date (no request after 2027-02-09T23:59:59Z).
  • Amendment, 9 November 2026, by the scheduled run (in force from the next read). Usage preferences (schedule row use-reference-1, after LH047's finding R4). Where a site's Content-Signal asks for use=reference or use=immediate (Cloudflare: "index, excerpt, and link back"), a whole copy kept in Lighthouse's public repository goes beyond what it asks.
    • The baseline's bodies. Four bodies are withheld: the probes read-2026-11-08T1920Z/0140.b (DOAJ, A19) and 0190.b (jurnal.itg.ac.id, P18), and the statements statements-2026-11-08T1920Z/0006.b and 0008.b (COAR, S03 and S04). Each now holds a placeholder. Beside it, <n>.withheld.json keeps the sha256 of the bytes received (as readlog.csv gives it), the hash of the copy kept until now, the headers file's name, the Content-Signal line as read and the robots.txt copy and line it came from, the date, and the extracts the records use: the panel marker in context for the probes, and for the statements the quotations in statements/coded.csv. For the two probes it also keeps the class derived from the whole body before it was withheld. Headers files and robots.txt copies are kept whole. classes.csv is unchanged: --classify reproduces it from the kept derivations, and scripts/test_lh044.py checks each record against readlog.csv and classes.csv. Earlier commits still hold the bodies; history is not rewritten on a shared branch.
    • From the next read. collect.py reads the Content-Signal lines of the robots.txt that governs each host, through the shared scripts/content_signal.py. It reads the groups the keeper's rule selects, together with Lighthouse-research's group or else *, so that a file naming only training crawlers still has its * signal read. It also reads a Content-Signal response header; this is a cautious extension, since Cloudflare defines the line for robots.txt only. Where either asks for use=reference or use=immediate, a 2xx body is not kept, whether it is a probe, a statement, a Wayback Machine response or an HTTP Archive response. The run writes the placeholder and the record instead, and holds the whole body in <n>.b.withheld-local for that run only (git ignores the file), so that --classify and the coder can read it. --classify writes the class, the marker in context and any echoed addresses into the record, so a later re-classification needs no body. A statement's quotations go into coded.csv as before, and they are its bounded extract. robots.txt bodies are always kept.
    • Not changed. ai-train=no (Lighthouse trains no model). ai-input=no, which bears on whether a model may read a body at all; no baseline file carries it, and a rule for it is left to the keeper. AI Preferences' Content-Usage, whose vocabulary has no term for keeping or excerpting. The panel, probes, measures, classifier 0.3, cadence and end date are unchanged.
  • Amendment 6, 10 November 2026: the shared __usage_ai-training token as measure 4's 18th entry (schedule row P-0025-amendment; proposals.md P-0025, started as a deepening of this study by the gate of 10 November 2026, notes/horizon/2026-11-10T1415Z.md, whose six conditions it follows). Fixed and committed before anyone searched this study's kept robots.txt files, or any later read, for the token. In force from read 1 (15 to 17 November 2026), and on the baseline's kept files as a tabulation that makes no request. Writer: a research subagent in Claude Code (Claude Opus 5.5) for the driving session of the scheduled run 2026-11-10T1415Z.
    • What is reported, as reported. Every claim here is a source's, read through scout 6 of the 10 November sweep (scout-6-policy.md, lead 1), not observed by Lighthouse. ppc.land, "Google, Microsoft, OpenAI and Anthropic back one AI training opt-out" (published 30 October 2026, read 10 November 2026, 14:23 UTC; scout 6, row 19): the four companies each announced on 30 October support for one robots.txt policy token, __usage_ai-training, written in a User-agent line and governing use, not crawling; Bing Webmaster Tools' transition opens on 15 December 2026 and Microsoft honours the token from 1 February 2027; Google says "early 2027", OpenAI "coming months", and Anthropic "has already started implementing"; Meta, Amazon and Apple were not among them. ppc.land is a secondary source and does not link the primary posts. Search Engine Roundtable (published 30 October 2026, read 10 November, 14:25 UTC; row 20) quotes Google's John Mueller, "We aim to support this new control along with other AI providers in 2027", and reports that Google will read the shared opt-out as disallowing Google-Extended where Google-Extended is not named. OpenAI's own announcement answered HTTP 403 to the scout (10 November); Google's Search Central post and Microsoft's Bing post were not opened. The IETF AI Preferences working group's attachment draft, draft-ietf-aipref-attach-05 (19 August 2026, read 10 November, 14:23 UTC; row 18), defines a different mechanism, a Content-Usage rule inside User-agent groups. The per-purpose policy-token approach is draft-kelley-aipref-training-preference-signal-00 (30 October 2026; author K. Kelley, Anthropic; read by scout 6 of the 7 November sweep, 08:56 UTC, row 4), which binds preferences "to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line". The gate and P-0025 say an Anthropic author drafted the token. The scout's quotation of that draft doesn't give the token's string, and nobody checked the draft's text for it for this amendment.
    • What the writer knows of the outcome. Nobody has opened or searched any robots.txt this study keeps, or any later read, for the token: not the sweep, not the gate (its record says so), and not this writer. The writer's offline checks re-read the baseline's kept files for the existing measures only, and counted its parsed files from services.json; nothing searched any of them for this token. Two things are known beforehand, both by the gate's account of 10 November 2026, not read again here: the community list ai.robots.txt already carries the token, and one panel file redirects to that list. That file is hackage.haskell.org's (R13; for-driver.md, 8 November). At the baseline its redirect wasn't followed (protocol 0.1), so the list isn't among the baseline's kept files. From read 1 the redirect is followed (amendment 3); whether the target can be read from this container isn't known in advance, and an unread hop is unknown, not a zero. So a file in the panel may carry the token at read 1 because a community list does, not because its operator decided anything about this token.
    • The measure. For each robots.txt parsed at a read (complete lines only, amendment 4; matched by scripts/robots_rules.py, so the product token is the leading run of letters, _ and -, case-insensitive): whether a user-agent line names __usage_ai-training; how many groups name it and the line each starts on; the other tokens those groups name; those groups' Allow and Disallow lines as read; whether those rules alone allow /; and whether the token's string appears anywhere in the text, comments included. Unlike the 17 tokens, there's no fallback to *: when no group names the token, "allows /" is blank. The token isn't a crawler, and Google's reading of the unnamed case (above) is one crawler's policy, not this measure. Other lines in the group (a Content-Usage line, for instance) aren't recorded by this entry; the amendment of 9 November left Content-Usage unmeasured, and that stands. Counted at each read by stratum, never pooled, with changes listed, as for the other 17 entries. A robots.txt that answered 4xx is "no file" and counted apart. One that was unreachable, still redirecting, walled, cut off by the host cap or refused by this container's policy is unknown for the token, never a zero.
    • The code. classify.py names the token (USAGE_TOKEN) beside AI_TOKENS, which is unchanged. collect.py's usage_token_entry and write_usage_token write the entry to usage_token.csv in the read's directory. robots.csv and its 17 columns are untouched. --classify writes the file for reads labelled from 2026-11-15 (read 1). For the baseline, python3 -I studies/LH044/scripts/collect.py --usage-token 2026-11-08T1920Z writes it from the 91 robots.txt files the baseline parsed (its robots.csv rows), and makes no request. The driving session runs it once, after this amendment is committed, and reports it with read 1's checkpoint. Those 91 files are what the baseline read: 64 KiB of each, CRAN's cut (complete lines only), and 41 services with no rules read at all (redirects not followed, unreachable or walled; amendment 5). The baseline is 8 November 2026, nine days after the 30 October announcements, so it isn't a reading from before the announcement either. scripts/test_lh044.py, check 9, tests the entry on synthetic files only. Check 1 reproduces the baseline's robots.csv byte for byte and confirms that --classify on the baseline writes no usage table. The token isn't added to scripts/robots_tokens.json: it states a wish about training use, not an agent for Lighthouse to obey, and Lighthouse trains no model (as with ai-train=no, amendment of 9 November).
    • No new requests or hosts. The token is read from the robots.txt each read already fetches, under the same caps. The top-sites sample P-0025 proposed is a new collection and stays in the backlog. The resource ceiling is unchanged.
    • What a count can and can't say. The panel is 161 services of shared public infrastructure: 101 named by the writer's judgement and partly for their reports, and 60 drawn from two frames of repositories and Debian mirrors. It isn't publishers, top sites or the web. A zero in it isn't "the web before 15 December", and a count isn't a prevalence anywhere else. A file may carry the token before any crawler honours it: by the reports above, Microsoft starts on 1 February 2027, Google in 2027 and OpenAI at no stated date, while Anthropic's "already started" is ppc.land's account, unchecked here. A token in a file is the operator's stated wish (or a list's, where the file copies one). It isn't evidence that any crawler complies. This study observes no crawler's behaviour and can't see compliance. A rise or fall is a description; why an operator added or removed the token needs the operator's own words (measure 7), and no pre-set reading is made.
    • Model disclosure. By the gate's and P-0025's account an Anthropic author drafted the token, and Anthropic co-announced it on 30 October 2026 (ppc.land, as reported). Lighthouse runs on Anthropic's models, Anthropic's crawlers are among the agents the token speaks to, and this study's requests leave from Anthropic's address block. Any wording about how widely the token is used, or about whether it works, is reviewed for being easier, or harder, on AI companies, Anthropic included, than the counts and sources allow.
    • Ends with LH044. The entry is read at each remaining weekly read and ends with the last, on 7 February 2027 (no request after 2027-02-09T23:59:59Z). It adds no recurring read and no schedule row of its own. It's reported at the checkpoints and in the record with measure 4. Anything that would follow the token after 7 February 2027 is a new proposal.
    • Unchanged: the panel, probes, User-Agent strings, classifier 0.3, the rules obeyed (amendment 5), measures 1 to 3 and 5 to 8, the 17 tokens of measure 4 and their columns, the pre-set readings, the cadence and the end date.

From studies/LH044/brief.md in the repository, last changed 10 November 2026.