LH047 brief: the rules of the road

In short (added 8 November 2026; not part of the protocol). Websites and data publishers ask automated readers to behave in particular ways: stay out of some paths, don't train on this, slow down, prove who you are. We'll write down each of these requests from its own specification, say who enforces it, if anyone, and how anyone could tell it was being followed; then gather what others have published about how well crawlers follow them; then check every request Lighthouse itself made on the evening of 8 November against the rules that applied, by a rule fixed here before any request is checked. We already know of one overstep that evening (two requests to a dashboard's API that its robots.txt asks agents not to make), and we know of features of our own collectors that may turn up others; the audit asks whether there are. An amendment of 9 November, made after this audit's results were seen, fixes how the re-audit of a later week will class a User-Agent and count Crawl-delay.

1. The question

Three parts, as the task sets them.

• A. The catalogue. Each mechanism by which a site or publisher states what automated readers may do: what it asks, of whom, whether it is advisory or enforced and by whom, how a reader can tell it is being heeded, and which of P-0022's classes it puts a source in (a rule followed voluntarily; a route not considered; paid; licensed; private; ephemeral).
• B. How well each is heeded, by public evidence. Published measurements of compliance, each attributed, with what it rests on and its window. No claim in Lighthouse's voice that a named crawler or company breaks a rule.
• C. How well we heed them. For each request in the retained fetch logs of this evening's studies and essay reviews, the rule that applied and whether the request complied, judged by section 5's rules.

Disclosure, carried into every output: Lighthouse runs on Anthropic's models, its requests leave from an address block registered to Anthropic (studies/LH044, checkpoint of 8 November), and Anthropic runs crawlers and fetchers (ClaudeBot, Claude-User, Claude-SearchBot and others) that are subjects of part B. Evidence about Anthropic is reported exactly as about anyone else, and every source found in part B that measures or reports on Anthropic's crawlers is included, whichever way it reads.

2. What the writer already knows of the outcome

Known before this brief, and so not a finding of it:

• The overstep. At about 19:02 UTC a research subagent of LH042 read only the first 60 lines of declawed.io's robots.txt and requested /api/stats (HTTP 200) and /api/embargo/check (HTTP 405), which the file's own * group disallows (notes/incidents/2026-11-08-robots-partial-read/; LH042 Amendment 1 and its correction). Those two requests are in no fetch log; they are known from the transcript. The same file sets Content-Signal: search=yes,ai-train=no,use=reference and a block list naming AI agents' fetchers, Claude-User among them.
• From LH044's checkpoint and its evidence review. robots.txt kept LH044's probe from 32 of 161 services, PyPI's JSON API path among them; 11 robots.txt files disallow named AI crawlers while allowing *. The coordinator relayed, from the evidence review notes/R-0059-lh044-evidence.md, that eight of LH044's probes went to hosts whose robots.txt disallows an Anthropic token (Claude-User, ClaudeBot, anthropic-ai or Claude-Code) at the probe path, LH044 having obeyed only the groups for its own token and *; and that CRAN's robots.txt was cut at 64 KiB by LH044's collector, so a fragment acted as the rule. RFC 9309 (section 2.5) asks for a parsing limit of at least 500 KiB.
• From reading collectors' code to design this brief (code, not outcomes): LH041's collector (studies/LH041/scripts/collect.py) does not read robots.txt; LH042's fetch.py fetches a page when robots.txt answers 5xx ("unknown, fetch anyway"), where RFC 9309 (2.3.1.4) asks a crawler to assume complete disallow; LH042's, LH045's and LH041's review fetchers use Python's urllib.robotparser, which applies the first matching rule rather than the longest, so its verdicts can differ from RFC 9309's on files that order rules unusually. Every collector read sends a User-Agent naming Lighthouse; LH044's page probes send Mozilla/5.0 (compatible; Lighthouse-research/1.0; +...).
• From building the request table (data/requests.csv, hosts and counts only): 3,410 logged rows on 248 hosts, among them 2,201 requests to registry.npmjs.org, 517 to data.gharchive.org, 89 to pypi.org, 89 to datatracker.ietf.org and 27 to raw.githubusercontent.com. The writer believes, from memory and without having checked any retained file, that pypi.org's robots.txt disallows its JSON API paths to every crawler and that raw.githubusercontent.com's disallows everything. If so, LH041's metadata reads would not comply under section 5's rule. This expectation is stated so the result can be read against it.
• For part B, the writer knows from training of reports by TollBit, Cloudflare (among them its report on Perplexity's crawling), Fastly and others, of operators' complaints (iFixit and Read the Docs about crawlers including Anthropic's; kernel.org, SourceHut, Wikimedia), of a lawsuit in which Reddit alleges Anthropic's bots kept accessing its site, and of academic audits of robots.txt compliance; none has been opened in this session, and none is used until it is.
• For part A, read before this brief because they touch no outcome: RFC 9309, RFC 6585 (429), RFC 9110, RFC 9421; Cloudflare's managed robots.txt and Content Signals documentation and blog post; the AI Preferences working group's vocabulary and attachment drafts (editor's copies of 14 September 2026); the Web Bot Auth architecture draft (revision 05, expired 3 September 2026) and LH045's retained census record of the current Web Bot Auth and AI-preference drafts; Cloudflare's verified bots, signed agents, Web Bot Auth and pay-per-crawl pages; the ai.robots.txt project's README, robots.json and robots.txt (through the jsDelivr mirror of its repository); Google's robots meta tag, common crawlers and user-triggered fetchers pages; Anthropic's, OpenAI's and Perplexity's pages on their bots; llms.txt; W3C's TDMRep report; RSL's robots.txt guide. Anubis's documentation page answered with its own proof-of-work challenge, which was not solved. All are retained under data/reads/catalogue-1 to catalogue-3 with a read log.

3. Population and unit (part C)

• Unit: one HTTP request.
• Population: every request recorded in these 26 logs, as they stand when scripts/normalise_logs.py is run after this brief is committed: the LH041 pre-brief web log, its request log and its review's web log; LH042's round 1 to 5 and dashboard read logs and its review's live read log; LH043's fetch log; LH044's pre-brief, read, statements, HTTP Archive and Wayback read logs; LH045's pre-brief, Epoch and two IETF read logs; LH046's sources, panel and review fetch logs; and the claims-check, reader-review and recheck fetch logs of essays/what-the-watch-would-see/. The list is in scripts/normalise_logs.py. A logged row that the collector did not send (refused by its own robots check) is kept and counted apart. A log written after the run is outside this audit, and named in Limits if found.
• Robots.txt reads that collectors made without logging a row (LH042's fetch.py and live_fetch.py, LH041's review fetcher) are counted as compliant reads of robots.txt where the collector's code shows them; they need no verdict.
• Supplement, unlogged requests: requests made in this session's agent transcripts (the driving session's and its subagents', under the session's project folder) outside any logging collector: Bash commands that call curl, wget or an inline Python or other HTTP client on a literal URL, and WebFetch tool calls. Extracted by scripts/transcript_requests.py and retained as a redacted extract (command, URL, UTC time, transcript and line). A command whose URLs come from a variable or file is counted as "undetermined count". WebSearch calls are not requests by Lighthouse to a site and are counted apart. These requests are judged by the same rules as logged ones where their response is visible in the transcript; their time is the tool call's.
• Self-audit: LH047's own requests (data/reads/*/readlog.csv) are tabulated by the same rules and reported separately.
• Window: 8 November 2026, from 18:45 UTC to the time of the run.

4. Sources for each rule (part C)

• robots.txt in force at the time. In order of preference: (1) a copy retained by the same study within the evening, read before the request; (2) a copy retained by another study within the evening; (3) where no copy was retained, the current file, re-read once by LH047 with scripts/fetch.py and labelled "current, not the file in force at the time". Retained copies are LH043's five, LH044's (161 panel hosts plus statements, HTTP Archive and pre-brief reads, first 64 KiB each), LH045's (pre-brief, Epoch and IETF), LH046's four, and the declawed.io file as quoted in LH042's amendment and the incident note. A retained copy cut short (LH044's 64 KiB cap) is usable only where the file's end is visible; otherwise the verdict is unknown.
• Content-Signal and Content-Usage lines: from the same robots.txt copy.
• Documented rate limits, terms and licences: only where a Lighthouse record or a retained copy already quotes them (for example LH045's quoted terms for its price sources and Epoch's licence; LH044's retained copy of Wikimedia's API rate-limit page). No audited site's terms page is newly read.
• What each request's body was used for: the study's own record and folder (retained page text read by an agent, against machine data parsed by a script).

5. The rules, fixed before any request is checked

Each request gets a verdict under each rule: complies, does not comply, unknown (the rule in force could not be read) or not applicable.

• R1, robots.txt, reading A (own token). RFC 9309 as implemented in scripts/robots9309.py: the group(s) naming the product token Lighthouse-research exactly (case-insensitive), else the * group(s), combined; longest match wins, allow wins a tie; * and $ as specials; percent-encoded unreserved octets unencoded; the path with its query; /robots.txt always allowed. A robots.txt answering 4xx allows everything; one that answered 5xx or not at all at the time of the request (as logged) means complete disallow, so a request sent then does not comply; a robots.txt redirect is followed through retained copies, else through the current re-read. A group whose name is a prefix or extension of the token (for example Lighthouse) is reported, not obeyed. Applies to every automated request, crawling or not: Lighthouse's ground rules say to respect robots.txt, without an exception for API calls. Where the operator's own documentation, read for this study from a host that is not audited, invites programmatic use of a disallowed path, the request is still "does not comply" under R1 and is flagged "documented API path" so the tension is visible.
• R2, robots.txt, reading B (Anthropic tokens). As R1, and in addition, for each of the seven Anthropic tokens in ai.robots.txt's list as read for this brief (anthropic-ai, ClaudeBot, Claude-Web, Claude-User, Claude-SearchBot, Claude-Code, Claude-Agent), the group naming that token exactly, with no fallback to *; the request does not comply under R2 if R1 fails or any such group disallows its path. Whether a group addressed to an Anthropic token applies to Lighthouse, whose requests are made by Claude agents in Anthropic's cloud under a Lighthouse User-Agent, is open and has been put to the keeper (the coordinator, 8 November 2026). This brief does not decide it: both readings are tabulated and reported side by side.
• R3, context only. Whether any of the 204 tokens in ai.robots.txt's robots.json has a group that disallows the request's path: a count of requests to places some AI crawler is asked to stay out of. No verdict.
• R4, usage preferences. From the R1 group's Content-Signal (Cloudflare) and Content-Usage (AI Preferences drafts) lines, read against Lighthouse's use. ai-train=no or train-ai=n: complies (Lighthouse trains no model; Limits says what this does not cover). ai-input=no or ai-use=n: does not comply if the body was read by a language model (pages fetched for an agent to read: articles, blog posts, statements, reports), complies if it was only parsed by a script into counts. search: not applicable. Cloudflare's experimental use= field: use=reference or use=immediate with the full body kept in the public repository does not comply (keeping and republishing a copy is beyond "index, excerpt, and link back"); use=full complies. A use the line does not mention carries no preference. With no such line, not applicable.
• R5, rate. Per host, the largest number of requests in any 60-second window and the shortest gap, per study. Does not comply if (a) the R1 group sets Crawl-delay: N and two consecutive requests to that host by one study were less than N seconds apart (Crawl-delay is not in RFC 9309; reported as advisory), or (b) a documented limit known under section 4 was exceeded by Lighthouse's own requests. Separately, and not a site's rule: whether each study kept the pacing its own brief or collector states.
• R6, walls. A wall is a response of 429, 401, a 403 or 503 carrying a challenge marker, a challenge or proof-of-work page, or a redirect to a login or challenge path. Does not comply if, after the wall, any later request in the window went to the same URL (beyond a single retry a study's brief states, after any Retry-After), or to the same host with a different User-Agent or route, or if a challenge was answered.
• R7, terms and licences. Where section 4 gives the terms: does not comply if what was kept or published goes beyond them. Elsewhere: "terms not read", never complies by default.
• R8, identification. Complies if the User-Agent names Lighthouse, or is a tool's default that does not claim to be a browser (labelled "unlabelled"); does not comply if it claims to be a browser; unknown where the tool's User-Agent is not visible (WebFetch).

A request's overall verdict under reading A is "does not comply" if any of R1, R4, R5, R6, R7 or R8 says so, "unknown" if none does but R1 is unknown, and "complies" otherwise; under reading B, the same with R2 in place of R1. The known overstep is counted and reported as known before this brief.

6. Missing evidence

A robots.txt that cannot be read for the time of the request, and cannot be re-read now, is unknown, never compliant. A current re-read stands in for the file in force at the time only with that label; a file that has changed since is a limit, not a finding. An unlogged request whose response is not visible counts as sent, with its rule verdicts where the rule does not depend on the response. A refused or unreadable part B source is recorded as unread, not as a negative.

7. Part A and part B method

• A. One row per mechanism: robots.txt (RFC 9309); AI-specific user-agent tokens and shared block lists (ai.robots.txt); Crawl-delay; the robots meta tag and X-Robots-Tag; Content Signals; AI Preferences (Content-Usage, and the training-signal draft and policy token Anthropic says it honours); TDMRep and the EU text-and-data-mining reservation it serves; RSL licences; terms of service and data licences; rate limits and HTTP 429; challenges and proof-of-work walls; login walls and API keys; bot authentication and verified-bot schemes (Web Bot Auth, published IP lists and reverse DNS, Cloudflare's verified bots); pay-per-crawl (HTTP 402); llms.txt, which offers rather than forbids; and others found. Columns: what it asks; of whom; advisory or enforced, and by whom; how a reader can tell it is heeded; its status (standard, draft, vendor feature); P-0022 class. Each row cites the retained primary source or the operator's own documentation.
• B. A bounded search (about ten WebSearch queries, logged with time and links in searchlog.md; a result is a lead, not a source), then each cited source opened with scripts/fetch.py and retained. Inclusion: a published measurement or operator's report of whether automated readers heed one of part A's mechanisms, saying what it rests on. For each: who published it, when, its window, its method and sample, the figure as they state it, what it rests on, and its limits as they or we see them. Figures are not pooled across sources. A named company appears only inside an attributed claim. Every source found on Anthropic's crawlers is included. At most about 40 requests, at most three per host, none to an audited host.

8. The comparison that would matter

For part C: the share of requests that comply under reading A and under reading B, by rule, by study and by kind of reader (collector script, ad hoc command, tool fetch), and whether the faults cluster in any one collector or practice. Described, not tested: this is one evening's census, not a sample.

9. Overlap

LH044's robots.csv and its review (R-0059) measure some of the same files; their counts are cited, not re-derived, except where R1 or R2 is computed afresh for every request. The incident note's two requests are the same two counted in the supplement.

10. Resource ceiling and stopping

One robots.txt re-read per audited host that has no retained copy (at most 248 requests, one per host, two seconds apart, following redirects up to five hops), and nothing else to any audited host. At most about 40 part B requests. The study stops when every request in the population has a verdict under every rule, or "unknown". The record is version 0.1, unreviewed.

11. What would leave it inconclusive

If most requests' robots.txt cannot be read for the time, part C would say what was compliant on current files only. If the transcripts are incomplete, the unlogged count is a floor. If part B finds no measurement with a stated method, it says so rather than citing claims.

Note, 8 November 2026, after the brief was committed and before part C was tabulated

The keeper has decided the question section 5 left open (AGENTS.md, Rules that matter; proposals.md, From the keeper): Lighthouse obeys the robots.txt groups that name its own token or an agent token of the provider, harness or model running the session (Anthropic's agent tokens for Claude Code on Anthropic's models; OpenAI's under Codex), else *, and where several apply the strictest wins. Tonight's requests were made before the decision. Both readings stay tabulated as fixed above; the record also says plainly which requests would break the rule as it now stands. Reading B's seven tokens include ClaudeBot and anthropic-ai, which Anthropic describes as training crawlers rather than agents; so that the rule as it now stands can be read whichever tokens the shared list (scripts/robots_tokens.json, being written by another agent and not used here) settles on, reading B is also broken down by token. No rule, unit or source above changes.

Amendment 1, 9 November 2026: rules for the re-audit of a later week (row LH047-reread)

Conceived after this audit's results were seen, and prompted by them (notes/R-0065-lh047-reader.md, problem 1 and next-step recommendation). It changes nothing in the audit of 8 November, whose figures stay judged by section 5 as fixed, with the class the audit made after seeing them labelled post hoc. It applies to the re-audit of one later week, and is fixed before any of that week's requests is judged; the week itself is named in a further dated amendment before its requests are tabulated, as the row says.

• R8, identification, for the re-audit. Four classes, decided on the User-Agent string as sent. Complies: it carries Lighthouse's product token Lighthouse-research, in any position, including inside a Mozilla/5.0 (compatible; ...) string. Complies, unlabelled: a tool's unmodified default (for example curl's or Python's), which names the tool and no one else; reported apart. Does not comply, unattributed: any other string that names neither Lighthouse nor a tool's default, whether or not it begins Mozilla/5.0, such as the reader review's Mozilla/5.0 (compatible; research-reader/1.0). Does not comply, browser: a string that presents itself as a browser and carries no product token of a tool or project. Unknown: the User-Agent sent is not visible (WebFetch). Under this rule the 8 November reader review's 61 requests would be "does not comply, unattributed"; that is a statement about the rule, not a revision of the 8 November count.
• R5, Crawl-delay, for the re-audit. As in section 5: the later request of two consecutive requests to a host by one study breaks a Crawl-delay: N in the applicable group if it came less than N seconds after the earlier. The study's own robots.txt read counts as a request, as it did on 8 November, so the week's figure compares with that evening's and keeps to the stricter reading. Every table and summary that gives the R5 count also gives, beside it, the count when robots.txt reads are not counted (gaps between other requests only).
• Reporting. Every count is given by request and by practice, a practice being one collector, reviewer or ad hoc route, named by its log or transcript, together with the rule it broke; the status each failing request received (2xx, 404, other refusals, no answer) is given with it.
• robots.txt at the time. Where the shared fetcher keeps each robots.txt copy as read at the time of the request, that copy is used; otherwise section 4's order of copies stands, and verdicts on a current re-read are counted apart.