The rules of the road: a catalogue of what sites ask of automated readers
The rules of the road: a catalogue
In short. A website can ask automated readers to behave in more than a dozen different ways, from a line in a text file saying "please don't look here" to a wall that makes every visitor's computer solve a puzzle. Most of these requests are just that, requests: nothing stops a reader who ignores them, and the only way a site can tell is by reading its own logs. A few are enforced, by the server or by a company standing in front of it, and a few claim the force of contract or copyright law. This page lists each one, from its own specification or its operator's documentation as read on 8 November 2026, with who it asks, who if anyone enforces it, how anyone could tell it was heeded, and which of Lighthouse's classes of access it puts a source in. An evidence review checked it against the retained sources, and a reader-and-inference review corrected one sentence.
Lighthouse runs on Anthropic's models, its requests leave from an address block registered to Anthropic, and Anthropic runs crawlers named below. Every source is retained under data/reads/ with its read log (URL, UTC time, HTTP status, SHA-256 of the bytes received); the file is named in each row. Email addresses are withheld from the retained copies.
The classes are those of proposals.md P-0022: a rule Lighthouse follows voluntarily; a route Lighthouse does not consider; paid; licensed; private; ephemeral. "Ephemeral" describes data, not rules, and no mechanism below falls in it.
How to read the table
- Advisory means the mechanism states a wish and nothing technical stops a reader who ignores it. Enforced means a server or an intermediary refuses, slows or charges the reader. Claims legal force means the text says it binds the reader in law or contract; whether it does is not something this study decides.
- How a reader can tell it is heeded: what evidence would show compliance, to the site and to anyone outside it.
The catalogue
| # | Mechanism | What it asks, of whom | Advisory or enforced, and by whom | How anyone can tell it is heeded | Status | P-0022 class | Source (retained) |
|---|---|---|---|---|---|---|---|
| A1 | robots.txt (Robots Exclusion Protocol) | That "crawlers", defined as automated clients, not access the paths a group disallows. A crawler obeys the group naming its product token, else the * group; the longest matching rule wins. If robots.txt answers 4xx the crawler may go anywhere; if it cannot be reached (5xx or network error) the crawler "MUST assume complete disallow". Parsers must read at least 500 KiB. "These rules are not a form of access authorization." |
Advisory. The RFC's own security section says to use real access control for anything that must be kept out. | Only from the site's logs: whether a reader fetched robots.txt and then stayed out of disallowed paths. Outsiders can test it with sites they control (part B). | IETF standard, RFC 9309 (September 2022) | A rule followed voluntarily | catalogue-1/001.body |
| A2 | AI-specific user-agent tokens and shared block lists | robots.txt groups addressed to named AI crawlers and agents (GPTBot, ClaudeBot, Claude-User, Google-Extended and many more). The ai.robots.txt project keeps a list of 204 entries (203 user-agent tokens and one policy token) with each operator's stated purpose and whether it says it respects robots.txt, and ready-made files to paste in. Some tokens (Google-Extended) are "control" tokens with no crawler of their own: Google says it decides from that group whether content it crawled may train Gemini. | Advisory in robots.txt; the same project ships server rules (.htaccess, nginx, HAProxy) that refuse those User-Agents, which is enforcement by the site. | As A1, per token. A reader whose User-Agent names no listed token cannot be matched to these groups by the site. | Community project; vendor documentation | A rule followed voluntarily (since 8 November 2026 Lighthouse obeys groups naming the agent tokens of the provider, harness or model running it, AGENTS.md) | catalogue-2/013.body, 014.body, 017.txt |
| A3 | User-triggered fetchers' exceptions | Not a site's rule but operators' readings of A1: OpenAI says of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply"; Google says its user-triggered fetchers "generally ignore robots.txt rules"; Perplexity says its user fetcher "generally ignores robots.txt rules". Anthropic's page says its bots honour robots.txt and lists Claude-User as a token sites can disallow. | Operators' policy statements. | Affects how compliance is counted: a site that disallows ChatGPT-User and sees it fetch is a "bypass" to a measurer (part B) and within policy to the operator. | Operator documentation, read 8 November 2026 | (bears on A1 and A2) | catalogue-2/019.body, catalogue-3/001.txt, catalogue-3/002.txt, catalogue-2/018.txt |
| A4 | Crawl-delay | A minimum number of seconds between requests, in a robots.txt group. | Advisory and not in RFC 9309; Anthropic says it supports it; Cloudflare's verified-bot policy names ignoring it as a violation. | From logs: gaps between a reader's requests. | Non-standard extension | A rule followed voluntarily | catalogue-2/018.txt, catalogue-2/009.txt |
| A5 | robots meta tag and X-Robots-Tag | Per page or per response: don't index, don't show a snippet, and similar. Read by search engines when they fetch the page. | Advisory. | Whether a page appears in an index or a snippet. | Search-engine convention (Google's documentation) | A rule followed voluntarily | catalogue-2/016.txt |
| A6 | Content Signals (Cloudflare) | A Content-Signal: line in a robots.txt group saying yes or no to three uses after a page is fetched: search (an index with links and short excerpts, not AI summaries), ai-input (putting content into an AI model, such as retrieval or grounding for answers), ai-train (training or fine-tuning). Cloudflare is testing a fourth field, use=immediate, reference ("index, excerpt, and link back") or full ("summarize and reproduce"). Cloudflare's managed file defaults to search=yes, ai-train=no, use=reference. |
Advisory in mechanism; the policy text says "As a condition of accessing this website, you agree to abide by the following content signals" and calls a "no" an express reservation of rights under Article 4 of the EU copyright directive, so it claims legal force. Cloudflare separately enforces blocks on training crawlers for customers who choose it. | Not from logs: use happens after the fetch. Only the reader can show what it did with the content. | Vendor feature; Cloudflare says the policy is shown on millions of domains (search result, not read) | A rule followed voluntarily, with a licence claim | catalogue-2/003.body, 002.txt |
| A7 | AI Preferences (IETF aipref): vocabulary and attachment | A vocabulary of uses (train-ai, ai-use: "using an asset as input to a generative AI model, where the asset is not directly provided by the user", and search), each allowed or disallowed, attached by an HTTP Content-Usage header or a Content-Usage rule in robots.txt. Acquisition and use are separate: Allow and Disallow decide what may be fetched, Content-Usage what may be done with it. |
Advisory. The vocabulary draft says plainly that it does not "ensure that preferences are followed" and that a recipient "has a choice whether to follow those preferences". | As A6: only the reader can show use. | Working-group Internet-Drafts (vocabulary revision 08 and attachment revision 05 in LH045's census; editor's copies of 14 September 2026 read); "does not yet have consensus" notes remain | A rule followed voluntarily | catalogue-2/004.body, 005.body; LH045 census |
| A8 | A training-preference signal Anthropic says it honours | A robots.txt policy token, __usage_ai-training, defined in an Internet-Draft submitted by an Anthropic author (draft-kelley-aipref-training-preference-signal, 30 October 2026), to say "no AI training". Anthropic says that from 30 October 2026 it recognises such a signal, and that a group naming ClaudeBot takes priority over it. |
Advisory. | As A6. | Individual Internet-Draft; operator statement | A rule followed voluntarily | catalogue-2/013.body, 018.txt; LH045 census |
| A9 | TDMRep (W3C community group) | A machine-readable reservation of text-and-data-mining rights (tdm-reservation, with an optional tdm-policy link to licence terms), served in a well-known file, HTTP headers or HTML. Built for the EU directive's Article 4 opt-out. |
Claims legal force (an opt-out under EU law); nothing technical enforces it. | Not from logs. | W3C Community Group final report (10 May 2024) | Licensed | catalogue-2/021.txt |
| A10 | RSL (Really Simple Licensing) | A License: line in robots.txt (global, not tied to a user agent) linking to a machine-readable licence; "crawlers must retrieve and comply with the defined terms before accessing or processing the site's content". The specification also defines licence servers and crawler authentication. |
Claims contractual force; can be enforced where a licence server or an intermediary gates content. | Licence-server records; otherwise not. | Industry specification, RSL 1.0 | Licensed, or paid | catalogue-2/012.txt |
| A11 | Pay per crawl (HTTP 402) | An AI crawler either presents payment intent in request headers and gets HTTP 200, or receives HTTP 402 Payment Required with a price. | Enforced by Cloudflare as intermediary and merchant of record (closed beta). | Payment records; 402 responses. | Vendor feature | Paid | catalogue-2/022.body |
| A12 | Terms of service and data licences | Whatever the text says: some forbid storing or reproducing content (LH045's price sources), some forbid any "robot, spider, crawler, scraper, script ... or any other automated method to access" (Vast.ai, quoted in LH045's brief), some grant use with attribution (Epoch AI, LH045). | Claims legal force; enforced, if at all, by contract and courts. | Only by what the reader keeps and publishes, and by its access method. | Per site | Licensed (Lighthouse treats terms that forbid what a study needs as a wall: a route not considered) | studies/LH045/brief.md |
| A13 | Rate limits and HTTP 429 | "The user has sent too many requests in a given amount of time"; the response may carry Retry-After saying how long to wait. Documented limits vary: Wikimedia's API allows 10 requests a minute to a client known only by its address and 200 to one with a compliant User-Agent (LH044's retained copy). | Enforced by the server. A shared address can meet a limit spent by others (LH044). | Whether the reader slowed down or stopped after a 429. | RFC 6585 (2012); RFC 9110 for Retry-After | A rule followed voluntarily (pacing); a wall when met | catalogue-1/002.body; studies/LH044/data/statements-2026-11-08T1920Z/0004.b |
| A14 | Challenges and proof-of-work walls | A browser check (Cloudflare's "Just a moment..." page, AWS WAF, Debian's own challenge) or a proof-of-work puzzle (Anubis: "your browser is given a calculation task"). Asked of every client the operator's rules select. | Enforced by the server or its intermediary. Anubis's own documentation page met this study's reader with its challenge, and so did two part B sources. | Whether a reader answered the challenge; a reader that never solves one leaves only the refused request. | Vendor and open-source products | A route Lighthouse does not consider (a wall is an observation, never something to get around) | catalogue-2/011.txt; evidence-1/006.body |
| A15 | Login walls and API keys | That a reader identify itself as an account holder. | Enforced. | Account records. | Per site | Private, or paid; a route not considered unless a task names an account (the keeper's route, P-0022) | (none read) |
| A16 | Bot authentication: Web Bot Auth | That an automated client sign its requests with HTTP Message Signatures (RFC 9421), with a Signature-Agent header pointing to its keys, so a site can verify who sent it; the User-Agent alone "can be spoofed". |
Voluntary for the client; sites and intermediaries can require it and refuse unsigned traffic. | Verifiable signatures in logs. | IETF working group (webbotauth): protocol draft revision 01 of 29 October 2026 in LH045's census; the architecture draft read here (revision 05) expired on 3 September 2026 | (an identification scheme, not a request about use) | catalogue-2/006.body, 007.body; LH045 census |
| A17 | Verified bots and signed agents (Cloudflare) | To be "Verified", a bot must identify itself deterministically (Web Bot Auth, a published IP list with a stable User-Agent, or reverse DNS) and be "non-abusive": obey robots.txt and crawl directives, keep reasonable rates, not evade preferences. Since 1 July 2026 signed agents are folded in, labelled "Direct" or "Intermediary". Cloudflare de-listed Perplexity in August 2025 (part B). | Enforced by Cloudflare's allow and block lists for its customers. | Cloudflare's directory; whether a bot is listed. | Vendor programme | (an identification scheme; Lighthouse is not enrolled, so its requests are unverified to sites that use it) | catalogue-2/008.body, 009.txt, 010.txt |
| A18 | Published IP ranges and reverse DNS | Operators publish their crawlers' address ranges (Anthropic, Google, OpenAI and Perplexity do, by their pages), so a site can tell a real crawler from one using its name. Draft formats for such lists exist (draft-illyes-webbotauth-jafar; draft-kelley-webbotauth-sail, in LH045's census). | Verification aid; sites enforce by blocking. | Address match in logs. | Operator practice; drafts | (identification) | catalogue-2/018.txt, 017.txt; LH045 census |
| A19 | llms.txt | Not a restriction: an offer of a curated, model-friendly summary of a site at /llms.txt. |
Neither. | n/a | Community proposal | (an offer) | catalogue-2/020.txt |
| A20 | Crawler best practices | A proposed code for crawlers: identify, publish ranges, obey robots.txt, back off (draft-illyes-webbotauth-cbcp, 22 October 2026, in LH045's census; not read in full). | Advisory. | As A1, A4, A13. | Individual Internet-Draft | A rule followed voluntarily | LH045 census (abstract only) |
What the catalogue shows (interpretation, unreviewed)
- Most of the rules are requests. robots.txt, its AI tokens, Crawl-delay, the robots meta tag, Content Signals, AI Preferences, the training signal and the crawler best practices (A1, A2, A4 to A8, A20) are advisory by their own text, Content Signals while also claiming contractual force; the IETF's newest work says so outright ("does not ensure that preferences are followed"). Enforcement, where it exists, sits mostly with intermediaries (Cloudflare, AWS) or with the site's own server, not with the rule.
- Fetching and using are different questions, and only fetching shows in logs. robots.txt governs fetching; Content Signals, AI Preferences and TDMRep govern use after the fetch, which no site can see. Whether those are heeded can only be shown by the reader.
- Whether a rule applies depends on who the reader says it is. A group naming ClaudeBot binds ClaudeBot; whether it binds an agent built on Claude but calling itself something else was open for Lighthouse until the keeper's decision of 8 November 2026. Operators differ on user-triggered fetchers: OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Google and Perplexity say theirs "generally ignore" robots.txt, and Anthropic's page says its bots, Claude-User among them, honour it (row A3).
- Verification is the missing half. A User-Agent can be spoofed; published IP ranges and Web Bot Auth let a site tell. Lighthouse identifies itself honestly but unverifiably, and its requests come from a shared cloud address.
What was not read
The aipref drafts' datatracker records, ai.robots.txt's GitHub pages and the npm and GitHub API documentation sit on hosts Lighthouse audits in part C, which this study does not contact beyond robots.txt; the editor's copies, the jsDelivr mirror and LH045's census stood in. Anubis's documentation answered with its own challenge. The EU directive's text, the IETF rate-limit header draft and the full crawler best-practices draft were not read.
Corrections
Version 0.2, 9 November 2026, after the evidence review (notes/R-0063-lh047-evidence.md): "about a dozen" ways became "more than a dozen", since the table lists twenty rows and at least fourteen are asks; ai.robots.txt's 204 entries are 203 user-agent tokens and the policy token __usage_ai-training (part C's R3 uses the 203). No row's substance changed; version 0.1 is at commit 60b8110.
Version 0.3, 9 November 2026, after the reader-and-inference review (notes/R-0065-lh047-reader.md, problem 4): "Operators of user-triggered fetchers read the protocol as not covering them at all" overstated what the operators' pages say and left out Anthropic's contrary statement; it now quotes each operator, as row A3 does. No row changed; version 0.2 is at commit c23a350.
From studies/LH047/catalogue.md in the repository, last changed 9 November 2026.