Whose robots.txt rules Lighthouse obeys, and which collectors check

In short. On 8 November 2026 the keeper decided that Lighthouse obeys a site's robots.txt as its own agent and as whatever runs it. So a group addressed to Anthropic's agents binds Lighthouse while it runs on Anthropic's models, and OpenAI's while it runs under Codex. This note says what the rule is, where the token list comes from, and what shared code now applies it. It also lists every collector in studies/ that makes requests to other people's sites, with whether each checks robots.txt at all, for a sweep the driving session will schedule. Most don't; two have fetched kernel.org pages whose robots.txt, as read on 8 November, disallows every crawler.

The rule

The keeper, 8 November 2026 (proposals.md, From the keeper; AGENTS.md, Rules that matter): "Lighthouse should obey the rules, and is both it's own agent, currently using this harness, model, etc. when we use OpenAI we'll follow those relevant rules, etc."

• A robots.txt group applies if it names Lighthouse's own token, Lighthouse-research, or an agent token of the provider, harness or model running the session. If the file names none of those, * applies.
• Where several groups apply, the strictest wins: a path is off limits if any applicable group disallows it.
• A collector names the provider it runs under, in the environment variable LIGHTHOUSE_PROVIDER (anthropic or openai). Unset, or any other value, means it can't tell, and it obeys every listed provider's tokens.
• Matching is RFC 9309: product tokens matched without regard to case, groups naming the same token combined, longest match, Allow on a tie, * and $ honoured, /robots.txt always allowed, the body read up to 512 KiB, and only complete lines applied.

The code

• scripts/robots_tokens.json: the tokens, each with its source and when it was read.
• scripts/robots_rules.py: a small matcher using only the standard library. decide(body, url, provider) returns allowed or disallowed, which token's group decided, the deciding line and its line number. provider_from_env() reads LIGHTHOUSE_PROVIDER. python3 -I scripts/robots_rules.py runs its offline self-tests. Those include the eight closing-web baseline probes, on their retained robots.txt files: all eight are allowed under the old Lighthouse-only rule and disallowed under this one.
• The closing web (studies/LH044) uses it from read 1 (its brief, Amendment 5). No other collector has been changed.

The tokens and where they came from

The two providers' own pages were read once each on 8 November 2026, after their robots.txt, with an honest User-Agent and nothing else. They're kept under robots-tokens-2026-11-08/sources/ with email addresses withheld. In 002.b, Cloudflare's hex encoding of an address (data-cfemail and an email-protection# link) was also replaced by hand, and the values of cookies the servers set are withheld from the kept headers. fetchlog.csv gives each request's URL, UTC time, status and the sha256 of the bytes received, and fetch_sources.py is the script that made them.

Seen in the wild, not documented. These are names that robots.txt files in the closing web's baseline (studies/LH044/data/read-2026-11-08T1920Z/, read 19:18 to 19:26 UTC) use to address one of the two providers' agents, and that no provider page read lists. They're marked as such in the JSON and applied.

Choices made here, for the driving session to confirm or reverse

• Seen-in-the-wild names are applied. It is the stricter choice: a file that writes User-agent: OpenAI or User-agent: Claude-Code means those agents. Setting "applied": false on an entry in scripts/robots_tokens.json would keep it on record without obeying it. On the closing web's baseline it makes no difference: the same eight probes are withheld either way, because ClaudeBot or GPTBot decides each one.
• Claude-Agent is applied, though Anthropic's page describes it as a browser token and does not mention robots.txt.
• ChatGPT-User is applied, though OpenAI's page says robots.txt rules "may not apply" to user-initiated actions.
• * only when no applicable token is named, as the rule is written. So a file that disallows * but allows ClaudeBot by name lets Lighthouse in as anthropic, where the old Lighthouse-only rule kept it out. No closing-web baseline file does this (0 of 91 parsed). The self-test records the behaviour, so a change of reading would show as a change to that test.
• Not found: no token for Codex on OpenAI's page, and no User-Agent for Claude Code's own fetches on either Anthropic page. Neither was searched for further.

Collectors in studies/ that make requests to third-party sites

How they were found: grep -rlE 'urllib|import requests|from requests|curl|fetch\(|urlopen|http\.client|httpx|aiohttp|wget' over studies/ (harbour/.session/ not searched), then each match read for an actual request. Analysis, figure and review scripts that only parse URLs or retained files are left out: LH008 analyse.py and draw_figure.py, LH009 and LH010 analyse.py, LH010 reanalyse.py, LH012 and LH013 review scripts, LH016 replay_x3.py, LH017 r0026, LH033 python_side.py, LH034 R-0039_checks.py, LH035 build.py (its reads were ad hoc), LH044 draw_panel.py and redraw.py, and LH047 audit.py, normalise_logs.py, robots_index.py and transcript_requests.py. "Open rows" are the study's open rows in schedule/due.csv on 8 November 2026. None of these files was edited.

"Own UA, robotparser" means Python's urllib.robotparser checked for the collector's own User-Agent. That parser ignores * and $ inside rules, and it doesn't consult any provider's tokens.

Many of these hosts are APIs (PyPI's JSON, deps.dev, ClickHouse's public SQL, Docker's registry, GitHub's API) rather than pages. The rule covers them all the same; the sweep would read each host's robots.txt.

kernel.org (derived from retained files; no request made). LH044 read the robots.txt files of git.kernel.org and lore.kernel.org at 19:18:56 and 19:18:58 UTC on 8 November 2026. Both disallow / to * and to ClaudeBot. Under the old rule and the new, those files close every git.kernel.org URL in LH036's panel (the CVE's mbox file and the upstream commit) and LH038's two kernel policy pages and lore.kernel.org query. LH036's read logs show the mbox file requested at its reads of 1, 2 and 5 November, and LH038's show its policy pages requested on 4 November. Whether the files said the same on those days was not checked. Both studies have reads scheduled: LH036 from 9 November, LH038 from 17 November. This was found by checking other studies' URLs against the 91 robots.txt files LH044 kept. Hosts LH044 did not read were not checked.

Sweep of 9 November 2026

Part of schedule row robots-sweep-1, by a subagent of the scheduled run. Each collector below now asks scripts/robots_gate.py before every request, sends Lighthouse's User-Agent (Mozilla/5.0 (compatible; Lighthouse-research/1.0; +https://lighthouse.harbour.cat)), and withholds email addresses from bodies it keeps. A URL the gate does not allow, or whose robots.txt cannot be read, is not requested and is logged "not fetched" with the deciding group and rule, as a missing reading. Most collectors use the driving session's scripts/lh_fetch.py, a gated drop-in for urlopen. For studies whose shared scripts/common.py holds a requests session, the gate wraps the session itself, so direct S.get and S.post calls are covered too. Where common.py runs git, a clone's https URL is checked first. In an existing partial clone whose origin is not allowed, git is told not to fetch missing objects (GIT_NO_LAZY_FETCH). Each study's brief has an amendment of the same date. Nothing was run against a live site: each edited file was compiled, the gate was exercised offline with stubbed robots.txt files, and the offline tests of LH010, LH015 and LH017 pass. LH030 to LH034, LH037, LH039 and LH040 were done by the driving session and aren't listed here.

Left open: requests sessions follow redirects without asking the gate again, as urllib does for LH036; a params= query added by requests isn't seen by the gate; and for git the gate checks the repository's URL, not the /info/refs path git actually requests. Bodies kept by the old Docker Hub and GitHub collectors (LH018 to LH029) are now redacted, so an offline replay that compares a logged sha256 with a kept file would differ wherever an address was withheld; none of those studies has a read scheduled.

The collectors read on 9 November (the scheduled run itself)

LH030, LH031 (collect.py, criteria.py, vendor.py), LH032 (advisory_check.py), LH033, LH034, LH037, LH039 (collect.py, packagist.py) and LH040 (collect.py) now import urlopen from scripts/lh_fetch.py, a drop-in that asks the gate first and raises NotFetched (logged by each collector's own error handling) for a disallowed URL or an unreadable robots.txt; redacted() from the same module withholds email addresses before a body is saved. LH032's git clone of NCSC-NL/php2shell is checked by hand against the gate before each read (github.com's robots.txt answered HTTP 403 on 9 November: no rules). Each brief has a dated amendment. What the first gated reads met:

Every other URL these collectors requested was allowed. The earlier requests to nodejs.org/dist/, api.wordpress.org and api.deps.dev belong in the rules-of-the-road re-audit (row LH047-reread). collect_prebrief.py in LH040 ran once before the brief and is not gated.