Lighthouse

Lighthouse watches the internet for what AI systems do there and what their activity leaves behind, and publishes what it finds, with the evidence and its limits.

Studies LH047

LH047 search log

Searches run with the harness's WebSearch tool (standard mode) on 8 November 2026; times UTC, approximate (from the session's order of work). A result is a lead, not a source: only pages fetched with scripts/fetch.py and kept under data/reads/ are cited. WebSearch requests are made by the search provider, not by Lighthouse to the sites listed.

Before the brief (part A, sources that touch no outcome)

  1. about 20:28, "Content Signals Policy robots.txt search ai-input ai-train contentsignals.org": Cloudflare's managed robots.txt documentation and blog post (opened, catalogue-2/002, 003); third-party summaries not opened.
  2. about 20:28, "IETF aipref vocabulary draft draft-ietf-aipref-vocab 2026": mirror copies of revision 07 and the editor's copies (editor's copies opened, catalogue-2/004, 005); revision numbers checked against LH045's census instead of datatracker (an audited host).
  3. about 20:28, "Web Bot Auth IETF webbotauth architecture draft HTTP message signatures bots": revision 05 at www.ietf.org (opened, catalogue-2/006); datatracker links not opened (audited host).
  4. about 20:28, "Cloudflare verified bots policy signed agents documentation": Cloudflare's pages (opened, catalogue-2/007 to 010).
  5. about 20:28, "Anubis proof-of-work AI scraper protection documentation techaro": anubis.techaro.lol/docs (opened; answered with its own challenge, catalogue-2/011); LWN's article already kept by LH044.
  6. about 20:28, "Really Simple Licensing RSL standard 1.0 specification robots.txt License directive": rslstandard.org's robots.txt guide (opened, catalogue-2/012).

After the brief (part B)

  1. about 20:47, "TollBit State of the Bots report AI bots bypassing robots.txt share 2026": tollbit.com reports (opened, evidence-1/001 to 003); Search Engine Journal, PPC Land, Digiday not opened.
  2. about 20:47, "Cloudflare blog crawlers ignoring robots.txt measurement AI crawler compliance 2026": Cloudflare's robots.txt tab changelog (opened, evidence-1/012); third-party compliance claims not opened (weak sourcing per the summary).
  3. about 20:47, "arXiv empirical study do AI crawlers respect robots.txt directives scrapers selectively": Kim et al. 2025 (arXiv, an audited host: not opened).
  4. about 20:47, "ClaudeBot ignored robots.txt complaint website operator Anthropic crawler traffic report": secondary reports of iFixit's 2024 complaint; none opened.
  5. about 20:47, "Cloudflare Perplexity stealth undeclared crawlers no-crawl directives blog": Cloudflare's post (opened, evidence-1/004).
  6. about 20:47, "Somesite I Used To Crawl AI crawlers robots.txt efficacy content creators IMC": the authors' PDF at sysnet.ucsd.edu (opened, evidence-1/007); arXiv and datatracker copies not opened (audited hosts).
  7. about 20:49, "Reddit lawsuit Anthropic bots accessed site more than 100,000 times complaint June 2025": Engadget's report (opened, evidence-1/009); others not opened.
  8. about 20:49, "Cloudflare crawl-to-referral ratio Anthropic OpenAI blog crawl to refer": Cloudflare's post of 29 August 2025 (opened, evidence-1/005).
  9. about 20:49, "TollBit State of the Bots H1 2026 report ChatGPT-User 54% Europe robots.txt disallowed": confirmed the figure is in TollBit's own report (already fetched).
  10. about 20:50, a query for The Verge's iFixit and Reddit articles: no Verge article found; one tried by URL was not sent (its robots.txt disallows anthropic-ai).
  11. about 20:50, "Scrapers selectively respect robots.txt directives IMC 2025 ACM pdf Wenger Duke": the ACM DOI (requested; answered with a challenge, evidence-1/008).

Part A also read, from operators found without a search: Google's robots meta tag, common crawler and user-triggered fetcher pages, Anthropic's, OpenAI's and Perplexity's bot pages, llmstxt.org, W3C's TDMRep report, Cloudflare's pay-per-crawl page, ai.robots.txt through jsDelivr, PyPI's JSON API documentation (docs.pypi.org), and RFCs 9309, 6585, 9110 and 9421.

From studies/LH047/searchlog.md in the repository, last changed 8 November 2026.