Lighthouse

Lighthouse watches the internet for what AI systems do there and what their activity leaves behind, and publishes what it finds, with the evidence and its limits.

Studies LH012

LH012 sources

What was read for LH012, when, and what each source is. All reads were anonymous: no account, key or credential was used or sent. Times are UTC on 29 September 2026; data/read_log.csv holds every logged read's time, endpoint, query text, HTTP status, rows returned and, for ClickPy, the rows and bytes read as ClickHouse reported them. This file is started in phase 1 (before the brief was fixed) and is completed in phase 2. Repository sources are cited by their number in the repository's sources.md.

id source publisher; publication date or window read (UTC) retained as
K1 ClickPy public SQL, https://sql-clickhouse.clickhouse.com/?user=demo (repository S21) ClickHouse; a copy of PyPI's download table (S20); server 26.10.1.39301, time zone UTC; last day held 2026-09-28 for the top 500 06:19:15 to 06:19:22 (phase 1: 9 queries, none returning a download count) data/clickpy_server.csv, clickpy_schema.csv, clickpy_columns.csv, clickpy_coverage.csv, clickpy_last_day_by_table.csv
K2 PyPI JSON API, https://pypi.org/pypi/{project}/json and /{project}/{version}/json Python Package Index; files, Requires-Python, upload times and declared requirements as listed at the read 06:19:49 to 06:23:15 (500 project reads, 128 version reads) data/versions.csv, files.csv.gz, dependent_versions.csv, top500_screen.csv, requirements.csv, week_versions.csv
K3 Source distributions of pip 26.2.1 and uv 0.12.8 from files.pythonhosted.org (the versions LH011 read, its K3) the pip and uv projects via PyPI 06:26:17 not retained (scratch only); the lines relied on are described below
K4 studies/LH010/data/top500.csv ClickPy's August 2026 totals, read by LH010 on 27 September 2026 at 09:53:19 reused, not re-read (LH010's file)
K5 studies/LH011/data/releases.csv and LH011's retained counts; studies/LH008/data/lags.csv and events.csv LH011, read 28 September 2026; LH008 reused offline (their files)
D01 Python Packaging User Guide, "Analyzing PyPI package downloads", https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/ PyPA; page undated as read 06:26:13 scratch only
D02 linehaul-cloud-function README, https://raw.githubusercontent.com/pypi/linehaul-cloud-function/main/README.md PyPI; main branch at the read 06:26:14 scratch only
D03 linehaul-cloud-function user-agent parser, https://raw.githubusercontent.com/pypi/linehaul-cloud-function/main/linehaul/ua/parser.py PyPI; main branch at the read 06:26:14 scratch only
D04 https://raw.githubusercontent.com/pypi/linehaul-cloud-function/main/linehaul/schema.json guessed path 06:26:14, HTTP 404, not read
D05 ClickPy README, https://raw.githubusercontent.com/ClickHouse/clickpy/main/README.md ClickHouse; main branch at the read 06:26:15 scratch only
D06 pip documentation, "Repeatable Installs", https://pip.pypa.io/en/stable/topics/repeatable-installs/ pip 26.2.1 documentation 06:27:44 scratch only
L01 SPEC 0, "Minimum Supported Dependencies", https://scientific-python.org/specs/spec-0000/ Scientific Python 06:26:15 scratch only
L02 NEP 29, https://numpy.org/neps/nep-0029-deprecation_policy.html NumPy; created 2019-07-13, status Final, superseded by SPEC 0 06:26:15 scratch only
L03 Decan, Mens and Constantinou, "On the evolution of technical lag in the npm package dependency network", ICSME 2018, https://decan.lexpage.net/files/ICSME-2018.pdf 2018; npm 06:26:16 (abstract and text searched) scratch only
L04 Stringer, Tahir, Blincoe and Dietrich, "Technical Lag of Dependencies in Major Package Managers", APSEC 2020, https://kblincoe.github.io/publications/2020_APSEC_tech_lag.pdf 2020; libraries.io, 14 package managers 06:26:17 (abstract and text searched) scratch only
L05 Hugo van Kemenade, "Python version share over time, 6", https://hugovk.dev/blog/2020/python-version-share-over-time-6/ 2020 blog post; pip downloads January 2016 onwards, via BigQuery 06:27:43 scratch only

Two web searches were made between 06:26 and 06:27 through the harness's search tool (not logged in data/read_log.csv): "PyPI download logs older package versions share Python version empirical study" and ""technical lag" PyPI downloads deployed versions pinned requirements empirical". Opened from their results: L04, L05 and D06. Not opened: https://arxiv.org/pdf/2608.12853 (bytecode security), https://arxiv.org/pdf/2603.07919 (social proof and downloads), https://arxiv.org/pdf/2408.10327 (package-level deprecation in Python), https://arxiv.org/pdf/1907.11073 (an empirical analysis of PyPI; LH011's check listed it from a summary), the medium.com and dev.to copies of van Kemenade's series, https://arxiv.org/pdf/1806.01545 (the arXiv copy of L03), a scribd copy of an npm technical-lag paper, three github.com issue pages (pip issue 9284 and two unrelated repositories; github.com's issue pages return HTTP 403 here) and two pypi.org project pages for a freeze-requirements tool. pypdf 6.19.0 was installed from PyPI into scratch space between 06:19 and 06:23 to read L03 and L04; that download was not logged.

What each source is, and its limits, as far as the brief relies on it

  • K1, ClickPy. A copy, not the original. The tables the brief reads are materialised views of pypi.pypi (data/clickpy_schema.csv): pypi_downloads_per_day_by_version, ..._by_python (python_minor), ..._by_system, ..._by_installer_by_type; pypi.pypi carries ci, libc (lib, version), type and filename besides. The staging table pypi_raw holds python as a full string and distro.libc; the step from it to pypi.pypi is not in the catalogue. ClickPy's own README (D05) documents a backfill import that sets python_minor to the first two dot-separated parts of python and system to system.name, and keeps only rows where both are non-empty; whether the live ingestion does the same was not checked.
  • K2, PyPI JSON. The versions and files listed at the read; a deleted version is absent, yanked is the state at the read, and a file's requires_python is the metadata the installer reads. Declared requirements (requires_dist) are the version's own metadata; for a version read through the project JSON they are the latest version's.
  • K3, the installers' source. pip 26.2.1 (src/pip/_internal/network/session.py, user_agent()): python is platform.python_version(); on Linux, distro.libc is {"lib": "glibc", "version": ...} when a glibc version is found and absent otherwise (utils/glibc.py, libc_ver()), so pip never reports musl; system.name is platform.system(). uv 0.12.8 (crates/uv-client/src/linehaul.rs): python is the full Python version of the PEP 508 marker environment passed to it, or null when none is; libc is glibc or musl with its version, from the platform; system.name is the markers' platform_system. Which interpreter's markers each uv command passes was not read.
  • D03, linehaul's parser. pip 6 and later and uv send a JSON user agent that linehaul parses whole; older clients' user agents are parsed by pattern and many carry no Python version.
  • L01, L02. SPEC 0 recommends dropping support for Python versions three years after their initial release and for core package dependencies two years after theirs; NEP 29 recommended supporting the Python minors released in the prior 42 months. Policies, context only.
  • L03, L04, L05. Measure declared constraints (L03, L04) or the Python field of pip downloads across all packages (L05); none measures which versions of a library are fetched.
  • Not read. BigQuery itself (S20); linehaul's BigQuery schema (D04 returned 404); ClickPy's live ingestion; github.com's pages for aiobotocore's issues on its botocore bound (programme.md, Next), which returned HTTP 403 to the previous check and were not tried again.

Added after the snapshot: phase 2 reads (29 September 2026)

This section was added after the brief's snapshot on issue #14 (06:35 UTC); the text above it is as snapshotted. Phase 2's reads are logged in data/read_log_phase2.csv (amendment 1), all anonymous, all HTTP 200 at the transport level.

id source what was read read (UTC) retained as
K1 (phase 2) ClickPy public SQL, as above 425 queries, 9,955,627,613 rows read by ClickHouse's statistics: per project, downloads by day and version over W (pypi_downloads_per_day_by_version), W sums by version and Python minor (..._by_python), one grouped pass of pypi.pypi (class, installer, ci, system, libc library, type) and, for ten projects, older downloads by Python minor and file tags; the dependents' downloads by day and version over W; W sums by version and Python minor for 60 dependents with a Python marker; the 37 projects' daily totals for 1 October 2025 to 31 May 2026 and 14 to 28 September 2026; per-version downloads on days 1, 2 and 30 of each November to March release; and checks C1 to C3 06:40:36 to 07:13:23 data/week/, data/deps/, data/firstday/, data/checks/
K2 (phase 2) PyPI JSON API, https://pypi.org/pypi/{project}/{version}/json 3,844 single-version reads: 87 versions at or above R of the 37 projects, and 3,757 dependent versions by the coverage rule 06:49:51 to 07:09:28 data/dependent_reads.csv, data/requirements_phase2.csv

ClickHouse answered 15 or so of these queries with HTTP 200 and then streamed an error into the body (a result over 10,000 rows); they are in the log with the rows they read, and the collector discarded and split them (amendment 2). No documentation page was read in phase 2.

Added for version 0.4 (29 September 2026, late evening)

Anonymous reads, none of them a download count, made to check when pip stops fetching each version it tries from PyPI (the record's walk paragraph; notes/R-0025.md, problem 2).

id source what was read read (UTC) retained as
L06 PEP 714, https://peps.python.org/pep-0714/ (created 6 June 2023, accepted 27 June 2023) that PyPI supported PEP 658 only from shortly before June 2023 ("until just recently"), that pips from at least 22.3 fail on the old key in the JSON index, and that the JSON index must use core-metadata 21:43 (the reader review) and 21:49:16 (the driving session), 29 September 2026 review/r0025_pip_metadata_sources.txt; review/v04_pip_metadata_read.txt
L07 pip's NEWS.rst, https://raw.githubusercontent.com/pypa/pip/main/NEWS.rst (main at the read) the 22.3 entry (use of data-dist-info-metadata) and the 23.2 entries (parsing it from JSON; no second download) 21:43 and 21:49:27 as L06
L08 the pip 23.1.2 and 23.2 wheels from files.pythonhosted.org, by the URLs PyPI's JSON API gave pip/_internal/models/link.py (which metadata keys each reads) and 23.1.2's pip/_internal/index/collector.py (which index format it asks for first) 21:50:05 review/v04_pip_metadata_read.txt

From studies/LH012/sources.md in the repository, last changed 29 September 2026.