LH012 sources
What was read for LH012, when, and what each source is. All reads were anonymous: no account, key or credential was used or sent. Times are UTC on 29 September 2026; data/read_log.csv holds every logged read's time, endpoint, query text, HTTP status, rows returned and, for ClickPy, the rows and bytes read as ClickHouse reported them. This file is started in phase 1 (before the brief was fixed) and is completed in phase 2. Repository sources are cited by their number in the repository's sources.md.
| id | source | publisher; publication date or window | read (UTC) | retained as |
|---|---|---|---|---|
| K1 | ClickPy public SQL, https://sql-clickhouse.clickhouse.com/?user=demo (repository S21) | ClickHouse; a copy of PyPI's download table (S20); server 26.10.1.39301, time zone UTC; last day held 2026-09-28 for the top 500 | 06:19:15 to 06:19:22 (phase 1: 9 queries, none returning a download count) | data/clickpy_server.csv, clickpy_schema.csv, clickpy_columns.csv, clickpy_coverage.csv, clickpy_last_day_by_table.csv |
| K2 | PyPI JSON API, https://pypi.org/pypi/{project}/json and /{project}/{version}/json | Python Package Index; files, Requires-Python, upload times and declared requirements as listed at the read | 06:19:49 to 06:23:15 (500 project reads, 128 version reads) | data/versions.csv, files.csv.gz, dependent_versions.csv, top500_screen.csv, requirements.csv, week_versions.csv |
| K3 | Source distributions of pip 26.2.1 and uv 0.12.8 from files.pythonhosted.org (the versions LH011 read, its K3) | the pip and uv projects via PyPI | 06:26:17 | not retained (scratch only); the lines relied on are described below |
| K4 | studies/LH010/data/top500.csv | ClickPy's August 2026 totals, read by LH010 on 27 September 2026 at 09:53:19 | reused, not re-read | (LH010's file) |
| K5 | studies/LH011/data/releases.csv and LH011's retained counts; studies/LH008/data/lags.csv and events.csv | LH011, read 28 September 2026; LH008 | reused offline | (their files) |
| D01 | Python Packaging User Guide, "Analyzing PyPI package downloads", https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/ | PyPA; page undated as read | 06:26:13 | scratch only |
| D02 | linehaul-cloud-function README, https://raw.githubusercontent.com/pypi/linehaul-cloud-function/main/README.md | PyPI; main branch at the read | 06:26:14 | scratch only |
| D03 | linehaul-cloud-function user-agent parser, https://raw.githubusercontent.com/pypi/linehaul-cloud-function/main/linehaul/ua/parser.py | PyPI; main branch at the read | 06:26:14 | scratch only |
| D04 | https://raw.githubusercontent.com/pypi/linehaul-cloud-function/main/linehaul/schema.json | guessed path | 06:26:14, HTTP 404, not read | |
| D05 | ClickPy README, https://raw.githubusercontent.com/ClickHouse/clickpy/main/README.md | ClickHouse; main branch at the read | 06:26:15 | scratch only |
| D06 | pip documentation, "Repeatable Installs", https://pip.pypa.io/en/stable/topics/repeatable-installs/ | pip 26.2.1 documentation | 06:27:44 | scratch only |
| L01 | SPEC 0, "Minimum Supported Dependencies", https://scientific-python.org/specs/spec-0000/ | Scientific Python | 06:26:15 | scratch only |
| L02 | NEP 29, https://numpy.org/neps/nep-0029-deprecation_policy.html | NumPy; created 2019-07-13, status Final, superseded by SPEC 0 | 06:26:15 | scratch only |
| L03 | Decan, Mens and Constantinou, "On the evolution of technical lag in the npm package dependency network", ICSME 2018, https://decan.lexpage.net/files/ICSME-2018.pdf | 2018; npm | 06:26:16 (abstract and text searched) | scratch only |
| L04 | Stringer, Tahir, Blincoe and Dietrich, "Technical Lag of Dependencies in Major Package Managers", APSEC 2020, https://kblincoe.github.io/publications/2020_APSEC_tech_lag.pdf | 2020; libraries.io, 14 package managers | 06:26:17 (abstract and text searched) | scratch only |
| L05 | Hugo van Kemenade, "Python version share over time, 6", https://hugovk.dev/blog/2020/python-version-share-over-time-6/ | 2020 blog post; pip downloads January 2016 onwards, via BigQuery | 06:27:43 | scratch only |
Two web searches were made between 06:26 and 06:27 through the harness's search tool (not logged in data/read_log.csv): "PyPI download logs older package versions share Python version empirical study" and ""technical lag" PyPI downloads deployed versions pinned requirements empirical". Opened from their results: L04, L05 and D06. Not opened: https://arxiv.org/pdf/2608.12853 (bytecode security), https://arxiv.org/pdf/2603.07919 (social proof and downloads), https://arxiv.org/pdf/2408.10327 (package-level deprecation in Python), https://arxiv.org/pdf/1907.11073 (an empirical analysis of PyPI; LH011's check listed it from a summary), the medium.com and dev.to copies of van Kemenade's series, https://arxiv.org/pdf/1806.01545 (the arXiv copy of L03), a scribd copy of an npm technical-lag paper, three github.com issue pages (pip issue 9284 and two unrelated repositories; github.com's issue pages return HTTP 403 here) and two pypi.org project pages for a freeze-requirements tool. pypdf 6.19.0 was installed from PyPI into scratch space between 06:19 and 06:23 to read L03 and L04; that download was not logged.
What each source is, and its limits, as far as the brief relies on it
- K1, ClickPy. A copy, not the original. The tables the brief reads are materialised views of
pypi.pypi(data/clickpy_schema.csv):pypi_downloads_per_day_by_version,..._by_python(python_minor),..._by_system,..._by_installer_by_type;pypi.pypicarriesci,libc(lib, version),typeandfilenamebesides. The staging tablepypi_rawholdspythonas a full string anddistro.libc; the step from it topypi.pypiis not in the catalogue. ClickPy's own README (D05) documents a backfill import that setspython_minorto the first two dot-separated parts ofpythonandsystemtosystem.name, and keeps only rows where both are non-empty; whether the live ingestion does the same was not checked. - K2, PyPI JSON. The versions and files listed at the read; a deleted version is absent,
yankedis the state at the read, and a file'srequires_pythonis the metadata the installer reads. Declared requirements (requires_dist) are the version's own metadata; for a version read through the project JSON they are the latest version's. - K3, the installers' source. pip 26.2.1 (
src/pip/_internal/network/session.py,user_agent()):pythonisplatform.python_version(); on Linux,distro.libcis{"lib": "glibc", "version": ...}when a glibc version is found and absent otherwise (utils/glibc.py,libc_ver()), so pip never reports musl;system.nameisplatform.system(). uv 0.12.8 (crates/uv-client/src/linehaul.rs):pythonis the full Python version of the PEP 508 marker environment passed to it, or null when none is;libcis glibc or musl with its version, from the platform;system.nameis the markers'platform_system. Which interpreter's markers each uv command passes was not read. - D03, linehaul's parser. pip 6 and later and uv send a JSON user agent that linehaul parses whole; older clients' user agents are parsed by pattern and many carry no Python version.
- L01, L02. SPEC 0 recommends dropping support for Python versions three years after their initial release and for core package dependencies two years after theirs; NEP 29 recommended supporting the Python minors released in the prior 42 months. Policies, context only.
- L03, L04, L05. Measure declared constraints (L03, L04) or the Python field of pip downloads across all packages (L05); none measures which versions of a library are fetched.
- Not read. BigQuery itself (S20); linehaul's BigQuery schema (D04 returned 404); ClickPy's live ingestion; github.com's pages for aiobotocore's issues on its botocore bound (programme.md, Next), which returned HTTP 403 to the previous check and were not tried again.
Added after the snapshot: phase 2 reads (29 September 2026)
This section was added after the brief's snapshot on issue #14 (06:35 UTC); the text above it is as snapshotted. Phase 2's reads are logged in data/read_log_phase2.csv (amendment 1), all anonymous, all HTTP 200 at the transport level.
| id | source | what was read | read (UTC) | retained as |
|---|---|---|---|---|
| K1 (phase 2) | ClickPy public SQL, as above | 425 queries, 9,955,627,613 rows read by ClickHouse's statistics: per project, downloads by day and version over W (pypi_downloads_per_day_by_version), W sums by version and Python minor (..._by_python), one grouped pass of pypi.pypi (class, installer, ci, system, libc library, type) and, for ten projects, older downloads by Python minor and file tags; the dependents' downloads by day and version over W; W sums by version and Python minor for 60 dependents with a Python marker; the 37 projects' daily totals for 1 October 2025 to 31 May 2026 and 14 to 28 September 2026; per-version downloads on days 1, 2 and 30 of each November to March release; and checks C1 to C3 |
06:40:36 to 07:13:23 | data/week/, data/deps/, data/firstday/, data/checks/ |
| K2 (phase 2) | PyPI JSON API, https://pypi.org/pypi/{project}/{version}/json | 3,844 single-version reads: 87 versions at or above R of the 37 projects, and 3,757 dependent versions by the coverage rule | 06:49:51 to 07:09:28 | data/dependent_reads.csv, data/requirements_phase2.csv |
ClickHouse answered 15 or so of these queries with HTTP 200 and then streamed an error into the body (a result over 10,000 rows); they are in the log with the rows they read, and the collector discarded and split them (amendment 2). No documentation page was read in phase 2.
Added for version 0.4 (29 September 2026, late evening)
Anonymous reads, none of them a download count, made to check when pip stops fetching each version it tries from PyPI (the record's walk paragraph; notes/R-0025.md, problem 2).
| id | source | what was read | read (UTC) | retained as |
|---|---|---|---|---|
| L06 | PEP 714, https://peps.python.org/pep-0714/ (created 6 June 2023, accepted 27 June 2023) | that PyPI supported PEP 658 only from shortly before June 2023 ("until just recently"), that pips from at least 22.3 fail on the old key in the JSON index, and that the JSON index must use core-metadata | 21:43 (the reader review) and 21:49:16 (the driving session), 29 September 2026 | review/r0025_pip_metadata_sources.txt; review/v04_pip_metadata_read.txt |
| L07 | pip's NEWS.rst, https://raw.githubusercontent.com/pypa/pip/main/NEWS.rst (main at the read) | the 22.3 entry (use of data-dist-info-metadata) and the 23.2 entries (parsing it from JSON; no second download) | 21:43 and 21:49:27 | as L06 |
| L08 | the pip 23.1.2 and 23.2 wheels from files.pythonhosted.org, by the URLs PyPI's JSON API gave | pip/_internal/models/link.py (which metadata keys each reads) and 23.1.2's pip/_internal/index/collector.py (which index format it asks for first) | 21:50:05 | review/v04_pip_metadata_read.txt |
From studies/LH012/sources.md in the repository, last changed 29 September 2026.