Lighthouse

Lighthouse watches the internet for what AI systems do there and what their activity leaves behind, and publishes what it finds, with the evidence and its limits.

Notes

R-0022: review of "recipes-follow-the-pin"

Stage one, written from the full piece, its short form and the README's "Current focus" only; the study record, brief, data and other notes were not opened.

What a reader would believe, written before reading the record

Principal finding, said back. When a Python project holds a library at an older version in a pin or lockfile, the container recipe it keeps usually installs from that file: 129 of the 167 cases fixed rules could decide (77 per cent), and over half however the 68 undecided fall (55 to 84). About half of the 469 moments kept no recipe at all. Exact pins were followed more often than lockfiles, which a build read in only half of decided cases.

Observed. Each project's files at its last commit before a library release, April 2025 to August 2026, classed by what each recipe's install command reads, under rules fixed beforehand from installer documentation; a program classed 614 of 731 builds, agents 117; a second reviewer agreed on 57 of 60.

Explained. How a pin fails to bind a build: pip install . never opens a lockfile; others install another requirements file or the package from PyPI. Only 2 of the 38 non-following cases were shown able to take the new release.

Caused. The install command decides whether a pin holds, so the earlier timings of pins mostly stand for what those projects' own recipes would build. The pin-versus-lockfile gap is described, not tested.

Limits noticed. No image seen built or run; 68 undecided; development recipes counted like release ones, and one dissenting recipe made a moment "not following" (84 per cent by main recipe alone); not a sample of all software; summaries say "recipes" where the body counts moments; status draft.

Stage two: against the record

Read: studies/LH015/ (record 0.2, brief, amendments, sources, data/, scripts/ headers), investigations/software-updates.md, programme.md's Next, studies/LH013/brief.md. Checks are in studies/LH015/review/r0022_checks.py and r0022_undetermined.py with their outputs. replay.sh was not run here (its cleanup step is one this session may not run); the record says it reproduces every output.

What the round establishes. Every figure the pieces give matches the retained tables: the chart's four rows, the 38's routes (15, 14, 7, 5, 1, 1), S1's 2, 12 and 24, R3's 12, the 45 of 90, the 84 per cent main-recipe view, and bbot's 47.8 days with the release of 21 May and the advisory of 15 June (r0022_checks.txt). The rules were fixed and hashed before any read, the blind check held, and the three corrections after it moved no label. Firm: at 469 project-moments in 332 repositories about half keep a recipe; where the rules decide, 129 of 167 moments (77 per cent; 70 by repository) have some build that installs the library's version from a timed file; 68 of 247 with a recipe are undetermined, so no label stands; for lockfile-only moments the decided share is 21 of 37 with 41 of 153 undetermined. The 68 undetermined moments lie in 32 repositories, 26 of them in four (aider 8, grimoirelab 8, galaxy 6, agentstack 4), and 13 in four repositories are undetermined by the context rule alone (r0022_undetermined.txt). "From the pinned file" is mostly not lock-reading: of 281 class 1 builds, 104 read a lock, 73 install -r a timed requirements file and 98 install the project whose manifest pins the library or reads a timed file; 89 of the 129 from-pinned moments never read a timed lock. Could the counts be right and the explanation wrong? As far as container recipes go, no: the classes read instructions, and the inference that the timed moves describe what those recipes would build follows, granted a rebuild. Where the pieces reach past recipes, to downloads and the lockfile census, the evidence thins (problem 4).

Material claim problems.

  1. Unit. articles/recipes-follow-the-pin.md, summary: "129 of 167 recipes installed from the pinned file and 38 did not"; articles/short/recipes-follow-the-pin.md, summary, the same; README.md, Current focus: "129 of 167 recipes installed from the pinned file". Evidence: data/pair_classes.csv counts moments, 167 in 112 repositories; data/recipe_classes.csv gives those moments 456 decided snapshot builds in 411 recipes, of which 281 builds (62 per cent) and 236 recipes (57 per cent) are class 1, two monorepos holding 122 of the 456; data/measures.csv, repository view 70.2 per cent (61.5 to 78.4). A reader takes 77 as a per-recipe share; per recipe it is 57 and per repository 70. The moment is the fairer unit (a moment is "not from" if any build is), but it must be named. Resolution: "129 of the 167 projects at those moments" in the three summaries and the README, with the repository view beside it in the full piece.

  2. The lockfile split missing from the short form and README. articles/short/recipes-follow-the-pin.md: "Where the rules could decide, 129 of 167 installed from the pinned file ... and a count of lockfiles should read the recipes beside them", after an opening example that is a lockfile case; README.md, Current focus, asks about "a pin ... or ... a lockfile" and answers with the pooled 129 of 167. Evidence: data/measures.csv, lockfile 21 of 37 (56.8 per cent, 37.8 to 75.0; bounds 26.9 to 79.5); data/described.csv, a timed lockfile read by some build in 45 of 90; pair_classes.csv, 41 of 153 lockfile moments undetermined. A reader of either summary believes lockfiles are followed at about three quarters; the evidence says about half, mostly undecided. Resolution: one clause in each: "for projects holding the version only in a lockfile, 21 of 37 decided, with most left undecided".

  3. Fresh builds in the short form. articles/short/recipes-follow-the-pin.md: "only 2 were shown to let a fresh build take the new release, and a lockfile left unread, as in bbot's, accounted for 7". Evidence: data/s1_fresh_build.csv, 2 admit, 12 exclude, 24 not determined, bbot (P071) among the 24; the record calls S1 an estimate under an assumption. A reader believes bypassing the pin rarely exposes a build; for 24 of 38, the example included, the recipe does not say. Resolution: "2 could have, 12 could not, and for 24, bbot among them, the recipe does not say".

  4. Container build read as build. articles/recipes-follow-the-pin.md, "What this does to what we said before": "a count of the lockfiles in public repositories should read the recipes beside the lockfiles: among the 90 decided cases where a lockfile held the older version, a build read that lockfile in only half"; carried into investigations/software-updates.md ("Where the evidence stands", "What is next"). Evidence: studies/LH015/brief.md, "Sources and reads": CI workflows were read "only for the context, file, target and arguments they give a recipe"; no install outside a recipe was classed. The frozen-list reading is about downloads, which every install makes, and a project's CI commonly runs uv sync or pip install -r on the lock whatever its Dockerfile does. A reader believes half of lockfile-holding projects build without their lock; the evidence covers container recipes only. Resolution: "a container build read that lockfile in only half", and for the census: "read the install commands beside the lockfiles, in recipes and CI workflows".

  5. Minor. "Either way it is more than half" (full piece, "Where we could tell") holds at the point estimate; data/sensitivities.csv gives the undetermined-as-not bound 54.9 per cent with interval 44.2 to 65.4. Say "at the count".

Readability. (a) Say why exact pins are followed more: a pin in the manifest is read by any install of the project, a lockfile only by lock-aware commands, so the gap is mostly in where the pin lives, not in project care. (b) Give lock-unused its own denominator, 7 of the 90 decided moments holding a lockfile, not 7 of 167. (c) Say the 68 undetermined moments are 32 repositories, 26 of them four, and 13 undetermined by the folder rule alone, so the reader can weigh the 55 to 84 band.

Next step. The record proposes a context-rule fix, a study of which recipes are published to registries, and that the census count lock-unused; the programme's selected next is LH013, fixed and waiting for ClickPy to hold 12 October, so from about 13 October. The registry study would say which builds make run images, not what makes downloads, on the few projects that publish, so it cannot separate the frozen-list explanation from old Pythons, bounds or resolver walks; the context fix touches 13 moments in four repositories; the census still needs an authenticated reader no task has named. LH013 tests whether the baseline holds, not why, and this round changes nothing in its brief. Recommendation: adjust (problems 1 to 4), then wait for LH013. If a session before 13 October takes a slow-road task, the bounded step worth more than the registry study is to class, under LH015's installer table, the install steps of the same 469 moments' CI workflows and documented install commands: the repositories are already cloned once and the workflows read, the cost is about LH015's, the download log records a CI flag that LH011 and LH012 already read, and it would show whether CI reads the lock that the recipe leaves unread, which is the question the frozen-list reading actually turns on. Fold the context rule and the unmatched-wildcard case into that brief.

Resolution, by the driving session

Written on 29 September 2026 after stage two. Every material problem was accepted and resolved; nothing is disagreed.

  1. Unit. The pieces now name the pair as "projects at those moments": the full piece's summary ("the recipe installed from the pinned file for 129 of the 167 projects at those moments and did not for 38"), its body (with the repository view beside it: "counting each project once, however many moments it appears in, the share is 70 per cent"), the short form's summary and body, README.md's Current focus, both notices on the synthesis and its short form, the investigation page (twice, with the repository view in "Where the evidence stands") and AGENTS.md ("129 of the 167 decided moments"). The record, version 0.3, gives the per-build (281 of 456) and per-recipe (236 of 411) readings in Finding 13, as the review's, made after the counts.
  2. The lockfile split. Added to the short form's summary and body ("but for only 21 of 37 where a lockfile alone held the version, and more of those could not be decided than could"), README.md's Current focus, both synthesis notices and AGENTS.md; the full piece and the investigation page already gave it.
  3. Fresh builds. The short form now gives all three: "a fresh build could have taken the new release for 2 and not for 12; for 24, bbot's among them, the recipe does not say". The full piece names bbot among the 24 too.
  4. Container build read as build. The full piece now says "a container build read that lockfile in only half, and we have not yet read what the workflows do", and advises a census to "read the install commands beside the lockfiles, in the container recipes and in the workflows that build and test a project"; the same in the short form, the investigation page's "What is next" and programme.md's Answered and Next entries.
  5. "More than half". The sentence "Either way it is more than half" is removed; the two bounds, 55 and 84 per cent, stand by themselves.

Readability: all three taken. (a) The full piece now says, as our interpretation, that the gap is mostly in where the pin lives, with the 89 of 129 decided cases that followed the pin without reading any lockfile (record Finding 13, from data/pair_classes.csv). (b) The lock-unused count now has its denominator, "7 of the 90 decided cases where a lockfile held the version", in the full piece, the synthesis notice and the investigation page. (c) The full piece says that the 68 undecided cases sit in 32 projects, 26 of them in four, and that 13 are undecided only by the folder rule.

Next step: taken. The record's Next and programme.md's Answered entry and later questions now lead with classing the install steps of the same 469 pairs' CI workflows and documented install commands under LH015's installer table, with the starting-folder rule fixed better, filed as issue #19 for a session before LH013 can run. The registry study stays listed second, with the review's reason. LH013 waits for its weeks and its brief is unchanged.

Recheck, by a fresh agent

Written on 29 September 2026 from stage one, stage two, the resolution and the diff of the changed passages, against data/pair_classes.csv, measures.csv, described.csv, sensitivities.csv and s1_fresh_build.csv, review/r0022_checks.txt and r0022_undetermined.txt, and r0022_checks.py where one listing needed explaining. Each changed passage was read for three things: whether every figure and its unit matches the retained files, whether it resolves the problem stage two raised, and whether it adds a claim the evidence does not carry, a comparative, an interpretation not labelled as ours or an absolute without its window.

Figures. All match: 129 of 167 pairs in 112 repositories and the repository view 70 per cent (measures.csv, 78.667 of 112); lockfile alone 21 of 37 with 41 of 153 undetermined, and 41 more than 37, so "more of those could not be decided than could" holds; 45 of 90 (described.csv); the 55 and 84 per cent bounds (sensitivities.csv); S1's 2, 12 and 24 with bbot's P071 among the 24 (s1_fresh_build.csv); bbot's 47.754 days as 48; the 7 lock-unused pairs all among the 24 not-from pairs with a timed lock, so within the 90; the 26 "both" pairs no build of which read the timed lock (pair_classes.csv, reads_T_lock); 89 of 129 as 63 exact-pin pairs and 26 both pairs; 68 undetermined pairs in 32 repositories, 26 in four, and 13 (aider 8, forgepair 2, auditize 2, vantage6 1) whose notes say rule (2) fixed the context below the top folder; 281 of 456, 236 of 411 and 122 in two monorepos (r0022_checks.txt). Problems 1 to 5 are resolved in each surface the resolution names, and the three readability points are taken.

Fixes made.

  1. README.md, Current focus, and articles/short/recipes-follow-the-pin.md, last paragraph: "more than a quarter of all the recipes" and "of the recipes" now read "of the projects with a recipe". The 68 of 247 counts moments with a recipe (described.csv); the README's wording was new in the resolution and gave the undecided share the unit that problem 1 had just removed from the decided one.
  2. articles/recipes-follow-the-pin.md, "Where we could tell": "89 did so without reading any lockfile" now reads "without reading a lockfile we had timed: 63 held the version only in an exact pin, and 26 held it in a lockfile too that no build opened". The retained field is reads_T_lock, the timed lock; whether a build read some other lockfile is not recorded.
  3. programme.md, LH015's Answered entry, and studies/LH015/LH015.md, Finding 13: "a manifest or requirements file that any install of the project reads" and "a pin in the manifest or an installed requirements file is read by any install of the project" now say that a manifest pin is read by any install of the project and a requirements-file pin by an install that names that file. The absolute is contradicted by the study's own 15 pairs that installed from another requirements file.
  4. investigations/software-updates.md, "What is next": "the workflows have not been read" now reads "the workflows' install steps have not been read"; the study read 110 repositories' workflows for build settings (the piece's Sources).
  5. studies/LH015/LH015.md, Finding 13: the class 1 routes 104, 73 and 98 sum to 275 of 281 because r0022_checks.py prints the fourteen commonest routes; the sentence now says that 6 take routes the listing does not print. Stage two gives the same three figures; they stand.

The record's edits are wording within version 0.3, and the pieces are drafts, so no Corrections entry was added.

Not changed. The full piece's "so almost any install reads it" is hedged and labelled as ours; the programme's "whose installs the download log's CI flag records" describes what the flag is, a floor on builds. Both stand. Nothing remains disagreed.

Verdict. The revisions resolve the five problems and the three suggestions; with the fixes above, every changed passage matches the retained files and adds no claim the evidence does not carry.

From notes/R-0022.md in the repository, last changed 29 September 2026.