R-0026: release review of LH017 (blind check and evidence review)

Reviewer: a subagent on Fable 5.1 (claude-fable-5-1), 30 September 2026, in the repository at commit 94fbf09 with the study's uncommitted files. Allowance about ten dollars at list rates. No key or token was read from the environment; no search touched harbour/.session/.

Written before opening data/job_classes.csv, data/judgements.csv, data/pair_classes.csv or data/pairs.csv, and before opening LH017.md, the pieces or any note.

What I read: AGENTS.md; studies/LH017/brief.md and amendments.md; studies/LH017/data/review_sample.csv (40 jobs: 20 script-decided, 20 judged, with their stored install-relevant lines); and, at the snapshot commits in the blobless clones, the files the brief's rules name for these jobs:

galaxyproject/galaxy: scripts/common_startup.sh, requirements.txt and lib/galaxy/dependencies/dev-requirements.txt at bbac30d66bac; tox.ini and lib/galaxy/dependencies/pinned-typecheck-requirements.txt at 907cbe3ee352.
emcek/dcspy: the tree and pyproject.toml at 746f63d8821a (no uv.lock in the tree).
mlflow/mlflow: the tree and requirements/test-requirements.txt at 5709d813e178.
phenobarbital/ai-parrot: the tree, the root pyproject.toml (workspace, [tool.uv] overrides) and packages/ai-parrot-tools/pyproject.toml and packages/ai-parrot/pyproject.toml at dd4c70085cf5 (no uv.lock in the tree).
asozialesnetzwerk/an-website: pyproject.toml, setup.py (install_requires read from pip-requirements.txt at every commit sampled), pip-requirements.txt and pip-constraints.txt at 7a5038611dad, e7c0708e149d, 1389088b9c4c, d0ff5122f7de and d58d85413d27; .github/workflows/deploy.yml at 1389088b9c4c and 7a5038611dad (triggers, needs, artefact names, the download-artifact expression); generate-badge.sh at 1389088b9c4c; scripts/build-zipapp.sh at d58d85413d27.
aider-ai/aider and tvick64889/forgepair: pyproject.toml (dependencies dynamic from requirements.txt) and requirements.txt at edfe0c801b49, bdb4d9ff8ef8, 5dc9490bb35f and 928bb49c4b5f.
chaoss/grimoirelab: .github/workflows/release.yml at d1f039bb596c (the build job is one external action).
mervinpraison/praisonai: the tree and .github/workflows/release.yml (triggers and inputs) at ffd78f0eb6d4.
talkinglibrary/talklib: pyproject.toml and requirements.txt at 6107a228a8d8. vida-nyu/bdi-kit: setup.py and requirements.txt at 1cb42ff0aa22. nucypher/nucypher: setup.py, pyproject.toml and requirements.txt at 547a9646d929. ethyca/fides: noxfile.py at c59d98bf48cc (grep for installs).
studies/LH015/brief.md, grepped for a rule on --no-deps (none).

My classes are in studies/LH017/review/r0026_blind.csv (order_hash, pair_id, job_key, class, sub_label, T_lock, reason). Counts: script-decided 20 = class 1 x 8, class 2 x 1, class 4 x 1, class 5 x 10; judged 20 = class 1 x 11, class 2 x 1, class 4 x 1, class 6 x 7.

Readings I had to make that the rules leave open, recorded here so the comparison can weigh them:

A manifest whose dependencies are read from a T file (setuptools dynamic in aider, forgepair and talklib; open('requirements.txt') in bdi-kit, nucypher and an-website's setup.py) takes L's version from that T file, so pip install . and a wheel or sdist built from that project in a needed job are class 1 with that T file deciding.
pip install --no-deps . (an-website build-source-tarball) installs no dependency, so no version of L comes from any file: class 2 (lock unused) by the letter of the sub-labels, not class 1.
pip install -c pip-constraints.txt <other packages> where pip-constraints.txt is not in T but pins L: class 4 (another pinned file), since the brief does not resolve whether L is installed.
A Python script the step runs is not read; where its command line names a T file (zipapps_patched.py -r pip-requirements.txt), it is a step the rules cannot decide that could have made the job class 1, so class 6 (script not read) replaces class 4.
uv sync with no uv.lock in the tree resolves the manifest afresh; where the manifest in T carries an exact pin (dcspy) or a workspace member's manifest in T does (ai-parrot), the job is class 1 from the exact pin, not "may re-lock", because no T lock applies.
An external action that builds or compiles the project (mypycify, grimoirelab's build action) whose installs cannot be read is class 6 (external) rather than class 5, because it could have installed from a T file.

Agreement and resolutions

Compared with data/job_classes.csv by studies/LH017/review/r0026_blind_compare.py (output in r0026_blind_compare.txt), which also recomputes every pair class from the job table with the brief's pair rule (0 mismatches in 469 pairs).

Script-decided: 17 of 20 agree on class (and on sub-label and T_lock). Judged: 19 of 20 agree on class, 17 of 20 on sub-label as well (two class 6 jobs where I gave a different reason, which the brief does not rank).
Per resulting pair class: 32 of the 33 sampled pairs keep their class and lock reading with my classes substituted; P327 (phenobarbital/ai-parrot) would move from "does not read the pin" to "reads the pin".

The disagreements, resolved by the brief's rules:

P163 galaxy, tool_form_harness.yaml test (script class 6 "script not read"; mine 1). common_startup.sh, the level read, runs ${PIP_CMD} install -r requirements.txt -r lib/galaxy/dependencies/dev-requirements.txt, and PIP_CMD is assigned once literally (python -m pip) and once from a command substitution ($(command -v uv) pip), so by the letter the deciding value is one the rules cannot substitute: class 6 stands, though its reason should be "build argument", not the depth-2 source it names. By the question's own test every branch reads T. Not counted as wrong; no pair effect.
P327 ai-parrot, ci.yml test-tools variant 2 (script class 2 "other file"; mine 1). uv sync --package ai-parrot-tools with no uv.lock in the tree resolves the workspace afresh; ai-parrot-tools depends on the workspace member ai-parrot ([tool.uv.sources] ai-parrot = { workspace = true }), whose manifest packages/ai-parrot/pyproject.toml is in T and pins urllib3==2.6.3 in [project] dependencies; the root's override-dependencies do not name urllib3. The script took the root pyproject.toml as the manifest and never followed --package to the member (ci.py treats --package only as a valued flag). The script is wrong. All 11 class 2 jobs of P327 are uv sync --package ai-parrot|ai-parrot-tools|ai-parrot-loaders (both members depend on ai-parrot), so the pair becomes "reads the pin". The brief's rules do not name --package, so under the stopping condition this is a case for the question's own test and a dated amendment.
P051 an-website, deploy.yml test-zipapp variant 2 (script class 5 "no Python install"; mine 4). The step is grep "^pytest-" pip-constraints.txt | xargs pip install -c pip-constraints.txt html5lib pytest time-machine; pip runs, so it is an install step, and pip-constraints.txt, not in T, pins pillow==12.1.1, so class 4, as the writer judged the same construct in github-pages (pip install -c pip-constraints.txt coverage, reading B3). ci.py's command splitter looks past sudo, env, xvfb-run and the like but not xargs, so it saw no installer. The script is wrong. 34 job rows carry an xargs installer (an-website at six snapshots and one homeassistant-ai/ha-mcp job); none changes a pair class, since every such pair reads the pin through another job.
P050 an-website, deploy.yml build-source-tarball (judged class 1, reading A2; mine 2 "lock unused"). pip install --no-deps . installs the project and no dependency, so no version of aiohttp is taken from any file; reading A2 (the manifest takes its dependencies from pip-requirements.txt) did not weigh --no-deps, which ci.py itself respects elsewhere (project(), no_deps). The writer is wrong; the same reading gives class 1 to the same job at three other an-website snapshots (P049, P051, P052; at P053 and P054 the script left it class 6). No pair class changes.

Sub-label differences only: P052 github-pages (both class 4; the writer gives no sub-label), P261 praisonai (both class 6; "external" against my "build argument", for a checkout of a fork named by a dispatch input).

Threshold verdict. Script-decided: 2 of 20 wrong after resolution, which is not more than 2, so the brief does not compel a script correction; judged: 1 of 20 wrong and it changes no pair class, so no re-judging. The check passes its threshold, at the edge for the script, and the two script errors are real defects (an unrecognised wrapper; an unfollowed workspace flag), one of which changes a pair's class. What follows from that is under Stage B.

Stage B: evidence review

What I checked

Replay. sh studies/LH017/replay.sh: 45 of 45 synthetic cases pass; every one of the 17 outputs is reproduced byte for byte; the brief's SHA-256 matches the one posted on issue #19 (fc17c66e...9cbf53); ceilings 332 clone attempts, 1.64 GB, 3,723 file reads, 30 GET requests (27 PyPI and 3 documentation attempts, which the replay's line labels together as "PyPI requests").
Figures in the record against the data. Every number in the Answer and in Findings 1 to 12 was checked against measures.csv, described.csv, sensitivities.csv, beside_lh015.csv, beside_lh015_shares.csv, s1 to s6, pypi.csv, pairs.csv, pair_classes.csv, job_classes.csv, jobs.csv, ci_files.csv, judgements.csv and frame.csv (studies/LH017/review/r0026_checks.py and the session's ad hoc counts): all hold, including the class counts (1,961 / 160 / 109 / 14 / 3,092 / 779 of 6,115), the mixed breakdown, the lock-reading installers (88 uv, 28 Poetry, 14 pip, 13 pip and uv, 8 Pipenv, 2 pip and Poetry, 1 PDM, 10 judged), S2 (236 / 30 / 16; 124 / 158), S3 (271 / 11; 10 GitLab), S5 (313 / 163 / 198), the PyPI readings (15 / 2 / 9 / 2), the judged readings (A1 to A7, B1 to B8: 164 class 1 manifests, 18 snowflake-cli, 10 data-safe-haven, 14 class 4 pip-constraints, 32 grimoirelab, 4 brownie, 4 praisonai) and the one-fifth rule per frame (14.1, 23.0, 13.4, 2.1 and 17.9 per cent). The primary shares recomputed from pairs.csv agree, and my own bootstrap (a separate implementation, 4,000 repository resamples, seed 26) gives 79.8 to 88.2, 93.5 to 98.6 and 80.1 to 93.4 against the record's 79.7 to 88.3, 93.4 to 98.6 and 80.2 to 93.3.
Comparisons with LH015 against studies/LH015/data/pair_classes.csv: 129 / 38 / 68 / 12 / 222; the two-way table (38, 2, 20, 5, so 40 read and 25 did not, of which CI reads 20); 21 of 24 and 4 of 5; 127 of 133 and 59 with neither; 321 of 469 and 94 of 415. The figure's container bars (129 / 38 / 68 of 235; 39 / 6 / 45 / 58 of 148) are computed by draw_figure.py from LH015's pair and recipe tables, and the 45 of 90 matches LH015's record.
The figure, rendered light and dark (scratchpad r_fig_light.png, r_fig_dark.png): both legible, the legend and hatching distinguishable in both themes, every number in the bars matching the data and the piece's alt text; the two SVG files are identical. The 11 "does not" segment of the CI pin bar is too narrow for a label (the alt text carries it).
Spot checks against the repositories at the snapshots, beyond the 33 pairs the blind sample covered: P324 datawrapper-mcp and P424 opentelemetry-mcp-server (uv sync and uv run with uv.lock at the root holding L at the timed version; class 1, may re-lock: holds), P160 galaxy (tox with -r of pinned-requirements.txt: holds), P063 devsecops-engine-tools (cd tools/; pip install -r requirements.txt, PyJWT==2.10.1: holds), P081 browser-use (no uv.lock in the tree, uv sync on a pyproject.toml pinning pillow==12.1.1: holds, and rightly not "may re-lock"), P009 root_painter (amendment 1: a script that installs -r trainer/requirements.txt: holds), P116 dana-python (GitLab pip install -r requirements.txt, urllib3==2.3.0: holds), P345 railtracks ("does not read", pip install -e packages/railtracks[all] beside an unread uv.lock: holds), P068 ai-evals (pip install ai-blackteam from the index: holds), P362 ttlock-ble (four calls to another repository's reusable workflows, undetermined: holds), P126 martin-binance (CodeQL, a container build and pip install build only, nothing of the requirements: holds), P071 bbot, the piece's example (tests.yml on push and pull_request runs poetry install with poetry.lock holding pyjwt 2.11.0 against the fix 2.13.0 of 21 May 2026, event kind "fix", lag 47.75 days; the Dockerfile does COPY . . then pip install .: every statement in the opening holds), and P278 chain-gang, where the class does not hold: the job runs PyO3/maturin-action (build --target-dir=target) and then pip install --find-links=target/wheels tx-engine, and the script classed it 3, "own package from the index". The wheel is the one built in the job from the project, whose pyproject.toml is T (cryptography==48.0.0); by the brief it is a wheel "built from the project in the job" and, since an external action built it, "one whose origin the rules cannot trace is class 6 (external)". The same idiom decides P277, the repository's other pair. Both pairs, now "does not read the pin", should be "undetermined" (or "reads the pin" if the origin is taken as traced), so class 3 has been given to 2 of the 11 "does not read" pairs wrongly. No class 1 decided wrongly was found in any pair checked: the errors run the other way.
uv run and uv sync without --locked or --frozen: flagged "may re-lock" wherever a T lock applied in the pairs checked, counted (83 of 164, 81 of 161 strictly) as the record says, shown as a separate hatched segment in the figure, and named in every summary that gives the lockfile figure.
Absolutes and near-absolutes. The title's "almost always" rests on 282 of 293 decided pairs and the summary carries the window (April 2025 to August 2026, checked against frame.csv's release times, 10 April 2025 to 13 August 2026) and "where our rules could decide"; "31 widely used libraries" and "332 projects" match the frame. Labels: the six labelled shares carry the brief's labels; the lockfile pin class's label is withheld by the one-fifth rule and every surface that gives 97 of 104 says so or gives no label; the record and the piece keep "as a rule" out of the piece's prose. The post hoc comparison (6 of 45 recipes re-lockable against 83 of 164 workflows) is labelled in the piece; it was not in the record's text, only in its figure, so I added it (below).
Consistency. README, the synthesis's and the short synthesis's notices, investigations/software-updates.md, programme.md's Answered entry and Next paragraph, AGENTS.md's "Where things stand" and observations.md give the same figures at the same certainty (395; 282 of 293; 164 of 188; 83; 45 of 90; the withheld lockfile label), except programme.md's checkpoint sentence, which said container recipes "mostly read" the pin (LH015 has no label standing) and CI jobs read them "almost always, lockfiles included" (the lockfile pin class's label is withheld and the lockfile itself is read in 87 per cent, "mostly" in the same entry's own words): corrected below.
Language. No em or en dash in any file touched; no American spellings (the one -ize match is the project name cloudimized); the short form's body is about 230 words.

What I changed

programme.md, checkpoint of 30 September 2026. Before: "pins move over weeks, container recipes mostly read them, and CI jobs read them almost always, lockfiles included." After: "pins move over weeks, container recipes install from them in most of the cases the rules could decide, with no label standing, and CI jobs almost always, the lockfile itself mostly."
studies/LH017/LH017.md, Answer, P3, after "50.3 per cent (38.6 to 60.9)." Added: "Of LH015's 45 pairs whose container build read the timed lock, the build did so only through such commands in 6, by LH015's flags: a comparison made after the counts, for the figure, and not tested (post hoc, one comparison)."
articles/tests-read-the-lockfile.md, "What it does to what we said before". Before: "the tests install it, again at each change." After: "the tests are set to install it again at each change." (A workflow is not a run; the sentence stated the run as fact two sentences before the paragraph said it could not.)

Problems that must be fixed before release

A. Three pair classes are wrong by the brief's rules, and the headline count carries them. P327 (ai-parrot) reads the pin, and P277 and P278 (chain-gang) are undetermined, so "does not read the pin" is 8, not 11, undetermined 50, not 48, and P2 is 283 of 291 (97.3 per cent), not 282 of 293; by pin class, exact pin 94 of 95, not 93 of 97; P3 and every lockfile figure are unchanged, no label moves, and the one-fifth shares stay below one fifth except the lockfile class's, which is unchanged (r0026_checks.txt). The figures 282, 293 and 11 appear in the record, the piece, its alt text and figure, the short form, README, both synthesis notices, programme.md (twice), AGENTS.md and the record's Findings 3, 5 and 11 and S6. Proposed resolution, in order of preference: (i) correct ci.py for the three constructs (look past xargs as past sudo and env; for uv sync|run --package <member> take the member's manifest, recorded as amendment 3 since the brief does not name the flag; treat pip install --find-links <dir> <own package> where <dir> holds a wheel built in the job as that wheel, class 6 external when an external action built it), correct reading A2 to respect --no-deps, re-run classify and analyse (the replay makes this cheap), list the corrections in Method and amendments.md, and update every surface; or (ii) leave the data and state the three known errors and the corrected P2 in the record's Findings 13 and Limits and in the piece's limits paragraph. Either way the figure's CI pin bar must match what the record says. B. Finding 13 must carry the blind check's result, not only point at this note: 17 of 20 and 19 of 20 agree on class; 2 script-decided and 1 judged job wrong after resolution; the threshold not exceeded; P327's class changed by it; the two script defects found, and the third the spot check found.

Optional suggestions

The title says "automated tests" while the count includes every CI job (lint, publish, docs); "test workflows" or "test and build jobs" would be exact, and the body already says so.
Galaxy's class 6 jobs that call common_startup.sh are labelled "script not read" (the depth-2 source) where the operative reason is a variable-named installer; a "build argument" label would tell a reader why, and a later study could read ${VAR} install where every assignment is an installer of the table.
The replay's last line labels all GET rows "PyPI requests (60)" and prints 30 where the record says 27; count GET rows whose URL is pypi.org, or label the line "GET requests".
In the figure, give the 11 (or 8) "does not" segment of the CI pin bar a label beside the bar; it is the only number in the chart a reader cannot read from it.
The Limits could say that the blind check and spot checks found the classifier's errors running against class 1 (a class 2 or 3 given where class 1 or 6 was right), not inflating it, since that is what a reader of "almost always" would want to know.

Verdict

Something must be fixed before release: problems A and B above. The blind check passes its threshold (17 of 20 script-decided and 19 of 20 judged on class, 2 and 1 wrong after resolution), the replay reproduces every output and the brief's hash matches, every figure on every surface matches the data, and the figure, labels, consistency and language pass, but the review found three pair classes wrong by the brief's own rules, all in the direction of understating "reads the pin", and the record must either carry the corrected counts (283 of 291, 8 and 50) on every surface or state the known errors; and the record's Finding 13 must report the blind check. The three small changes above are listed with before and after. The reviewer's spend is in its transcript (harbour/hb tokens --latest).

Resolution, by the driving session

Written on 30 September 2026, 08:23 to 08:55 UTC, after the verdict above. Both problems were resolved by the first of the proposed routes; the record is now version 0.2 (its Corrections give before and after).

A. ci.py now looks past xargs; follows uv sync and uv run with --package <member> to the member's manifest and the members it takes from the workspace, and reads a workspace root's uv sync with no lockfile as resolving every member's manifest (amendment 3, since the brief names neither; uv's workspace documentation was read afterwards, at 08:45:44 UTC, to check it, and is logged); treats pip install --find-links <dir> of the project's own package, where an external action built the wheels in <dir> in the job, as class 6 (external); and respects --no-deps wherever the project is installed, so reading A2 no longer applies to such jobs. Two further faults found while correcting were fixed by the same rules: a constraints file that pins L no longer overrides a reading of the pin, since a constraint installs nothing, and bash -c '<command>' is read as the command it runs. The amendment's effects carry no "may re-lock" flag, as no lockfile exists to rewrite, the same as the rule for a manifest in T without a lock. The corrections are listed in amendments.md; 51 synthetic cases pass and the replay reproduces every output.

The counts differ from the ones this note expected by one pair. P327 now reads the pin and P277 and P278 are undetermined, as the note found; and P076 (boneio-eu/app_bbb), whose only install of the project is pip install -e . --no-deps, followed by aiohttp named without a version, moved from reads the pin to does not, by the --no-deps rule applied throughout. So P2 is 282 of 291 (96.9 per cent, 94.4 to 98.9), not 283 of 291; does not read the pin 9, not 8; undetermined 50; exact pin 94 of 95 and both 91 of 92, the latter the one change the note did not foresee. P1, P3, P2 for lockfile pairs and every label are unchanged. The record's Answer, Findings 2 to 8 and 10 to 13, Method, Limits and Corrections, the piece and its alt text, the short form, README.md, both synthesis notices, the investigation page, programme.md (the Answered entry and the checkpoint's undetermined count) and AGENTS.md now give 282 of 291, 9 and 50, and the figure's CI pin bar reads 282, 9 and 50. S1's tally of installers had counted a row twice when a job ran uv twice; it now counts each row once (uv 934, pip 734). The judgements made after the review, 27 in an-website for jobs whose xargs installs the correction exposed, are marked in judgements.csv's made column, and the judgements as made before the sample was drawn are kept in data/judgements_before_check.csv.

B. Finding 13 now gives the blind check's result: 17 of 20 script-decided and 19 of 20 judged jobs agree on class; 2 and 1 wrong after resolution, within the threshold; the three faults, the third script fault found by the spot checks, and the four pairs the corrections moved.

Optional suggestions. Taken: the title now says "test and build workflows"; the replay's last line counts only PyPI's JSON requests (27); the figure's narrow "does not" segment is labelled above the bar; the Limits say which way the faults ran, with the one correction that ran the other way (P076). Not taken: the galaxy jobs' reason label stays "script not read", since the class and pair are unchanged, and a variable-named installer is left for a later study to read.

The clones were deleted at 08:52 UTC, after the corrections' last reads, as the brief requires. This resolution is read once by a fresh agent, with the reader review's, before release.