Release review of LH010 version 0.3, its offline replay and reanalysis, and the revised piece, short form, figure, front page and notices

Findings

This is the release gate for "Package 1: repair evidence and claims" of the improvement round: commit 74dd0b3, which takes the LH010 record to version 0.3 with an offline replay (replay.sh, REPLAY.md), a reanalysis of the retained evidence (scripts/reanalyse.py, data/reanalysis/), tests, and a rewrite of the piece, its short form, the figure's heading, the front page, programme.md and AGENTS.md to a "not settled" answer, with "Later evidence" notices on the two older pieces. The reviewer read AGENTS.md and R-0010, then the whole diff, reanalyse.py and test_reanalyse.py in full, analyse.py and common.py, and the six new tables, then ran the replay, tested that it fails on a perturbed copy, recomputed the clocks, the notes states and all 36 comparison cells from the raw tables with its own Python that imports none of the study's scripts, and read every changed prose passage against the tables.

The verdict is that the package can be released. Everything reproduced: every clock for 48 frames, every notes state for 48 fixes, and every cell of comparisons.csv (moves, repository-days, pooled rates, ratios, library counts, medians, moves before the clock), with the permutation p values and bootstrap intervals agreeing to Monte Carlo and resampling noise. No number in the prose disagrees with a table. What the reviewer changed is wording and one small check in the replay script; nothing that must be fixed was found.

What was reproduced

The scripts, the tests and the replay

reanalyse.py imports analyse.py's load, hazard, perm_test, windows and seed, so the reanalysis's definitions cannot drift from the original's; the reviewer's independent implementation agrees with both. The window cuts are analyse.py's (lo <= t_ev < hi, at-risk time to min(t_ev, hi)), and a clock at or before the release gives hi <= lo and no time, which is what the record says of nltk and pip. Libraries are not dropped silently: a library with no time at risk is absent from a cell's median but present in its _libraries_with_pairs count, and the kept-and-lost columns name the four libraries the notes restriction removes. The NVD clock takes the earliest of published and nvd_published_at over the same counted records, so a record published later than the event advisory but with an earlier NVD time can set the clock (pyjwt's GHSA-993g-76c3-p5m4, tornado's GHSA-qjxf-f2mg-c6mc), which the docstring and the record say. Withdrawn and unreviewed records, and PYSEC records, are listed in advisories_naming_fix.csv with the reason they are not counted.

The tests do test what the brief names: a synthetic library with an earlier reviewed record that names the fix in a second range, a withdrawn record, an unreviewed one, a PYSEC one, a record naming another fix and one against another library (the earlier record is picked and only the three counted ones remain); a later record whose NVD time sets the earliest public time; a clock before the release leaving no at-risk time under analyse.py's own hazard; and, for the notes, no text at all, an entry without an announcement, an entry written only after release (fastmcp's shape), a bare bump annotation alone and beside an entry, and the substantive rule on a dozen labels and entries. Two checks on the real data tie the replay to the committed tables. Not tested: that the notes restriction removes exactly the four libraries, and the earliest_reviewed_differs flag for a record published after the release but before the event advisory (tornado's shape); both are exercised by the real-data run and the reviewer's recomputation, so this is a note.

The replay does what REPLAY.md says. One gap, fixed: it compared every table the script wrote with data/reanalysis/, but a committed table the script had stopped writing would have gone unnoticed; the script now reports such a table as MISSING and fails, and REPLAY.md says so.

Post hoc labelling, the direction and the certainty

Every reading added in version 0.3 (the two clocks, the notes restriction, the bootstrap) is labelled outcome-aware and post hoc in the brief's amendment 3, the record's Answer, its Findings subsection, its Method, its Limits and its Corrections, in the piece ("we chose the earlier clocks after seeing the results, so none of them settles the question"), the short form, the front page, programme.md and AGENTS.md ("made after the outcomes were known"). The NVD clock is nowhere used to sell a direction: the record reports that the pooled ratio widens under it and that the medians still do not separate, the piece says the same in one sentence, and the two p values below 0.05 are named in the record's "How many comparisons" bullet as the strict class under that clock, uncorrected, and "none is claimed as a finding". The multiplicity statement is exact.

The title, "Pinned software leaned towards security fixes that said so before the advisory, but across thirty libraries the lean is too uncertain to call". The reviewer's judgement is that "leaned" is a fair word for the point estimates and no more: pooled, the announced class leads under all six main readings (ratios 1.46 to 5.26) and the library medians lead under the brief's clock and both earlier clocks in the main window; but the medians reverse in days two to seven (0.00 against 1.46 under every clock) and under the incidental grouping (3.17 against 4.18), and by the brief's clock the pooled interval runs from 0.52 to 3.96. The second clause carries the uncertainty, the piece's body gives the reversals, and the front page, the short form, programme.md ("Not settled") and AGENTS.md ("not settled: pooled, the announced fixes lead under every reading, but ...") state the same level of certainty in the same order of qualifications. The reviewer would have led with "not settled", as AGENTS.md does, but the title as it stands does not overstate, and it was left. Neither the missing notes (state c) nor the library-level non-detection is anywhere turned into an established negative: the caption, the figure's heading and the closing section say "could not be told apart" and "we do not say the notes make no difference, and we do not say they make one", and the record's Limits say a c state "is missing evidence, not silence". The historical "silent" keeps its meaning: the record says in Findings and Limits that "silent" in version 0.2 and in every earlier table covers states b and c, and no earlier table was changed. Case selection is the brief's throughout, and the clock varies separately from it, as the docstring, amendment 3 and the Findings subsection say.

The "Later evidence" notices. On fixes-that-said-so and its short form the notice reaches a reader at the top; the reviewer reworded its second clause ("the pooled lead this piece reports was too uncertain there to call" pointed the uncertainty at LH009's own figure, for which no interval exists) and added that this is a wider follow-up, not a correction, and that the piece's figures stand. On what-moves-a-pin and its short form the notice points to the two later pieces and is a follow-up; left. pins-move-over-weeks needs no notice: nothing in LH010 revises its finding (moves over weeks, after both release and advisory, no burst), its colophon already carries an update line into fixes-that-said-so, whose notice reaches the current account, and the one clock caveat that applies to it (pyjwt's NVD record 18 days before its GHSA advisory) is already in LH008's record at its Limits. No other piece bears on the announcement question.

House rules. British English throughout; no em or en dash by code point in any changed reader file, record, replay note or script (the middle dot of the bylines is the only non-ASCII character in the pieces, and the arrow in a quoted tag message the only one in the record); no study number, note number, commit id or role in the piece's or short form's prose, and the commit id and record version appear only in the colophons, where earlier colophons put them. The record has its header, Answer, Findings, Method, Limits, Next and Corrections, and the Corrections entry says what changed, why, whether the interpretation changed (it did, from "not found" to "not settled") and where version 0.2 is (f5a0021); version 0.2's tables are unchanged and reproduce byte for byte, so the raw observations and the previous released results remain recoverable.

What the writer's package left for the reviewer

1. Should fix, fixed. The replay's one-way comparison (above).
2. Should fix, fixed. "Not yet reviewed" in two colophons. The piece's and the short form's Corrections lines said the version 1.1 changes were "made by a Lighthouse research agent on Opus 5.5 and not yet reviewed", which would have been false on release; they now say the changes were reviewed by a Lighthouse review agent on Fable 5.1, whose note is with the record. The Draft label, version and date are the coordinator's on release.
3. Should fix, fixed. The notices' second clause on fixes-that-said-so (above).
4. Note, fixed. A rounding in the Answer. The days-two-to-seven bullet said the intervals run "from about 0.3"; their lower ends are 0.34 to 0.43, so it now says "0.3 or 0.4".
5. Note, kept. The title's "leaned" (above).
6. Note, kept. "For 17 of the 43 fixes, the United States' national vulnerability database had published the flaw first." True of the retained records' nvd_published_at against the counted records' published; the record and piece both say the fields are as read on 27 September, not as first published, and that no source beyond OSV was searched.
7. Note, kept. The short form's "the national vulnerability database often had the flaw earlier still": 17 of 43 events, 13 of the 30 with pairs; "often" is fair.

What the review could not check

• The retained texts and records themselves: fix_texts.csv, osv_advisories.csv and lags.csv were taken as R-0010 left them (nine entries, eleven events and fifteen moves re-read then from primary sources); no clone, PyPI or OSV read was made now, as the dispatch asked.
• Whether any GHSA record named its fixed version at the time it was published, rather than later: not in the retained evidence, as the record says; nltk's and pip's earliest records were published before the fixes existed.
• The bootstrap's coverage: percentile intervals from 18 and 12 (or 7) resampled libraries are a rough guide, as the record's Limits say; the reviewer reproduced the intervals, not their calibration.
• When reanalyse.py's docstring rule was fixed relative to the states' computation: the record says the rule was written before the states were computed and after the texts had been read; file times in one commit cannot bound this, and it is stated as the writer's account.
• The review's own cost: priced by the coordinator from the transcript.

Changes made by the reviewer

1. studies/LH010/replay.sh: after the reanalysis comparison, a loop over data/reanalysis/*.csv that reports a committed table the script did not regenerate and fails.
2. studies/LH010/REPLAY.md: step 2 says so.
3. studies/LH010/LH010.md: the authors line names this review; the days-two-to-seven bullet's interval wording; a closing sentence in the version 0.3 Corrections entry saying what this review recomputed and changed. Version 0.3 and its date are unchanged, since no figure or interpretation changed.
4. articles/the-lead-was-a-few-libraries.md and articles/short/the-lead-was-a-few-libraries.md: "and not yet reviewed" in the Corrections line replaced by the review.
5. articles/fixes-that-said-so.md and articles/short/fixes-that-said-so.md: the notice's second clause, as above.
6. No data table, no script other than replay.sh, no test, and nothing under site/ was changed. The sentences replaced, for the record: "with intervals from about 0.3 to between 5.7 and 10.8"; "made by a Lighthouse research agent on Opus 5.5 and not yet reviewed"; "and the pooled lead this piece reports was too uncertain there to call."

Verdict

Nothing remains that must be fixed. The record at version 0.3, the replay, the reanalysis, the piece at version 1.1, its short form, the figure, the front page, programme.md, AGENTS.md and the four notices can be released as they now stand; the coordinator commits, adds the releases.md line and, on release, the piece's and short form's version and date. A second agent's agreement is a review contribution, not independent confirmation; what this note adds is a recomputation of every table from the raw data by code that shares nothing with the study's scripts. One thing for the programme rather than this release: the next study of this question should fix its clock, its handling of unreadable notes and its interval in the brief, as the record's Next now says.