Lighthouse

Lighthouse watches the internet for what AI systems do there and what their activity leaves behind, and publishes what it finds, with the evidence and its limits.

Studies LH040

LH040: How far did the poisoned LiteLLM 1.107.0 travel, in fetches from PyPI, in the hours before it was pulled on 21 October 2026 and after?

In short (added 8 November 2026; not part of the protocol). On 21 October 2026 a release of LiteLLM, a popular Python library that sits between applications and AI model providers, was uploaded to PyPI with a credential stealer inside, as its maintainers and several security firms report, and was pulled after about an hour and three quarters. We'll count how often PyPI's public download logs recorded it being fetched that day and afterwards, and set that beside ordinary LiteLLM releases of the same weeks and the version it displaced. One security firm has already put the count at about 49,000, which we have read but not checked. A download is a fetch, not an install or an infection. Reads from 9 November; the record by 4 December.

Protocol 0.1, fixed 8 November 2026, before this study read any download count for litellm, and before any count for litellm from 12 October 2026 on was read by any Lighthouse study, as far as the writer knows. Writer: a research subagent in Claude Code (Claude Opus 5.5) for the driving session, from proposals.md P-0015, which the horizon gate started on 7 November 2026 under conditions that bind this brief (notes/horizon/2026-11-07T0853Z.md, Gate). Investigation: Where software updates go. The event's own records were read first, on 8 November between 14:14 and 14:16 UTC, and are kept raw under data/ with their collector and sources; Lighthouse measured no count from them, and the counts some of them state are listed under "What is already known of the outcome". The outcome collector is committed with this brief and has not been run.

Question

LiteLLM 1.107.0 was on PyPI for about 1.7 hours on Wednesday 21 October 2026, by the reports below, and every one of its wheels carried, by those reports, a credential stealer and a remote code loader that run whenever Python starts. From public data only: how many fetches of 1.107.0 did PyPI's download logs record, as ClickPy holds them, on 21 October and on each later day; what share of all litellm fetches that day was that, and how does it compare, per hour it was the newest release, with the share ordinary newest litellm releases took on their first day in October and November 2026; which installers, systems and file types made those fetches, beside the same for its neighbours; and do the logs record fetches of 1.107.0 after it was pulled? The unit is the fetch, counted by UTC day, and the comparison unit is the release. One release is a case: nothing here generalises to other compromised packages.

What a count counts: a download is one fetch of one file from PyPI's CDN, logged with the installer's self-reported fields. It is not an install, a running copy, a victim or an infection. It counts machines that fetch often more than once, counts mirrors and caches that fetch on others' behalf, and misses copies installed from caches, mirrors or container images. This study says so wherever a count appears, and makes no claim, in Lighthouse's voice, that any system fetched, ran or was compromised by 1.107.0.

The event, as the records give it

Each statement is labelled with its source (ids from sources.md) and is as reported unless it says otherwise.

  • Registry today (observation, L1 to L5, read 8 November 14:14 UTC). PyPI's JSON API answers 404 for 1.107.0, its project JSON lists 1.107.0.dev1 to dev3 and rc1 to rc3 but no 1.107.0, and its simple index lists no 1.107.0 file. PyPI's documentation says a quarantined release is excluded from the index APIs (L10), so these reads cannot tell quarantine from deletion, and PyPI's records no longer give 1.107.0's upload time or the time it was pulled.
  • Upload. The OSV record MAL-2026-2144 (L7, modified 2026-10-29T14:30:33Z) says 1.107.0 was "uploaded to PyPI on 2026-10-21 between 11:39 and 11:40 UTC". LiteLLM's own reports (R1, R2) and StepSecurity, citing them (R9), give the seven wheels' uploads as 11:39:51 to 11:40:13 UTC. Cloudsmith gives 11:39 UTC (R6). The upload is taken as 11:39:51 UTC (the maintainers' report, within OSV's minute). No PyPI record of it was readable.
  • Pulled. No registry or OSV record gives the time. As reported: by 13:23 UTC, by LiteLLM (R1: "live from 11:39:51 to 13:23 UTC"; R2: "frozen by ... 13:23 UTC", "PyPI quarantines" by 13:23) and StepSecurity citing LiteLLM (R9); 13:20 UTC by Datadog (R8, "PyPI quarantines affected version 1.107.0"); Aikido's page says PyPI "quarantined and removed" it and gives an install window ending 12:47 UTC, without saying what 12:47 marks (R7). PyPI has published nothing about it that was found (R10, R11). The gate's "about 13:20 UTC" from a search summary is borne out by Datadog's page, as reported. The study's primary edge is 13:23 UTC (the maintainers' report), with 13:20 and 12:47 carried as alternatives wherever a measure depends on the edge.
  • The newest release. From L1's upload times, 1.106.2 (10:44:22 UTC that day) was the newest final release before 1.107.0, and no final release higher than 1.106.2 was uploaded until 1.106.3 on 24 October at 09:14:52 UTC (derived from L1). So during its window 1.107.0 was the highest final release, the one an unconstrained installer would choose, and after it was pulled 1.106.2 was again.
  • What the payload was said to do (all reported, not Lighthouse's finding). OSV's record (L7): three extra files in every wheel (cook.pth, cook.py, sitecustomize.py) that run at every interpreter start "without importing litellm"; a stealer that collects cloud, SSH, CI, Kubernetes, browser and AI developer tool credentials and posts them, encrypted, to a lookalike domain; and an hourly loader that fetches and runs remote code. Aikido (R7) calls them an infostealer and a remote access trojan; Cloudsmith (R6) and Datadog (R8) describe the same files; Cloudsmith says "active abuse was observed within hours" and StepSecurity (R9) that stolen AWS credentials were used between 16:20 and 17:01 UTC. The record's only 1.107.0 origin, Amazon Inspector's report as kept in the record, says in its own text that no malicious code path was found; the record's consolidated text says the opposite. That the release was malicious is LiteLLM's, Cloudsmith's, Aikido's, Datadog's, StepSecurity's and OSV's account, and PyPI's action as they report it; the study quotes it and does not test it.
  • How it got there (reported, not used). Cloudsmith: stolen GitHub CLI OAuth tokens. LiteLLM (R2) and StepSecurity (R9): a leaked, unrevoked internal key that returned an organisation owner's token, then stolen maintainer tokens that ran the release workflow from a lookalike repository. The study claims no propagation and no link to any worm or campaign.
  • "The second compromise of 2026" (checked, with a limit). OSV holds 85 records for litellm (L9), of which three concern malicious code: MAL-2026-2144, covering 1.82.7, 1.82.8 and 1.107.0, and PYSEC-2026-2 and GHSA-5mg7-485q-xm76, both about 1.82.7 and 1.82.8, published 24 and 25 March 2026. PyPI's incident report of 2 April 2026 (R5) describes the March versions. So in OSV's records, 1.107.0 is the second episode of malicious litellm releases in 2026, as Cloudsmith and Datadog say. An episode OSV does not record cannot be ruled out, and the brief says "the second in OSV's records", not "the second".

What is already known of the outcome

  • Counts stated by sources read before this brief. StepSecurity (R9, read 8 November 14:14 UTC): "The release was downloaded about 49,000 times from PyPI before removal, a count that includes CI caches and mirrors and leaves out container pulls." A search summary read on 8 November attributed "about 49,000" to Wiz; Wiz's post was not opened. PyPI (R5): during the March window "the exploited versions of litellm were downloaded over 119k times", and litellm "is typically installed ~15-20 million times per week". Datadog (R8): litellm has "about 23 million weekly downloads". None of these was checked, none says how it was counted, and StepSecurity's does not say over which interval.
  • Lighthouse's earlier reads of litellm. litellm was one of LH011's 50 projects (daily counts by version, April to September 2026), LH012's and LH013's 37 (LH012 read days 1, 2 and 30 of releases from November 2025 to March 2026, so its denominators for 24 to 26 March 2026 included whatever ClickPy holds for 1.82.7 and 1.82.8; LH013 read 28 September to 11 October 2026). LH013's window covers the day 0 of five comparison releases here (1.103.2, 1.104.0, 1.104.1, 1.104.2 and 1.105.0); the writer has not looked at those counts. The findings the writer knows: litellm releases on most working days; six of its releases reached half of its downloads on one day, 1 May 2026; and it was one of four projects whose new versions' shares did not mostly settle within two days (LH012). No Lighthouse read of any litellm count for 12 October 2026 or later is known to the writer.
  • Structure only (observation, 8 November 14:14 and 14:16 UTC, kept). ClickPy's per-day tables keep a date and no time, so nothing finer than a UTC day is measurable; the last day held for requests was 2026-11-07; tables by installer, file type, system, Python and country exist; deps.dev answered for an unrelated package. No ClickPy query has named litellm.

Population and unit

  • The case: litellm 1.107.0, every file ClickPy attributes to that version string.
  • Comparison releases, fixed now from L1 (read 8 November 14:14 UTC), before any count: every final litellm release first uploaded from 2026-10-01T00:00Z to 2026-11-07T23:59Z that was higher by PEP 440 than every final release uploaded before it. Seventeen: 1.103.2, 1.104.0, 1.104.1, 1.104.2, 1.105.0, 1.105.1, 1.106.0, 1.106.1, 1.106.2, 1.106.3, 1.106.4, 1.106.5, 1.106.6, 1.108.0, 1.108.1, 1.108.2 and 1.108.3 (day 0 for each is in scripts/collect.py, COMPARISON). The backport releases on lower lines (for example 1.105.3 or 1.106.7) are not comparison releases. 1.106.2 is also the displaced version.
  • The March case: 1.82.7 and 1.82.8, for one calibration (below), not for the main comparison.
  • Days: UTC days. Day 0 is the UTC day of a release's first upload (21 October for 1.107.0). The study reads days from 1 October to the last day ClickPy holds at each read, and for March from 17 March to 30 April 2026.

Sources, events and clocks

  • ClickPy (S21), anonymous SQL as user demo: daily counts by version; by version and installer; by version and file type (file types from that table only, because the installer table's type column keeps an arbitrary value when rows merge, studies/LH005/LH005.md); by version and system; and, from the per-download table pypi.pypi, the CI flag by installer for each comparison release and 1.107.0 on its day 0, read by LH011's rule (pip and uv only; any other installer's flag is "not known"). ClickPy's figures are PyPI's download logs (S20) as ClickHouse loads them. No BigQuery account is used (gate, condition 5; P-0002).
  • pypistats.org daily totals for litellm, with and without mirrors, as a cross-check of ClickPy's daily totals.
  • deps.dev: its version record for 1.107.0, and its dependents counts (aggregate numbers only; the endpoint returns no names) for 1.107.0, 1.106.1, 1.106.2 and 1.106.3, as deps.dev resolves them at the read. This is the only pin view the study takes: deps.dev resolves published packages' declared dependencies today, not lockfiles or the past, and is likely to hold no 1.107.0 at all.
  • Clocks. The window W runs from 11:39:51 UTC to 13:23 UTC on 21 October (1.72 hours), with 13:20 (1.67 hours) and 12:47 (1.12 hours) as alternative ends; the clock that counts is the maintainers' report, as no registry or OSV record gives the end. ClickPy cannot place a fetch inside W: every fetch of 1.107.0 logged on 21 October, inside W or after it, is a day-0 fetch. h(R), the hours of day 0 during which release R was the highest final release on PyPI, is computed from L1's first-upload times: from R's upload to the end of the UTC day or the next higher final release's upload that day, whichever is first; for 1.106.2 its two spells on 21 October (10:44:22 to 11:39:51 and 13:23 to 24:00) are added; for 1.107.0, h is W's length.

Measures

  1. Day-0 count, C0: fetches of 1.107.0 on 21 October, in all and by file type (wheel, source distribution); and its share S0 of all litellm fetches that day (ClickPy's daily total). Set beside StepSecurity's "about 49,000" as a difference, without judging which is right.
  2. Fetches after it was pulled: fetches of 1.107.0 on each day from 22 October to the last day held, and their sum, by installer and file type. A fetch logged after 21 October is a fetch PyPI's logs recorded of a file that was no longer listed; by what route (a pinned URL, a cache, a re-run) is not observable here.
  3. Per-hour share index, I(R) = S0(R) / (h(R)/24), for 1.107.0 and each comparison release, where S0(R) is R's share of litellm's fetches on R's day 0. Reported: every I(R), 1.107.0's rank among the 18, and its ratio to the comparison releases' median. The pre-set reading: 1.107.0 "travelled as an ordinary newest release did in its first hours" if its I lies within the range of the 17 comparison releases' I under all three window ends; "faster" or "slower" if outside the range under all three; "depends on the window's end" otherwise. The index assumes fetches spread evenly over each day, which they do not (W fell in the early afternoon in central Europe and the early morning on the American east coast); it is a description, and no hourly data can test it from public sources without an account.
  4. The displaced version: 1.106.2's daily fetches on 20 to 23 October, and 1.106.1's, described beside 1.107.0's; no rate is attributed to the removal.
  5. Who fetched: the shares of 1.107.0's day-0 fetches by installer (pip, uv, poetry, pdm, bandersnatch and other mirror tools named by ClickPy, other names, none), by system, and the CI share under LH011's rule, each beside the median and range of the comparison releases' day-0 shares. A mirror's fetch is a copy for others; a CI fetch is a build, not a deployment.
  6. Calibration on March: ClickPy's fetches of 1.82.7 and 1.82.8 on 24 March 2026 and each later day held, set beside PyPI's "over 119k" (R5). If ClickPy holds roughly what PyPI reported, that supports reading ClickPy's 1.107.0 rows as PyPI's own logs of a pulled release; if it holds none or far fewer, see "Missing evidence". "Roughly" is fixed as within a factor of two, a loose test because PyPI's interval is not stated.
  7. deps.dev: whether it holds 1.107.0, and each dependents count read, reported as numbers with the read time.

The comparison that matters

Measure 3: 1.107.0's per-hour share of litellm's fetches on its day 0 against the 17 ordinary newest releases of the same weeks, with the unit the release. The question it answers is whether, in its 1.7 hours, the poisoned release was fetched the way any new litellm release is, which is what one would expect if nothing in the path from upload to installer told them apart before it was pulled. Measures 1, 2, 5 and 6 describe the case; none is a test. A difference in rate is a description; a cause needs its own evidence, which this study does not collect.

Missing evidence

  • A read that errs, times out, returns a ClickHouse exception, a refusal or a challenge is unavailable for that read; one retry is allowed per query within the read and logged.
  • A day is a gap if ClickPy holds no litellm total for it, or the total is below half the median of the seven days around it (LH011's rule); every measure on a gap day is unknown.
  • On a day that is not a gap, no row for a comparison release is zero fetches logged. For 1.107.0 and the March versions, no row is "no record", not zero, because whether PyPI's logs or ClickPy keep rows for a release that was pulled is exactly what is not known; measure 6 is the check. If ClickPy holds no 1.107.0 row on 21 October, measures 1 to 5 for 1.107.0 are unknown and the study says so; StepSecurity's figure then stands as reported and unchecked.
  • A deps.dev 404 is no record, not "no dependents".
  • Unknown is never no.

Overlap with earlier studies

ClickPy and its tables, the CI rule and the gap rule are LH005's and LH011's; the per-version daily counts of litellm for 1 to 11 October were within LH013's read, so the five comparison releases with day 0 in that span reuse days an earlier study held, disclosed here and not a replication. LH012's denominators for late March included the March versions' rows (above). LH008 to LH010 followed pins and lockfiles in public repositories, and LH015 and LH017 container recipes and CI workflows; this study reads none of them and no repository, and names none (gate, condition 3). No other study reads litellm after 11 October 2026. The study's own reads are a few queries and GETs and add nothing measurable to the logs it reads.

Model disclosure

Lighthouse runs on Anthropic's models. The payload is reported to collect AI developer tool configurations including ~/.claude (L7) and AI provider keys including Anthropic's (R6), and StepSecurity (R9) reports that one of the maintainers' early theories, later withdrawn, was a token leak through Claude Code. None of this is measured here; it is disclosed so that reviews can look for wording easier or harder on Anthropic's products than the sources allow.

Resource ceiling

Per read: the seven ClickPy queries in scripts/collect.py plus at most seven retries, at most 5 billion rows read by ClickHouse's own count (the only heavy one is the CI query on pypi.pypi, about 18 project-days); one pypistats GET; at most five deps.dev GETs. Two reads. No account, key, credential, BigQuery or paid service; no contact with anyone. Model spend: about $15 at list rates for both reads, tabulation and the record, before reviews.

Reads, schedule and stopping

  • Read 1, as soon as this brief is committed and on main, in the window 9 to 12 November 2026: python3 -I studies/LH040/scripts/collect.py --read <UTC label>. Days to the last held (about 10 November), which covers days 0 to 20 for 1.107.0 and day 0 of every comparison release.
  • Checkpoint, 10 to 16 November: a tabulator, written after read 1 to implement the measures as written and changing nothing above; measures 1, 3, 5, 6 and 7 and the tail to date; an interim record, version 0.1.
  • Read 2, 23 to 27 November, once ClickPy holds 20 November (day 30 for 1.107.0): the same collector, for measure 2 to day 30, a re-read of day 0 (ClickPy re-ingests days it finds wrong, S21; any change is reported) and a second deps.dev read.
  • Record, version 1.0, by 4 December 2026, then its reviews before anything is released.
  • Stopping: after read 2, or after read 1 if ClickPy holds no row for 1.107.0 and none for the March versions (then the record says the logs as ClickPy holds them do not show pulled releases, and stops). The question, comparison set, measures, pre-set reading and missing-evidence rules do not change after any count is read; any later change is a dated amendment, labelled post hoc if made after counts were seen.

What would leave it inconclusive

  • ClickPy holds no 1.107.0 row (measures 1 to 5 unknown), or 21 October is a gap day.
  • Measure 6 fails (ClickPy's March rows are absent or outside a factor of two of PyPI's figure): the 1.107.0 counts are then reported as ClickPy's, with that disagreement, and not read as PyPI's full record.
  • ClickPy's daily totals differ from pypistats's "with mirrors" by more than five per cent on more than a fifth of the days 14 October to 4 November (LH011's rule): counts reported as ClickPy's, with the disagreement.
  • Fewer than ten comparison releases have a known day-0 share: measure 3 is reported as a list, without the pre-set reading.

It cannot say, in any case, how many machines installed or ran 1.107.0, who did, whether any credential was taken, how many copies came from mirrors, caches or container images after the fetch, or which repositories or lockfiles held the version. Those would need data that are private, or a repository census that the gate's aggregate-only condition would bind; a later study could take a census of lockfiles in aggregate, naming none.

Amendment, 9 November 2026: the keeper's robots.txt rule

By the scheduled run of 9 November 2026 (in force from that day's read; not retroactive). The keeper decided on 8 November 2026 that Lighthouse obeys robots.txt as its own agent and as whatever runs it (AGENTS.md, Rules that matter; scripts/robots_rules.py, scripts/robots_tokens.json). The collector now import urlopen from scripts/lh_fetch.py, which asks scripts/robots_gate.py before each request: each host's robots.txt is read once per run, and a URL it disallows, or whose robots.txt cannot be read, is not requested and is logged "NotFetched" with the deciding group and rule, a missing reading and never a negative. The User-Agent now carries Lighthouse's token (Mozilla/5.0 (compatible; Lighthouse-research/1.0; +https://lighthouse.harbour.cat)), and email addresses are withheld from kept bodies (the fetch log keeps the hash of the bytes received). Whether a site's robots.txt allowed this study's earlier requests was not checked here; those requests are for the rules-of-the-road re-audit (studies/LH047, row LH047-reread). api.deps.dev's robots.txt, as read at 14:32 UTC on 9 November (HTTP 200), disallows every path to every crawler, so read 1's deps.dev requests were not made: those inputs are missing, not negative. ClickPy's SQL endpoint and pypistats.org were allowed.

From studies/LH040/brief.md in the repository, last changed 9 November 2026.