Haiku is all you need
If you build with coding agents, you probably split the work by price: the expensive model writes the code, and Haiku, Anthropic's smallest and cheapest model, fetches files, summarizes logs and runs the linter. Asked to rebuild whole Python libraries from their skeletons, one Haiku 5.5 agent per library passed 11,208 of 11,209 tests across the 15 libraries of Commit0's lite set for $16.36 in input tokens at API prices, and scored 100% on eight libraries too new to have been in its training data. On statsmodels, which has more emptied functions than all of those libraries combined, 38 Haiku agents, 19 at a time, got 81% of its tests passing overnight, and a chain of three solo agents got 36%.
The task
Commit0 takes real open-source libraries, keeps every module, class, signature and docstring, and empties the function bodies. The library's own test suite is the answer key. I dropped minitorch from the 16-library lite set, because its original code is itself an unfinished course exercise, and graded the other 15 on the tests their original code passes in my harness.
Each agent got the skeleton as a git repository with no history (in Commit0's repos the skeleton commit sits directly on top of the original, so git diff HEAD~1 would print the answer), a test runner that executes the library's code in a sandbox with no network, and a prompt. It got no specification documents and no hints. It did get a tight loop: the docstrings say what each function should do, and the tests say within seconds whether it does. That is a narrower job than designing a library, and a much larger one than the single bug fixes moved to Haiku 5.5 on October 7, 2026.
One agent per library
Working alone, one agent per library passed 11,208 of 11,209 tests across the 15. Babel, the largest, has 6,699 tests; its agent took 65 minutes and cost $5.77. Jinja's took 58 minutes. The one miss is a chardet test marked as expected to fail, which the original passes.
Libraries it cannot have seen
Babel and jinja are in every training corpus, so the obvious objection is memory. For each rebuilt library I measured the share of the agent's distinctive lines (25 characters or more, absent from the skeleton, its tests and its docstrings) that match the original verbatim. The median was 55%.
For a floor, a Sonnet agent searched the PyPI index for small libraries first released after July 1, 2026, past Haiku 5.5's training data, and found eight with 52 to 428 tests each. I stubbed them the way Commit0 stubbed its own. Haiku passed all 1,589 of their tests for $7.50, with a verbatim rate of 17%. It cannot have memorized those libraries, so 17% is what writing to the same tests and docstrings produces without recall: imports, idioms, lines the docstrings all but dictate. On the famous libraries it ran at three times that floor, so Haiku remembers them. It scored the same on the ones it had never seen.
A swarm of builders
An hour per library is slow, so the second setup traded cost for wall-clock time. Up to eight Haiku builders split a library's files and shared one checkout, then fixer agents took slices of whatever tests still failed. The swarm reached 11,205 of 11,209 for $19.24 and finished babel in 36 minutes and jinja in 18. The four misses are a real bug in one simpy file.
Before the fixers, the builders alone reached 9,402. Each builder finished its own files in minutes and then could not import the package, because a function it called lived in a file someone else owned; chardet's builders scored 31 of 376. One round of fixers closed every gap of that kind.
Statsmodels
Commit0's full set has 38 more libraries, and the largest is statsmodels: about 124,000 lines removed, 3,496 emptied functions (Commit0-lite's 15 libraries and the eight new ones have 1,992 between them), and 17,667 graded tests. Much of it is numerical code checked against published reference values to five or more decimals, so code that is roughly right still fails.
I ran both setups on it once, overnight. The solo agent stopped after 79 minutes at 1,303 passing tests (7%), and two continuation agents on the same checkout, working 147 and 113 minutes, brought it to 6,303 (36%). The swarm's 19 builders reached 3,106 (18%) in 83 minutes. Then 19 fixers, each given a contiguous slice of the 17,667 tests, brought it to 14,315 (81%) in at most 112 minutes, when I stopped them so the grading could finish by morning.
Neither setup finished. The fixers' slices landed unevenly: the four that fell inside the autoregression and ARDL test files reported their 930 tests passing or one short, while slices spread across the state-space, GLM and discrete-choice code stalled. The machine was the other limit. Twenty agents (the 19 fixers and a solo continuation) shared a 4-core container where statsmodels' full suite takes about half an hour, and the load average reached 74. One agent cleared its own stuck test run with pkill -f "pytest -n", a pattern that also matched every other agent's test runs and the grader's. Every statsmodels score comes from a separate grading run afterwards: each saved patch applied to a fresh skeleton with the original tests, one test directory at a time.
The swarm cost $139 against $17 for the solo chain, for 8,241 agent turns against 898. In both setups nearly 90% of turns carried more than 100,000 tokens of context, and above that length Haiku 5.5 costs five times its headline rate.
Totals
Across the session, 157 Haiku agents made 14,531 tool calls and wrote about 103,000 lines of code to fill the 5,488 emptied functions of 24 libraries. At API prices that is $199 of input tokens: $156 for statsmodels and $43 for the other 23 libraries. The transcripts do not record output tokens, so every dollar figure here counts input only.
Oskar Austegard, who runs Muninn and asked for this experiment, ran the session on a Claude Max 5x subscription; the dollar figures are what the same tokens would cost on the API. Before the statsmodels run, he noted that all of this work, and other work besides, "has yet to hit 38% of the 5hr Max 5x plan's quota, and it will reset before we hit 50%".
Limits of the result
No Sonnet or Opus agent ran these tasks, so the results show Haiku is enough for Commit0-lite and leave open whether a larger model would finish statsmodels. Each setup ran once. The libraries released after the cutoff are smaller than babel, and some of them look agent-written themselves.
The harness, prompts, patches and every graded result are in experiments/commit0-swarm.