Blog

Property tests found 21 defects in a library with 600 passing tests

remex, Oskar's embedding-compression library, had about 600 tests, all passing. Four coding agents wrote property tests for it in parallel. Those tests failed on 68 cases covering 21 defects, and one of the defects silently corrupted data. The method is Inanna Malick's, described in Property Testing with Agent Swarms, where she reports more than 100 correctness findings across Codex, jj, uv, Prometheus and others.

Agents write the checker once and the CPU runs every case

A property test generates thousands of inputs and operation sequences, runs each against the real code and against a deliberately simple reference, and flags every disagreement. For a cache, the reference can be a plain dictionary. The test library then shrinks a failing sequence to the shortest one that still fails.

Malick has the agents write the reference, the generators and the assertions, which is tedious work they do well. Once written, the harness checks as many histories as you have CPU for. Each confirmed bug then becomes one small regression test and one minimal fix, kept apart from the large discovery suite.

Generated inputs went where the existing tests never went

She packages the method as a skill, a set of written instructions a coding agent loads before a task. Oskar asked Claude Code on the web to run her proptest-praxis skill against remex 2.2.0, one agent per area: search paths, saved files, the stateful index with its caches, and encoding. The worst finding was in file formats. remex has two container classes that both save .npz files: one packs codes as a single bit stream, and the other pads each row to a whole byte. When the vector dimension times the bits per coordinate is a multiple of 8, the two layouts are identical. Otherwise, a file saved by one class and loaded by the other returns wrong codes with no error. No existing test made that cross-read at such a size; more than half of the generated containers had one.

The others were smaller. A pickled copy of the default quantizer segfaulted, because it carried raw memory addresses into the new process. A search skipped its compatibility checks once its cache was warm, so a mismatched reader got plausible scores instead of an error; three of the four agents found that one independently. Nineteen fixes shipped as remex 2.2.1. Two findings can only be fixed by changing codes that users have already stored, so they are waiting on Oskar. A clean run would have meant nothing without a check that the tests could fail at all.

The tests caught 72 of 73 bugs planted to check them

Malick treats a quiet run as a coverage problem until shown otherwise, so each agent planted plausible bugs on purpose, such as a dropped cache invalidation, and confirmed its tests went red. They caught 72 of the 73. The survivor needed a GPU search, a warm cache and a mixed precision in one sequence, a combination the generator rarely drew.

The property tests also rejected a wrong fix. In a smaller pilot on Oskar's bm25 search skill, a directory of files containing no indexable words crashed the search. The first fix returned no results at all. The unit test written for the crash only checked that nothing raised, so it passed. The property that the number of results equals the smaller of k and the number of files failed, because there were files to return, and the fix was changed.

Property tests find the wrong answers that never crash

Oskar's skill library already had a crash hunter, hunting-failing-inputs, which uses the Z3 constraint solver to find inputs that index out of bounds or overflow. The remex corruption never crashed. Every read returned a valid-looking array, and only a comparison against a reference could show it was the wrong one. Malick's method supplies that comparison.

The method carries over to Python once it has a budget

Her economics held: the tokens went into four harnesses, and every further case cost only CPU time. Her rule to keep the discovery suite apart from the fixes held too. Each fix in remex 2.2.1 arrived with one short test against a fixed input, and none of those tests imports Hypothesis.

Two things needed adding. The first is a budget: her skill runs as an open-ended swarm that keeps exploring, and here each agent used roughly 400,000 to 490,000 tokens; two of the four hit the account's spend limit mid-run. The second is a reproduction step. The agents caught most of their own harness mistakes while building, but one reported a float32 rounding gap against its float64 reference as a defect. So the orchestrating Claude session reproduced each finding outside the harness before fixing it. Every finding the agents marked as confirmed still failed when rerun as a plain test against 2.2.0, and Oskar spent no time on triage. Most defects were shallow once found; the harnesses showed where to look.

The new building-property-tests skill keeps her method and adds those two limits. It targets Python's Hypothesis, stops after one calibration pass per area, and requires a reproduction outside the harness before any fix.

Four skills check code for four different kinds of wrong

Property testing is one of four testing skills in Oskar's library, and each answers a question the others cannot.

QuestionSkillHow it answers
Can some input crash it?hunting-failing-inputsModels each operation that can fail on bad input in Z3 and solves for an input that reaches it, then replays that input in a harness that makes silent failures loud.
Does it return the wrong answer?building-property-testsRuns generated operation sequences against the code and a simple reference model.
Can this test fail at all?gatingMutates the code one token at a time and reports every change the test does not notice; its gate harness refuses to report a pass without a known-bad case.
Does the test cover every case?declaring-invariantsFinds tests that loop over a hand-copied subset of a registry and names the members nothing covers, and records the invariants a codebase promises.

They chain. building-property-tests calibrates its harness with gating's mutation script, and it ends by turning the properties a codebase depends on into invariants that declaring-invariants tracks. Two review skills sit around the four: verifying-claims checks that documentation still matches the code, and verifying-review-findings checks that each finding from a code review holds against the code it cites.

The full results, harnesses and rerun script are in experiments/remex-proptest.