The validation gate would have rejected the fix that worked
An earlier post reported that down-skilling v1.2.0 cut Haiku's architectural hallucination on a voice-rewrite task from 19/20 runs toward zero (When down-skilling makes Haiku worse). That edit shipped on a before/after contrast, not a controlled comparison — the "after" used hand-written examples at n=5, the "before" was a different author's prompt at n=20. So I re-ran it through the optimizing-skills gate, the skill whose entire job is "ship an edit only if it beats the version you already ship on a fixed check set." Run by the v0.1.0 rules it shipped with, the gate would have rejected the edit — for a reason that has nothing to do with the edit. I patched the gate on the strength of this case, then validated the patch with the gate it fixes.
The controlled run
Two SKILL versions, held fixed: best = down-skilling v1.1.0 (the four v1.2.0 edits reverse-applied), candidate = v1.2.0. For each version, a Sonnet agent read only that version and compiled a Haiku prompt for the same task — take the marketing paragraph about a "caching layer," rewrite it dry. Each compiled prompt then ran on Haiku 4.5 five times. The only variable between the two arms is the SKILL text. Full logs.
The candidate arm invented architecture in 0 of 5 runs. The best arm invented it in 3 of 5 (60%) — "data retrieval mechanisms," "scale across distributed environments," none of it in the source. The original un-anchored author, at n=20, was 19/20 (95%). On the criterion that prompted the revision — invented detail — the candidate wins; at n=5 the magnitude is rough, but the direction is clean and holds across two independent authors. Invented detail is not the only thing the gate scores, though.
Where the gate breaks
The prompt states two constraints: don't invent, and 60–90 words. Every run carries two pass/fail criteria. The gate's v0.1.0 rule said "score hard pass/fail per task" and "accept only if candidate beats best." Collapse two criteria into one task-level pass and you need both to hold — and neither arm ever hit the length window. The authored examples averaged ~37 words; Haiku followed the examples over the rule and came in short every time. Both arms score 0/5 task-passes. A tie. The gate rejects ties.
So the 60→0 win on invention disappears, dragged to zero by an unrelated length criterion. The same collapse surfaced a second thing: one of the four v1.2.0 edits — length calibration — never transmitted. The author wrote the 60–90 rule into the prompt and then wrote 37-word examples, the exact rules-lose-to-examples conflict the skill warns about, committed against the skill's own advice.
Patching the validator with itself
The fix to optimizing-skills v0.2.0: score per criterion, and decide accept/reject on the criterion that prompted the revision, treating the rest as regression guards that must not get worse. It also now requires ≥2 author samples per version when the artifact is an agent-written prompt — the measured benefit swung 95→0 versus 60→0 purely on which author wrote the examples, and one sample can't separate a real edit from a lucky author.
That patch had to clear the same gate. Its triggering failure is a scoring rule, not a model behavior, so the check set is decision scenarios with known-right answers: a clean win, a no-op, a primary win that regresses a guard, and the down-skilling case. The per-criterion rule gets all four right. The collapsed rule gets three: on the down-skilling case it sees both arms fail the length criterion, calls it a tie, and rejects — the wrong answer, for the reason above. Candidate beats best on the gate's own terms. Shipped.
down-skilling v1.2.0 stays shipped; the controlled run confirms it. optimizing-skills is now v0.2.0. The length-calibration step still doesn't hold at runtime — that's the next edit, and it goes through the same gate.