A decision model made of Haiku
If you route tickets or gate tool calls with Claude, you probably ask Haiku for a label and parse the reply. That gives you an answer with no number attached, so every answer gets the same trust.
Haiku 5.5 can attach a number. Asked for a probability on every option, three times in parallel, with a fitted correction on top, it matched the calibration of Jev, TypeSafe's purpose-built decision model, on 104 labelled questions and trailed it by about five points on accuracy, all of them on an eight-way intent question. When its three copies split, the answer was wrong more often than right.
Purpose-built decision models — Jev, OpenAI's Decisions API, AWS's Strands Decider — take some evidence and a fixed-form question and return a probability for every option instead of text. The question is a yes/no predicate, a choice among a few described options, or a score on an ordered rubric. This is for when none of them is in reach.
Four estimation steps
Haiku doesn't expose token probabilities, so Oskar asked me to estimate them in four steps:
- Haiku states a probability for every option, its pick and the rest.
- Three copies run in parallel. Choice options appear in a shuffled order for each copy, and yes/no options swap at random. Rubrics are never scrambled: two copies read them low-to-high and one high-to-low.
- The three distributions are averaged. When the copies disagree, the average flattens.
- The average is rescaled with one temperature per question type, fitted on labelled examples so that a stated 0.8 comes out right about 80% of the time.
A single copy already comes out underconfident, partly because the prompt tells Haiku to spread probability when an input can fairly be read two ways, and averaging three flattens it further, so the fitted temperature for choice questions is 0.62 and sharpens the average. Only the choice type had enough labels (64) to fit one; yes/no and rubric answers stay as stated.
The test
Sixty-four items are real customer messages from the banking77 test set, restricted to eight intents that are easy to confuse, such as "my card hasn't arrived" against "how long does card delivery take". I wrote and labelled the other forty: twenty-four yes/no guardrail questions modelled on the Strands examples ("are this tool call's arguments stated in what the user said?", "should the assistant ask before acting?") and sixteen severity and urgency ratings on three-level rubrics.
Jev was the decision model I could call, and it answers the same three question types; its probabilities are used as returned. I fitted Haiku's temperature on half the items, scored the other half, then swapped the halves.
| intent (64) | yes/no (24) | rubric (16) | all | calibration error | |
|---|---|---|---|---|---|
| Jev | 90.6% | 87.5% | 87.5% | 89.4% | 0.045 |
| Haiku, three copies | 82.8% | 87.5% | 87.5% | 84.6% | 0.041 |
Calibration error here is expected calibration error over ten bins of top probability, and on 104 items the two figures are level. The whole accuracy gap is on the intent question: Haiku got 53 of 64 right, Jev 58. The tie on the yes/no and rubric sets is weak evidence, since they are small (one item moves the score four to six points) and I wrote them.
Escalation rule
After calibration the two models' wrong answers sit at similar confidence: nine of Haiku's sixteen had a top probability under 0.7, and six of Jev's eleven did. Haiku also reports whether its three copies agreed. The 88 unanimous answers were 92% right and the 16 split ones 44%.
Sending the 16 split answers to a stronger model catches nine of Haiku's sixteen errors. Adding everything under 0.7 raises the escalations to 23 and leaves six errors. At the same budget Jev does better: its 23 least confident answers escalated leave four wrong, and its 12 least confident leave five, where Haiku's 12 least confident leave eleven. All of these assume the stronger model gets every escalated item right. I chose 0.7 on this same data and applied it to all three question types, although only choice answers were calibrated.
When to use it
It fits where a few seconds per decision is acceptable and no decision model is callable. Every number above comes from the Claude Code command line, where a decision took about four seconds and Haiku spent 0.5 to 0.8 s of it; Jev answered in 0.3 s. Each copy used about 2,000 input tokens, mostly cached instructions, and 150 output tokens, about $0.0001 a copy. A direct Anthropic API version is included but untested against the live API, so its speed is an expectation and its calibration should be refitted.
Answers come back in the format of OpenAI's Decisions API. Neither OpenAI nor Strands documents its confidence field. I checked one formula against their published examples: (k·p_max − 1)/(k − 1), the top option's distance above an even split over k options, reproduces OpenAI's 0.93 and 0.55 and Strands' 0.768, so the plugin reports confidence that way.
It ships as a Claude Code plugin: a decider subagent pinned to Haiku 5.5 with its instructions inline, so each answer needs no tool call, and a skill with the code that shuffles, pools and calibrates.
/plugin marketplace add oaustegard/claude-skills
/plugin install deciding-with-confidence@oaustegard-claude-skills