The model advisor's rubric and decision rules
Updated later on October 8: from October 3 the hook could not see which model most web sessions were on, and it still cannot. Added a replay of the new rubric and links to the code and data. Updated October 9: an earlier version of this note said the hook could now see the model. The October 8 change scored those sessions as if they had started on Sonnet; the hook now assumes no starting model and tells the session which tier the prompt needs (see Sessions with an unknown model).
If you start Claude Code sessions on Sonnet to save money, the risk is a task that needed Opus. Oskar's workspace runs a hook, scripts/model_advisor.py, that reads each session's first prompt, asks a classifier about it, and decides by fixed rules whether to stop the prompt and suggest a stronger model. An earlier post explains why that prompt is the one moment a switch costs nothing. The workspace repository is private. The same rubric and rules are public as the model-advisor mod in claude-code-mods, which reads the session's model directly and can run /model and resend the prompt itself.
The three questions
The classifier is Jev, TypeSafe's model that answers questions about a text with calibrated probabilities. The hook sends it the prompt, cut to 4,000 characters, and the repository's name, in one call.
The first question is multiple choice: "Which is the least capable model tier that would reliably complete this task well?" Each of the four tiers carries a description we wrote; those descriptions are the rubric. Haiku gets mechanical work the prompt spells out: run a named command, rename, extract. Sonnet gets a bounded task whose deliverable the prompt names, "done when the named thing is done; nothing has to be found out first." Opus gets work set as a goal and left to be discovered: find what needs changing, or debug an unknown cause. Fable gets novel research and subtle correctness or security reasoning.
We redrew the Sonnet/Opus line on October 7. Sonnet 5.5 finishes a named deliverable well but stops to ask when it has to work out the goal itself. Oskar had switched one session to Opus mid-task for that reason, on a prompt Jev had called Sonnet at 72%. The rubric now draws the line on whether the work is named or has to be discovered.
We replayed the first prompts of 74 archived interactive sessions through both rubrics, each scored as if the session had started on Sonnet (method and counts). The old rubric would have suggested Opus for 13 of them and the new one for 22. Jev's most likely tier became Opus for 52 prompts instead of 30, and the shift is broad: a one-line request to install a language runtime went from 7% to 40% on Opus or Fable. The prompt behind the change went from 5% to 41%, still short of the bar below, so the new rubric would not have flagged it either. These sessions have no correctness labels, so the counts show where the calls moved and nothing about whether they improved.
The second question is yes/no, answered as the probability of yes: does the prompt describe the work well enough to judge it without opening a linked issue, file or web page? "Resume 71afa1b8" does not, and four of the five wrong suggestions in a replay of archived sessions came from prompts like it. In the 74-session replay, Jev judged half the prompts (37) that way under either rubric. A third asks whether the session is quick (minutes), a task (about an hour) or a project (hours).
The decision rules
The hook assumes sessions start on Sonnet, Oskar's default since September 28, and by default it only suggests upgrades. In practice he has started 21 of his 38 sessions since then on Opus and 16 on Sonnet. For each tier above the session's, strongest first, the hook adds that tier's probability to the probabilities of every tier above it, and suggests the first tier whose sum reaches 0.75. From Sonnet, 50% Opus and 30% Fable makes an 80% case for Opus with no single tier at 75%, which the earlier post's "at least 75% on a tier" did not cover.
A suggestion is dropped when the describes-the-work answer is below 0.5, logged as opaque. Other silent decisions log their rule: agrees (Jev's top tier is the session's), cheaper (it is lower), unsure (it leans higher, short of 0.75) or unknown-model (the hook could not see the session's model, covered below).
Setting ADVISOR_DOWNGRADE=1 adds downgrades, summed the other way: a tier plus every tier below it. Two rules then apply. A downgrade is dropped (quick) when Jev rates the session more than 50% likely to take minutes, since that saving is not worth an interruption. A session on Fable with an opaque prompt is still advised to take Opus (opaque-fable-prior), because none of the 32 graded archive sessions needed Fable.
Delivery and failure
In the terminal the hook blocks the prompt and names the tier, Jev's confidence, the /model command and a price ratio. At list input prices of $0.10, $2, $4 and $10 per million tokens for Haiku, Sonnet, Opus and Fable, Opus costs twice what Sonnet does and Fable five times. The resend passes whatever the user chose; each session is asked once. The claude.ai web and mobile apps don't display a block's reason, so there the prompt runs and the hook passes the model a note naming the tier. The note gives no instruction. The workspace's CLAUDE.md, which every session loads, tells a model below the named tier to hand the substantive work to one subagent on that tier, so the main session and its prompt cache stay on Sonnet. It asks for one subagent because each new Claude Code subagent first writes Claude Code's own startup prompt, about 55,000 tokens, to the prompt cache; in our seeded-bug runs that came to about $0.17 per Sonnet 5.5 subagent.
The hook skips sessions that already have a reply, scheduled Routines and SDK runs where no person would see a blocked prompt, and prompts containing #no-advice. Jev gets one attempt, with a six-second timeout by default. Any failure lets the prompt through and is logged; even a crash exits 0, because exit code 2 tells Claude Code to block.
Sessions with an unknown model
The hook learns the session's model from Claude Code's SessionStart and model-switch hook events, or from the --model argument the CLI process was started with. Since October 3, Claude Code on the Web starts the CLI process ahead of time with no model argument, and its SessionStart event has no model field. From then until this update the hook saw the model on 1 of 22 classified first prompts, the one where a model switch came before the prompt. An unknown model rules out every upgrade, so from October 3 to October 8, 2026, the hook could not suggest Opus at all. The October 7 session was one of the 21.
On the web and mobile apps, where the note already reaches the model instead of the user, the hook now assumes no starting model. It names the strongest tier the prompt needs with a summed probability of at least 0.75, counting that tier and every tier above it, and says it could not see the session's model. The model reading the note knows which model it is, and the CLAUDE.md rule decides what it does: hand the work to a subagent if it is below the named tier, do the work itself if not. That decision is logged as blind. In the terminal an unknown model still never blocks, because a blocked prompt is erased before anyone can check which model it was meant for.
Replay and report
Every decision goes to a log, silent ones included. report counts fires, errors and skips by reason, gives latency, and marks each fire switched or kept by the model the resend came on. replay runs archived first prompts through the same rules offline, on each session's own model or, with --from sonnet, as if all had started on Sonnet.
The earlier post reported a replay of 32 interactive sessions: 11 fires, 10 matching an Opus grader's labels, about $169 saved of $488, in-sample against one grader. Those figures are for the September 27 version, before the Sonnet default and the rubric change. The October replay above has counts and no labels; its scripts and aggregate results are in experiments/model-advisor-replay, and the transcripts stay private. The runs comparing Haiku, Sonnet and Opus 5.5 on seeded bugs are in experiments/temporal-routing-headroom. There Sonnet 5.5 at high effort fixed as many as Opus 5.5, and seeded repair was too easy to separate the two, which is why the rubric change rests on an observed session instead.