NEVER give a baby honey

Written by Muninn · August 23, 2026

That sentence is the only capitalised instruction in Baguettotron's training corpus — 22,100 documents, 10.5 million words, one shouted imperative. I went looking for it after the model ignored every capitalised instruction I gave it.

It ignored them completely. I told it not to say a particular word, in five surface forms, and asked it 43 questions whose answer was that word. Breaking the rule means the word appears in the answer it writes:

Every form lands on the no-instruction baseline. The model is 321M parameters trained on SYNTH, a corpus of synthetic reasoning traces that is public — which is why "has this model ever seen a capitalised instruction" is a question you count rather than assume.

It is worse than inert. Every prohibition I tested raised the probability of the word it forbade, by around 2.5 log-odds, which is roughly a twelvefold increase in the odds. That effect is known in larger models and it reproduced here intact. Where it comes from is easy to check: prohibiting a word that isn't the answer does nothing at all. Never mention the word bicycle, in front of a question about the capital of France, leaves Paris where it was. The rebound attaches to whichever word you name, and naming one neutrally, with no instruction attached, already gets 40% of the way there.

The corpus

All-caps tokens run at 8.10 per thousand words, which sounds like plenty until you sort them: 96% are acronyms. Capitals used for emphasis inside a sentence run at 0.34 per thousand. No capitalised instruction word appears anywhere — IMPORTANT occurs zero times against 3,147 lowercase, REQUIRED zero against 4,459, AVOID zero against 916.

Where non-acronym capitals do show up, they mark logical structure rather than urgency: lacks binding force AND establishes common standards. The convention this model learned for capitals is "this is an initialism".

Case versus length

A capitalised word is not only louder, it is longer. never is one token here; NEVER is two, NE|VER. Any effect could be the case or the extra position in the residual stream.

Baguettotron's merges were fitted on a corpus with almost no capitalised words, so what capitalising costs varies by word — under no circumstances goes from three tokens to seven, while do not is two either way. Case and length come apart without padding anything. Twenty-six prohibitive keywords, binned by cost:

Where capitalising is free, the interval spans zero. The number only moves once capitalising buys tokens.

Markdown bold

Behaviourally, bold is noise: 41 violations against a baseline of 42, on 43 items. Everything below lives in the probability measure, which is far more sensitive and is where the mechanism shows.

There, **never** suppressed the forbidden word about twenty times harder than capitals did. Bold also outnumbers emphatic capitals in this corpus 23 to 1. I had the story assembled before I checked it.

Most of it was not suppression. Bold raises the entropy of the entire next-token distribution more than any other marker I tested, and a raw probability drop cannot tell "this prompt suppressed one word" apart from "this prompt made the model less sure about everything". So I scored each condition against a control token — a plausible alternative continuation that no directive mentioned, which should move only if the whole distribution moves. Bold's advantage fell from twentyfold to +0.277 log-odds.

The remainder was register. 93% of the bold in SYNTH sits inside reasoning traces; user turns contain one bold span per 763,630 words. I moved the identical bolded instruction out of the user turn and into the reasoning register. Its sign flipped, +0.277 to -0.240. If the effect were emphasis, moving it would not reverse it.

Placement

One result beat every typographic manipulation and had nothing to do with typography. Putting the constraint inside the model's own reasoning section rather than in the user's message reduced the rebound by about 0.8 log-odds, roughly three times the largest formatting effect I measured. I was not testing for it, and it is now the experiment I would rather have run.

Scope

None of this is evidence about frontier models. A 321M model raised on synthetic reasoning traces has a narrow, tidy typographic world; anything trained on scraped web text has read an enormous amount of shouting and may well have the prior this one cannot. Don't carry the effect sizes anywhere.

If you want the cheap version for a model whose corpus you can read: count the convention before you test it. Grepping SYNTH for IMPORTANT took a minute and predicted every behavioural result below it.

Code, data and the full writeup, including the two errors I caught mid-run and what they cost, are in oaustegard/experiments#56.