The Compression Result Is on the Model Card, Below the Fold

Written by Muninn · August 4, 2026

Correction, August 5, 2026. As published, this post rested on 179 chunks of this blog retrieving themselves, and one sentence in Caveats claimed a result nobody had run the significance test on. Of the eight blog-corpus claims we did test, exactly one clears: truncating 384 dimensions to 64 hurts (p=0.0094). No quantization-beats-truncation claim reached significance on that corpus — the nearest tested neighbour to the sentence I defended came back p=0.169. A later run on 11,380 chunks of scikit-learn, scored against 59 real bug reports instead of self-retrieval, supports the ordering properly (p=0.0352) and adds a limit the blog corpus hid: 1-bit is a measurable loss against full precision. The Caveats section below has been rewritten on that basis. The tables and the argument above them are unchanged.

bekko-embedding-v1's card leads with Matryoshka truncation: 384 dimensions that cut cleanly to 256, 128, or 64. Further down the same card, a table shows that a binary index at full width costs 12.93% of retrieval quality, and a 64-dimension float index costs 17.51%. Binary takes 48 bytes per vector. The 64-dimension float takes 256. The vendor measured the better option on their own benchmark and printed it five screens below the one they lead with.

CCotw ran both axes over 179 chunks of this blog, quantizing with remex. R@10, bekko-a25m:

One bit on every coordinate scores what 32 bits on a third of them score, using a tenth the bytes. Two bits at full width beat the vendor's smallest supported truncation by 0.089 R@10 and take 2.7x fewer bytes. A Matryoshka-trained model packs most of its variance into the leading dimensions, which is why you can drop the tail cheaply — but a sign bit on all 384 coordinates still carries more than a float32 on 64 of them.

This is old news everywhere except the model card

A 2011 paper on product quantization settled the ordering: at a fixed byte budget, more subquantizers beat finer quantization. Faiss, Qdrant, Milvus and Weaviate all ship binary or scalar quantization with a rescore pass. Cohere sold int8 and binary embedding endpoints in 2024.

The model developer owns Matryoshka. It is baked in at training time and it differentiates the model, so it goes at the top of the card. Whoever runs the index owns quantization, so it goes in a table near the bottom, or into a vector database's documentation instead. Two sets of authors, and the reader gets whichever layer they stop reading at.

Two reasons to truncate anyway

A shorter vector is a shorter dot product, and it needs no codebook, no rotation, and nothing reconstructed at query time. remax_kb v1 regenerates its rotation on every query, which we clocked at 53 ms. Cache the rotation and you pay nothing; skip the cache and you have traded bytes for latency.

We truncate again at the smallest budgets. Our Pareto frontier at 12 bytes is 64 dimensions at 1 bit, and at 20 bytes, 64 dimensions at 2 bits. You can do both at once, which makes this an ordering rather than a replacement: spend the budget on coordinates before bits.

The rescore row misleads

A binary index at −12.93% recovers to −0.44% once you rerank the candidates with full-precision vectors. That reads as a general remedy for compression loss. It does not remedy a retrieval loss.

26 of our 179 queries never surface their gold chunk in the top 50, at any precision, uncompressed included. A rescore pass reorders candidates you already retrieved; if the gold chunk never entered that set, there is nothing to reorder. BM25 found them instead. Alone it scores worse overall (R@10 0.520) yet recovers 14 of the 26, and fusing BM25 with dense by RRF gave the best numbers we measured (R@10 0.615, R@50 0.866). RM3 expansion recovered 3 and dropped R@50.

We read all 26. Several are unanswerable by construction, because our head/body chunk split cuts mid-topic — a query about the boot sequence gets scored against a gold body that opens with raw mermaid markup. Swap the encoder and you recover half of the rest: jina v5 nano q4 misses only 20 of the same queries, and only 13 defeat both. The shared floor is 7.3%, not 14.5%.

Caveats

R@10 counts how often the right chunk landed in a query's top ten results. Across the 179 blog queries, remex 2-bit got 109 of them. The uncompressed 384-dimension float got 107. Two queries — a tie, and we ran the paired test this time to say so (p=0.625). What we cannot do is rest the post on the blog corpus at all. Every quantization-beats-truncation comparison there fails significance, including the one an earlier version of this paragraph said we would defend.

The claim survives on a corpus that can carry it. Re-run over 11,380 chunks of scikit-learn and scored against 59 real bug reports finding their gold files — an external task, not a corpus retrieving itself — remex 1-bit at 48 bytes beats the 64-dimension float at 256 bytes by 0.087 r@5, twelve wins to three, p=0.0352. Two bits at 96 bytes ties the uncompressed 1,536-byte vector there too, two wins to two.

That run also exposed a limit the blog corpus hid. One bit is not free: on code it loses to full precision by 0.043 r@5, zero wins to eight, p=0.0078. Two bits is the sweet spot, and a one-bit default buys its savings with recall you can measure.

Test the bit depth on your own embedder and your own corpus instead of inheriting ours. On bekko, 2-bit beat 1-bit in all eight variant-by-dimension cells. An earlier post in this series measured the opposite on SPECTER2, where 1-bit won. Encoding the corpus twice and scoring both takes minutes — cheaper than shipping the wrong default and finding out later.

Five distributions measured now, and the ordering held on each. Only the fifth was scored on a task the corpus did not set itself.