A LateOn index pooled to a quarter at 4 bits keeps 99% of its SciFact score at 1/28 the size
Correction, 2026-11-12: I first scored these indexes by taking each query's dot product with decoded token vectors. remex's own index search scores the same codes differently — it scales each vector's residual by its stored length and divides by the expected cosine of a code — and at 1 bit that ranks better. Every quantized score below now comes from remex's RemexIndex search, and every size is remex's own byte count. Every token at 1 bit scores 0.749 on SciFact, not 0.718. Pooling's margin over it fell from 0.040 to 0.012, and only the half-the-tokens-at-2-bits margin now has an interval clear of zero. The quantizer-matched cut and the explainer video were rescored the same way.

A LateOn index that keeps a quarter of each document's token vectors at 4 bits scores 0.760 nDCG@10 on SciFact in 4.2 KB per document. The uncompressed index, 32-bit floats, scores 0.763 in 118 KB, and every token at 1 bit scores 0.749 in 5.5 KB. On NFCorpus the same three indexes score 0.377, 0.382 and 0.370. LateOn is a 149-million-parameter ColBERT model that LightOn released under Apache 2.0. Like any ColBERT model it stores a vector for each token — about 230 vectors of 128 numbers for a SciFact abstract — and the index shrinks two ways: pooling merges similar token vectors within a document, and quantizing stores each number in fewer bits.
The explainer is a Kinescript scene that Claude Code built, one JSON file of layers cued to the words of the narration. Kokoro speaks it in my voice, a blend of two of its preset voices, and Kinescript 2.5 rendered the 4,049 frames in 34 seconds on four CPU cores.
ColBERT scores a document by matching each query token to its best document token
Each query token is compared with every token vector in the document and keeps its highest similarity, and the document's score is the sum of those maxima. That rule is called MaxSim. A specific word keeps a vector of its own, so the D in "vitamin D" can find the D in a document instead of being averaged into one vector for the whole text. Pooling merges the vectors that sit closest together first, so near-duplicates go early and a token unlike its neighbours is among the last to be merged.
Half the tokens at 2 bits beat every token at 1 bit by 0.012 on SciFact in 16% fewer bytes
Half the tokens at 2 bits, 4.6 KB per document, scored 0.012 higher than unpooled 1-bit codes at 5.5 KB on SciFact (95% interval 0.001 to 0.022). On NFCorpus it scored 0.008 higher (0.003 to 0.014) at 4.8 KB against 5.7. A quarter of the tokens at 4 bits came out the same on SciFact, 0.012 higher, and 0.006 higher on NFCorpus, but both of its intervals start at −0.001. A quarter of the tokens at 2 bits, 2.3 KB, tied unpooled 1-bit codes at less than half their size. These are four of the five pooled settings measured; the fifth, a quarter at 8 bits, costs more bytes than 1-bit codes. The quantizer rotates vectors at random before rounding them, so every score is the mean over four random rotations.
Unpooled 8-bit codes score the same as floats. Against them, a quarter of the tokens at 4 bits is −0.002 on SciFact and −0.005 on NFCorpus, both intervals spanning zero.
Claude Code on the web ran the comparison on 2026-11-12 at Oskar Austegard's request, on two BEIR sets: SciFact (5,183 abstracts, 300 queries) and NFCorpus (3,633 documents, 323 queries). The experiment follows Tom Aarsen's Token Pooling-Aware Training (2026-11-11), which trains a model so that pooling costs it less; LateOn was never trained that way. Pooling was the Ward clustering that his post describes and sentence-transformers 6.3 ships as HierarchicalTokenPooling: each document's tokens are grouped, and each group is replaced by its mean. Quantizing was remex, which rotates each vector and rounds every coordinate to one of 2b levels. LateOn's unit-length token vectors point nearly the same way — their mean has length 0.972 — so remex ran in its centered mode, which subtracts that mean first. The sizes are remex's own byte counts: the codes plus two 4-byte numbers per vector that centered scoring needs. remex 2.2.0 takes the pooling module directly, as RemexIndex(model, bits=4, center=True, token_pooling=HierarchicalTokenPooling(pool_factor=4)). Every query was scored against every document, with no shortlist, and query-time cost was not measured. The full results list every arm, interval and rotation.
LateOn loses 0.010 to its last bit and under 0.005 to pooling by four
The NeoMME post of 2026-09-07 recommended quantizing token vectors without pooling. On NeoMME the 2-bit and 1-bit codes both scored 0.707 on SciFact, so the last bit cost nothing, and pooling by four cost 0.026 even in floats (0.694 against 0.720). On LateOn the step from 2 bits to 1 costs 0.010 on SciFact and 0.0085 on NFCorpus, intervals clear of zero, and pooling by four in floats scores 0.764 against 0.763 on SciFact and 0.377 against 0.382 on NFCorpus, both intervals spanning zero. The NeoMME run used the same pooler with an earlier remex release, and I have not checked which scorer produced its numbers, so the two models are not compared like for like. The experiments repo's methods ledger says to measure both numbers on the unpooled index before choosing between fewer tokens and fewer bits.
The quantizer-matched cut is inconclusive at +0.009 on SciFact and −0.003 on NFCorpus
Claude Code also tried tying the amount of pooling to the quantizer. It cut each document's merge tree where merging would move the tokens as far as quantization already does, measured from remex's own error at each bit width, so documents with many near-duplicate tokens pool further than dense ones. At 3 bits the cut keeps 73% of the tokens and scores within 0.002 of unpooled 4-bit codes on both sets at 56 to 57% of their size.
At 2 bits the cut keeps 38% of the tokens, 3.5 KB per document. It scored 0.769 on SciFact, above the float index, against 0.759 for uniform pooling to the same token count: a gain of 0.009 (0.000 to 0.019). On NFCorpus it scored 0.371 against 0.374, a difference of −0.003 (−0.008 to 0.001). Against a quarter of the tokens at 4 bits, the 3.5 KB cut is +0.009 on SciFact and −0.006 on NFCorpus, both intervals spanning zero. At 1 bit the cut over-pools and keeps only 10% of the tokens. The 2-bit result is the best of 16 settings of the cut (four bit widths, four thresholds), picked after the runs, so it is a hypothesis for a third collection to test.
The same tokens scored in floats split the same way. The cut's selection beats the full index by 0.013 on SciFact (0.002 to 0.024) and trails it by 0.007 on NFCorpus (0.002 to 0.012). Uniform pooling to a quarter showed no such split: +0.002 and −0.004 in floats, both intervals spanning zero. The cut is 0.6 KB per document smaller than a quarter of the tokens at 4 bits and carries a split I can't explain. On a new collection I would use uniform pooling: pool_factor=2 at 2 bits, or pool_factor=4 at 4 bits. I don't know what separates the two collections.