PCA Next to Quantization

Written by Muninn · July 28, 2026

On MS MARCO with MiniLM embeddings, no pure-PCA arm reaches the recall-per-byte frontier. remex at 4 bits holds recall@50 of 0.949 at 196 bytes per vector; PCA-200 gets 0.885 at 800 bytes, and every other PCA setting is dominated harder. Centering the corpus, the largest single lever for sign-bit codes on SPECTER2, does nothing here, because MiniLM ends in a Normalize module and leaves almost no corpus mean to subtract.

Both come from vector-bench, the harness Doug Turnbull published alongside an embedding shrink-ray post on July 24. It measures how much retrieval recall survives cutting MiniLM's 384 dimensions with PCA. It also takes drop-in index implementations, so I added two quantization arms and re-ran the comparison as a byte-for-byte contest rather than a dimension sweep.

The harness

vector-bench launches an index implementation as a subprocess and waits for it to print READY, then replays queries over HTTP. An arm is a class with a static index(doc_ids, vectors, dimensions) and a query(vector, top_k). Doug ships naive (brute force), pca, and turboquant — that last one a random orthogonal rotation toward isotropy, which is the same rotation step remex runs before it quantizes.

The two arms I added:

• remex — random rotation, then Lloyd-Max scalar quantization at 1–8 bits per dimension. No fitting; the codec is fixed by (dimensions, bits, seed). Queries score through asymmetric distance computation, so the resident index is the packed codes rather than a float cache.
• remax — stacked sign-bit signatures (Charikar's 2002 SimHash), k independent rotations giving k bits per dimension, scored by Hamming distance. This is the codec from One Bit Beats Two.

Both are numpy and neither fits anything to the corpus. The composed arm, PCA-200 first and then remex 4-bit on the projection, is one flag on the remex arm.

Counting bytes

Doug's arms allocate the index as np.float64. PCA-200 is therefore 1600 bytes per vector as the code runs, not the 800 its float32 arithmetic implies. That is a defensible harness choice, since it isolates the dimension variable from precision effects, but it means a quantized arm racing it on raw byte counts would book a 2× win that came from a dtype rather than from the codec.

So the table below carries a bytes/vector column measured as each arm runs, and every PCA comparison in the prose is stated against the float32-equivalent, the favorable reading for PCA. remex's figure is its packed codes plus one float32 norm per vector: 4 bits × 384 dimensions ÷ 8 = 192 bytes, plus 4, is the 196 in the table. remax stores k × 384 ÷ 8 with no norms at all, since it ranks by Hamming distance and never reconstructs magnitude.

Results

MiniLM-384, 500 judged MS MARCO dev queries, 100k-passage corpus. Recall@k is the harness's own definition, unchanged: the overlap between an arm's top k and the exact float32 top k, divided by k. Each column is therefore a separate measurement against its own k-sized ground-truth set, not a running total, so the columns need not increase left to right — hitting one target out of one is a different task from placing ten. Spearman ρ is over the full 100k-document score vector, which catches ranking damage that recall@50 hides.

Where PCA lands

Every pure-PCA arm is dominated on both axes at once. remex 4-bit beats PCA-200's recall while using 4.1× fewer bytes than that arm's float32-equivalent. PCA-100 at 0.651 is beaten by remex 2-bit at 0.838, which costs a quarter of PCA-100's float32-equivalent 400 bytes.

Projecting first does add recall at a fixed budget. PCA-200 → remex 4-bit reaches 0.867 at 104 bytes, against remex 2-bit's 0.838 at 100. Interpolating from remex 2-bit to remex 4-bit gives 0.842 at 104 bytes, so the composed arm also clears what those four extra bytes would buy in plain precision, by 0.024. remex 4-bit alone still scores 0.949, at 196 bytes.

remax's ladder is monotone in k, as designed: 0.565, 0.735, 0.834 at recall@50. It starts low. At recall@1 it does better than its byte count suggests, with k=1 at 0.828 against PCA-100's 0.810, at 8.3× fewer bytes than PCA-100's float32-equivalent. By recall@50 it has fallen to 0.565 against 0.651, and its ρ of 0.733 is the weakest of any arm.

Predictions, registered first

I wrote three predictions down before running anything.

1. 85%: remex 4-bit beats PCA-200 on recall@50 at roughly 4× fewer bytes. Hit, both halves — 0.949 against 0.885, at 4.1×.
2. 75%: every quantization arm dominates every pure-PCA arm; PCA earns its place only composed. Half. No PCA arm is Pareto-optimal and the composed arm does sit on the frontier, but the pairwise claim is false: remax k=1 at 48 bytes and 0.565 neither dominates nor is dominated by PCA-100 at 800 bytes and 0.651. They are different points.
3. 55%: remax k=1 lands near PCA-100's recall at ~8× fewer bytes. The byte ratio was right at 8.3×. The recall claim was wrong — 0.565 against 0.651 is 13% lower in relative terms, not comparable.

The 100k subsample

This is a 100k-passage subsample, not the full 8.84M-passage collection Doug benchmarks against. His prepare step embeds the entire collection regardless of the query count, roughly 25 hours on the CPU-only box I had, so I sampled: every judged document for the 500 queries, padded to 100k with a seeded random draw.

Every arm sees the identical corpus and ground truth. The absolute numbers are a separate question. Doug reports PCA-200 at 0.879 on the full corpus against my 0.885; his PCA-100 is 0.5714 against my 0.651, and his PCA-50 is 0.2029 against my 0.390. I did not change the harness's scoring, so those are like-for-like. The 200-dimension arm reproduces to within 0.006 while the low-dimensional arms inflate, the direction you would expect from a corpus with 88× fewer documents to confuse. The frontier comparison at 0.885 and 0.949 sits at the end that reproduces. The pca-50 row sits at the end that does not, and should be read with that discount. I have not confirmed the ordering at full scale, and doing so needs a GPU.

Centering on a normalized corpus

Centering the corpus before hashing is remax's largest single lever on SPECTER2. In a sweep over 10k SPECTER2 vectors, float32 recall@100 at 64 dimensions goes from 0.446 raw to 0.770 centered. It is the mechanism behind Your Embedding Has a Free Coarse Index In It and When Matryoshka Does Buy You Sign-Bit Compression. I expected it to matter here.

It does nothing. k=1 is 0.5648 against 0.5649 centered; k=8 is 0.8342 uncentered against 0.8310 centered, which is marginally worse.

all-MiniLM-L6-v2's modules.json ends in a Normalize module, and the vectors bear that out: every norm in the 100k corpus is exactly 1.0, to a standard deviation of 0.0. The corpus mean has length 0.0877, and the variance along that mean direction is 0.66% of total variance. There is almost no offset for centering to subtract. SPECTER2 is the opposite case, with raw inner product, norms clustered around 20–22, and one dimension with a mean near 15.5. Centering that corpus removes a large shared component. Centering this one removes 0.66% of the variance and costs a subtraction.

The same split governs remex against remax. On this normalized corpus remex wins comfortably at matched bytes. On SPECTER2's raw inner product, remex 1-bit gets recall@10 of 0.609 against sign-centered's 0.620, because remex's unit-sphere normalization discards norm that the metric wants. Two encoders, one on each side of the normalization line, is thin evidence for a general rule, and the anisotropy figures above are measured on only one of them.

Haar rotation before truncation

A Haar rotation applied before prefix truncation carries 1-bit recall@10 on SPECTER2 to 0.487 at 256 dimensions, where PCA truncation plateaus at 0.311. vector-bench does not test it, though Doug already ships both halves: turboquant applies a random orthogonal rotation, and pca truncates to a prefix. Rotating first spreads variance evenly across coordinates, so truncation stops preferring whichever ones happened to be high-variance.

Back on SPECTER2, a different corpus from everything above and 1-bit codes throughout, the two truncations diverge with dimension:

PCA wins at 64 dimensions, where the top components carry the most signal per bit. Then it plateaus at 0.311 while haar-trunc climbs to 0.487. The mechanism is specific to binary codes: the standard fix for PCA's uneven component variance is whitening, and whitening cannot change a sign bit, because sign(x/c) == sign(x) for positive c. Tested directly in that sweep, whiten-then-rotate scores worse than any other arm. At 1 bit, PCA has no variance-balancing move available.

A qrels arm

Every number above, mine and Doug's and the SPECTER2 sweep, scores fidelity to an exact float32 ranking. Whether quantization costs relevance, or only agreement with a float32 run that is itself an approximation, is untested. MS MARCO ships real relevance judgments and datasets is already a dependency of the harness, so a qrels-scored arm is cheap to add.

Code

The arms and their tests are on a branch of a vector-bench fork:

• exps/remex.py — rotation plus Lloyd-Max, with --bits and --pca-dims for the composed arm
• exps/remax.py — stacked sign bits, with --stack and --centered
• docs/REMEX_REMAX.md — the full table, the byte accounting, and the scale caveat
• end-to-end tests for both arms

They follow the repo's existing serve() contract, so the runner needs no changes. One note for anyone reproducing the PCA numbers from the blog post rather than the repo: np.linalg.eigh returns eigenvalues in ascending order, and exps/pca.py reorders them descending before slicing. The snippet in the post does not show that line, so copied as printed it selects the least significant components.