A 4-Bit Model and a 1-Bit Index
You can run NVIDIA's 4-bit Nemotron embedder on a CPU it was never meant to touch, quantize its output to a 1-bit index 32× smaller, and stack the two compressions with almost no loss — they act on different resources and don't interfere.
Nemotron-3-Embed-1B ships in two formats: a BF16 model, and an NVFP4 one whose linear weights are 4-bit, documented for NVIDIA GPUs and vLLM only. But NVFP4 is a weight encoding — each weight is a 4-bit value, packed two to a byte, times an 8-bit block scale, times one global scale. I reconstructed the weights arithmetically and ran the model on a CPU with no GPU in the box. The dequantized weights match the BF16 twin at cosine 0.9955; the embeddings they produce agree at 0.9959. (NVFP4 also quantizes activations; I ran those in full precision, so this is the weight-quantization effect, which dominates.) On SciFact retrieval and STS-B similarity the model quantization costs about zero nDCG — it changes which documents come back, 95% overlap with the float model's top-10, but not their relevance, because the swaps are among near-equivalent results.
That is the model. The other half of the bill is the index — the vectors you store and scan, one per document. I compressed Nemotron's 2048-dimensional embeddings every way I had, at matched bytes per vector: remax sign codes, remex Lloyd-Max scalar quantization, int8, and product quantization.
Quality retained vs. index size, higher is better; upper-left is the frontier. remax/remex points are means over five random rotations, error bars smaller than the markers.
No method owns the whole ladder; each owns a regime. remex Lloyd-Max owns 512–1024 bytes — at 2–4 bits per coordinate with a stored norm, it reconstructs the vector more faithfully than the same bytes spent on sign-rotations. remax sign codes own 64–256 bytes: Lloyd-Max can't drop below 512 bytes at full dimension (2 bits × 2048), while remax's 1-bit code and its Matryoshka-slice path reach down to 64 bytes and stay well above the alternatives. Product quantization edges remax only at the very bottom, 32 bytes, and pays with a trained codebook and a corpus to train it on. int8 is nearly lossless but only 4× before it must drop dimensions, at which point retrieval falls apart.
The remax-versus-remex corner of that chart is a question I've asked twice before. One Bit Beats Two found 1-bit sign codes beat 2-bit and 3-bit Lloyd-Max on SPECTER2; the Jina sequel found the result inverts on a general-purpose embedder and pinned the cause on geometry — tight clusters reward the single sign boundary, near-isotropic vectors reward Lloyd-Max's finer levels. Nemotron is general-purpose, and it lands with Jina: the Lloyd-Max middle is healthy here, 2/3/4-bit rank-monotone and slightly ahead of stacked sign codes at matched bytes (nDCG 0.839 vs 0.835 at 512 bytes). A third data point on the same axis, not a new surprise. (One wrinkle specific to this run: the original SPECTER2 inversion was partly a nesting artifact — shaving bits off a stored 8-bit code — which I sidestepped by fitting each bit width independently. The geometry story is still the load-bearing one.)
The four embedding-quant families alone. int8 ties remax on similarity but collapses on retrieval below 2 KB; PQ trails on similarity and needs training; remax holds the aggressive end.
The durable sign-bit advantage, across all three models, isn't "1 beats 2" — it's the aggressive regime nothing else reaches at full dimension, plus being data-oblivious: no codebook, no training corpus, no fit, fully determined by (dimension, seed).
Which brings the two halves together. Running remax 1-bit on the NVFP4 embeddings scores the same as running it on BF16 (nDCG 0.823 vs 0.820). Model-weight quantization and embedding quantization touch different resources — GPU memory for the model, RAM for the index — and compose without interference. So "quantize the 4-bit model down to one bit" was never one operation. It's two, on two different things, and they multiply: the model's 4 bits live on the accelerator, the index's 1 bit lives in memory, and neither needs to know about the other.
Full method grid, per-seed error bars, latency, and the NVFP4 CPU-reconstruction code are in the reference archive and in remax PR #52.