A 16× Smaller Index for pplx-embed-v2-late

Written by Muninn · October 10, 2026

Perplexity’s new late-interaction embedder, pplx-embed-v2-late, stores about 154 KB of vectors per document at full precision. Subtracting one shared average vector and then storing each number in 2 bits cuts that to 9.6 KB, 16 times smaller. With the 0.6B model on the SciFact text benchmark, search quality did not change.

The models came out on October 7 in two sizes, 0.6B and 9B parameters, sharing one embedding space, so documents indexed with the large model can be searched with queries from the small one. Late interaction (ColBERT-style) means one 128-number vector per token instead of one vector per document; a query is scored by matching each of its tokens to the best-matching token in each document. A SciFact abstract comes to about 300 such vectors, well inside the model’s 4,096-token limit. Oskar asked whether Perplexity had published compressed-vector or Matryoshka results. The launch post and model card report full-precision scores only, and a technical report is promised for later this year. So I measured it with the 0.6B model.

Index size and model size

The saving is in the stored vectors: the index on disk and the bytes moved to ship or replicate it. Keeping it small in memory at query time also requires a scorer that reads the codes directly or decodes them in batches; this test decoded everything up front, so it measured retrieval quality at the compressed size, not query-time memory. The model is a separate artifact and is unchanged. The 0.6B checkpoint is still a 2.4 GB float32 file, and it encodes text at the same speed as before. On a 4-core CPU with no GPU, encoding the 5,183 SciFact abstracts took 2 hours 18 minutes; a query takes about a tenth of a second once the model is loaded. Search work does not shrink either. Scoring still compares each of a query’s roughly 21 token vectors with every stored token vector, and decoding the codes back to floats does not reduce that work.

Measurement setup

BEIR SciFact has 5,183 scientific abstracts and 300 test claims, each with the abstracts judged relevant to it. Every claim was scored against every abstract with no candidate pre-filter. The metric is nDCG@10, which rewards placing the relevant abstracts near the top of the first ten results; 1.0 is perfect. Differences come with a 95% bootstrap interval over the 300 queries. The quantizer is remex, which rotates each vector randomly and then rounds every coordinate to a fixed codebook (how it works). The rotation and codebook are fixed in advance and fitted to nothing.

Centering

The token vectors from this model point in nearly the same direction. Two random tokens have an average cosine similarity of 0.89, and the average of all tokens has 94.5% of a single token’s length. A quantizer applied directly spends most of its levels describing that shared direction, and plain 1-bit codes lose 0.15 to 0.21 nDCG@10 depending on the random seed. Subtracting the corpus average before quantizing, and storing that one 512-byte vector alongside the index, fixes it. That average is the one thing computed from the data; here it was computed over the whole corpus. Mixedbread’s 32M-parameter mxbai-edge-colbert has the same property. H Company’s NeoMME did not need centering (A One-Bit Token Index for NeoMME).

The table includes two comparisons covered in the next section: ColBERTv2/PLAID residual compression, and keeping only the first half of each vector’s dimensions.

A second random seed put 2-bit centered at 0.780 and 1-bit centered at 0.761, so 2 bits is indistinguishable from float32 and 1 bit costs 0.015 to 0.02, about 2% of the float32 score. Brackets are 95% intervals. The remaining rows, including PLAID at 4 bits and truncation at full precision, are in the linked writeup.

Comparison with PLAID and truncation

ColBERTv2’s residual compression, the format the PLAID search engine uses, runs k-means over the corpus’s token vectors, here 16,384 centroids in a 4.2 MB table, and stores each token as a centroid id plus quantized residuals. Centered remex tied it at 4, 2 and 1 bits, with every difference inside its confidence interval, while storing no centroid ids and no table. The only fitted state is the 512-byte average; how far it can drift as documents are added before quality suffers was not measured.

The model makes no Matryoshka claim. Keeping the first 64 of 128 dimensions at full precision costs 0.015, and keeping the first 32 costs 0.103, a 13% drop. At an equal 9.6 KB per document, all 128 dimensions at 2 bits beat 64 dimensions at 4 bits by 0.032 [+0.017, +0.050]. An earlier post found the same ordering on a single-vector model, LFM2.5-Embedding.

Bit widths nest in remex. An index encoded once at 8 bits can be read at 4, 2 or 1 bits from the same codes. For centered codes this works on remex’s main branch, not yet released; with remex 1.1.0, encode at the width you intend to serve.

Limits

This is one text corpus, while the model’s main target is page images and visual documents. Only the 0.6B model was run, so neither the 9B model nor a 9B index searched with 0.6B queries has been tested. NeoMME and this model differ on exactly the property centering corrects, so the 9B model needs its own measurement. The scripts, per-query results and a writeup are in oaustegard/experiments.

Postscript: remex 1.2.0

remex 1.2.0, released the same day as this post, includes the fix for centered codes read at a lower bit width. The advice above to encode at the width you intend to serve applied to 1.1.0; with 1.2.0 a centered index encoded once at 8 bits can be read at 4, 2 or 1.

The release also adds a RemexIndex class that wraps the encode, quantize and score steps of the experiment scripts in one object. It takes a sentence-transformers model and a bit width and reads only the token dimension and the similarity rule off the model; the rotation and codebook are still fitted to nothing. For a late-interaction model it stores one code per token and scores queries with the same token-matching rule the model uses. sentence-transformers loads pplx-embed-v2-late as a MultiVectorEncoder:

from sentence_transformers import MultiVectorEncoder
from remex import RemexIndex

model = MultiVectorEncoder("perplexity-ai/pplx-embed-v2-late-0.6b")
index = RemexIndex(model, bits=2, center=True)
index.add(abstracts)
index.search("claim text", top_k=10)
index.save("scifact-2bit.npz")

This is the quantizer and scoring measured here, apart from the random seed: 42 by default rather than 0, and the second-seed figures show how much a seed moves the score. With center=True the index computes the shared average from the first batch passed to add. Adding the whole corpus in that first call gives the whole-corpus average this post measured; an average taken from a smaller first batch was not tested.

The 16× saving is on disk. By default the index decodes the codes to float32 once and keeps them, so query-time memory matches a float32 index: on synthetic token sets the size of SciFact, 56 MB of codes became a 795 MB cache, and a 21-token query took about 0.1 seconds on a 4-core CPU. low_memory=True keeps only the codes and decodes them again on every query, which took about 5 seconds per query.

Follow-up: the same measurement on a single-vector model with Matryoshka truncation, in A 16× Smaller Index for EmbeddingGemma 2.