SCV: Light and Effective Multi-Vector Retrieval with Sequence Compressive Vectors

Compressing document tokens into span vectors for fast multi-vector dense retrieval with coarse-to-fine search

COLING 2025 Industry

Cheoneum Park1,*, Seohyeong Jeong2, Minsang Kim2, KyungTae Lim3, Yong-Hun Lee2

1Hanbat National University 2SK Telecom 3Seoul National University of Science and Technology

*Corresponding author

Paper (ACL Anthology) PDF
Architecture diagram: query tokens q1 to qn pass through a query encoder into green query vectors; document tokens d1 to dm pass through the SCV encoder into a smaller number of pink span vectors; each query vector takes a MaxSim over the span vectors and the MaxSim scores are summed.
Figure 1. Sequence Compressive Vectors (SCV) architecture overview. Document tokens are compressed into fewer span vectors, and each query token is matched to its most similar span (MaxSim); the per-token maxima are summed into the relevance score.

Abstract

Recent advances in language models (LMs) has driven progress in information retrieval (IR), effectively extracting semantically relevant information. However, they face challenges in balancing computational costs with deeper query-document interactions. To tackle this, we present two mechanisms: 1) a light and effective multi-vector retrieval with sequence compression vectors, dubbed SCV and 2) coarse-to-fine vector search. The strengths of SCV stems from its application of span compressive vectors for scoring. By employing a non-linear operation to examine every token in the document, we abstract these into a span-level representation. These vectors effectively reduce the document’s dimensional representation, enabling the model to engage comprehensively with tokens across the entire collection of documents, rather than the subset retrieved by Approximate Nearest Neighbor. Therefore, our framework performs a coarse single vector search during the inference stage and conducts a fine-grained multi-vector search end-to-end. This approach effectively reduces the cost required for search. We empirically show that SCV achieves the fastest latency compared to other state-of-the-art models and can obtain competitive performance on both in-domain and out-of-domain benchmark datasets.

Keywords Multi-vector retrieval Late interaction (ColBERT) Dense retrieval Span representation Index compression Coarse-to-fine search Retrieval latency RAG retriever MS MARCO / BEIR

Method

SCV encoder for the sentence 'Sejong the Great was the inventor of Hangul': a passage encoder produces token vectors, overlapping sliding windows of three tokens are combined into span vectors for 'Sejong the Great', 'Great was the', and 'inventor of Hangul'.
Figure 2. SCV encoder for span representation (window width W = 3 in this example): contextual token vectors from the passage encoder are pooled over overlapping sliding windows into one vector per span.

Token-level multi-vector retrievers such as ColBERT index every document token, which makes indexes large and search slow, and usually need an approximate-nearest-neighbour (ANN) stage. SCV keeps late interaction but over far fewer vectors:

  1. Span compression. A sliding window of width WW with overlap rate rate\mathit{rate} turns a document of mm tokens into l=m−W(1−rate)W+1l = \frac{m-W}{(1-\mathit{rate})W} + 1 spans. Each span vector combines start and end token features, their product, sum- and max-pooled token vectors, and an attention-selected salient word, passed through a feed-forward layer with GELU.
  2. Late-interaction scoring. Query tokens are matched to document spans with MaxSim: f(Q,S)=∑i=1nmax⁡k=1,…,lhqi⊤hskf(Q, S) = \sum_{i=1}^{n} \max_{k=1,\dots,l} h_{q_i}^{\top} h_{s_k}
  3. Multi-task training. The span loss is trained jointly with a token-level loss and a single-vector [CLS] loss (L=LSCV+Ltok+αLcls\mathcal{L} = \mathcal{L}_{SCV} + \mathcal{L}_{tok} + \alpha \mathcal{L}_{cls}), using in-batch, pre-batch, and BM25 hard negatives, plus about 180k GPT-4-generated questions for MS MARCO passages.
  4. Coarse-to-fine search. At inference the same model first retrieves the top-N documents (N = 10k) with its single [CLS] vector, then rescores them with span-level MaxSim, without an external ANN module. A cross-encoder can optionally rerank the results.

The backbone is DistilBERT-base; span width is 8 in training (16 for indexing MS MARCO), overlap rate 0.2, and output dimension 128.

Main Results

  • Fastest multi-vector retriever. SCV answers a query in 0.86 ms on GPU, about 207× faster than ColBERT (178 ms), 4.6× faster than CITADEL (3.95 ms), 3.7× faster than CITADEL+ (3.21 ms), and about 1.5× faster than the single-vector DPR-768 (1.28 ms).
  • Compact index. The SCV index is 30 GB, close to DPR-768 (26 GB) and about 5.13× smaller than ColBERT (154 GB).
  • Competitive accuracy. On TREC DL 2019, SCV reaches 0.645 nDCG@10 with only BM25 hard negatives (DPR-768: 0.611); on 13 BEIR datasets it averages 0.436 nDCG@10 (ColBERT: 0.453) and beats ColBERT on ArguAna, NFCorpus, and Touché.
  • Strong reranking pipeline. SCV with a MiniLM cross-encoder reaches 0.744 nDCG@10 on TREC DL 2019, above monoBERT (0.723), SimLM (0.741), and ListT5 (0.718), and close to RankLLaMA-7B (0.756) with a 220M-parameter pipeline.
Table 1. In-domain evaluation on TREC DL 2019 with index size and search latency (query encoding plus search). Bold follows the paper.
ModelTREC DL 19Index (GB)Latency (ms/query)
nDCG@10R@1k
Models trained with only BM25 hard negatives
BM250.5060.7390.67×
DPR-7680.6110.742261.28
COIL-tok0.6600.80952.546.8
ColBERT0.6940.830154178
CITADEL0.6870.82978.33.95
SCV (ours)0.6450.712300.86
Models trained with further methods
coCondenser0.6740.820261.28
ColBERT-v20.7440.88229122
ColBERT-PLAID0.7440.88222.155
CITADEL+0.7030.83026.73.21

BibTeX

@inproceedings{park-etal-2025-scv,
    title = "{SCV}: Light and Effective Multi-Vector Retrieval with Sequence Compressive Vectors",
    author = "Park, Cheoneum  and
      Jeong, Seohyeong  and
      Kim, Minsang  and
      Lim, KyungTae  and
      Lee, Yong-Hun",
    editor = "Rambow, Owen  and
      Wanner, Leo  and
      Apidianaki, Marianna  and
      Al-Khalifa, Hend  and
      Eugenio, Barbara Di  and
      Schockaert, Steven  and
      Darwish, Kareem  and
      Agarwal, Apoorv",
    booktitle = "Proceedings of the 31st International Conference on Computational Linguistics: Industry Track",
    month = jan,
    year = "2025",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.coling-industry.63/",
    pages = "760--770"
}
More projects from KAIST MLP Lab →