Autoregressive retriever training raises BEIR scores

DREAM: Autoregressive Retriever Training

Contrastive training still dominates dense retrieval despite its data hunger. DREAM takes a different approach by deriving supervision directly from the next‑token prediction objective of a frozen LLM rather than from pre‑constructed positive and negative pairs [1].

Most dense retrievers such as DPR, ANCE, or ColBERT rely on contrastive loss built from mined query‑document pairs, often requiring billions of examples and expensive mining pipelines.

Training on three backbone sizes-0.5 B, 1 B, and 3 B parameters-DREAM consistently outperforms existing baselines on BEIR and RTEB [1].

Method and Integration

The method hinges on injecting retriever scores into selected attention heads of a frozen LLM, a non‑standard alteration that integrates the retriever with the LLM’s attention mechanism [1].

Open Questions and Implications

It also leaves unanswered how well the approach scales to multimodal corpora or ultra‑large models beyond 3 B.

If the claim holds, the default pipeline for retrieval‑augmented generation can drop the contrastive data construction stage and adopt a frozen‑LLM supervised retriever as the new baseline.

References

[1] DREAM: Dense Retrieval Embeddings via Autoregressive Modeling