← Journey

Retrieval Collapses a 25x Parameter Gap

The most important finding in the research: a small model with retrieval routinely beats a model 25 times larger without it.

The result that made the whole architecture feel feasible wasn’t from one paper. It showed up repeatedly across different groups, different tasks, and different years. When the same result appears independently across that much variation, it’s not a cherry-picked finding. It’s a property of the system.

Three Studies, Same Conclusion

The clearest example is RETRO, from DeepMind in 2021. They trained a 7.5B parameter model with access to a 2-trillion-token retrieval database. On language modeling benchmarks, it matched GPT-3 at 175B parameters. Not close to it, not within 10 points of it. Matched it. That’s a 23x parameter gap erased by architecture.

A year later, Atlas from Meta AI pushed further. An 11B model with retrieval beat PaLM at 540B on TriviaQA (84.7 vs roughly 81) and beat GPT-3 at 175B on MMLU (65.6 vs 60). These are knowledge-heavy benchmarks, which is exactly where you’d expect the bigger model to win. It didn’t.

The tool use numbers are similar. TinyAgent from Berkeley showed that a 1.1B model matches GPT-4-Turbo at 79% on a tool benchmark. The technique was ToolRAG: instead of giving the model all available tool schemas (which floods the context and confuses small models), you retrieve just the relevant tool schemas for each query. The model sees 3 or 4 schemas instead of 30.

Why It Works: The 2-Bit Ceiling

The mechanism behind this is the 2 bits/param ceiling from Allen-Zhu’s work at Microsoft Research. Language models store knowledge in their weights with a hard capacity ceiling of roughly 2 bits of information per parameter.

A 7B model has about 14 gigabits of knowledge budget, which sounds like a lot until you consider how much text exists. More importantly, that knowledge is frozen at training time. It can’t be updated without retraining, it can’t be cited, and it can’t be easily audited.

Retrieval sidesteps all of this. Facts live in the database, not the weights. The model doesn’t need to memorize them. It only needs to know how to use whatever gets retrieved. That’s a much smaller capability, and it generalizes much better to domains the model never saw during training.

The Cost Is Negligible

The embedder part of the retrieval pipeline is almost free in compute terms. All-MiniLM-L6-v2 is a 22M parameter model that encodes 14,000 sentences per second on CPU and fits in 80 MB of RAM. A query embedding takes under 5ms. Total retrieval overhead, end to end, is around 5ms per query. Generation of the answer takes seconds. So retrieval is not the bottleneck.

The pattern across all of these results is the same: move the knowledge and exact-compute requirements out of the model, into retrievable external stores, and the model only needs to reason over what it retrieves. Reasoning over a small context is something a small model can do well. Knowing everything is something only a large model can do, and only if you’re willing to freeze that knowledge at training time.

The Gap That Remains

The gap that remains is real and worth naming. Small models still lose on agentic coding (7-8B is at 19-23% on SWE-bench vs frontier at 77%), on the hardest reasoning (GPQA Diamond), and on tasks requiring multi-hop implicit world knowledge.

Retrieval closes the factual knowledge gap. It doesn’t fully close the reasoning gap on the hardest tasks. We’re not hiding that.

But for grounded question answering, research tasks, document analysis, and anything where the answer exists somewhere and you can retrieve it, a small model with retrieval is genuinely competitive with much larger ones.

Key takeaways:

Next: building the first version that puts all of this together.

← All entries