The Bet
Frontier models waste most of their parameters storing facts. What if you put the facts somewhere else?
There’s a number that keeps coming up in the research: about two bits of stored knowledge per parameter. Allen-Zhu’s group derived it from first principles. The implication is that a 7B model has roughly 14 gigabits of knowledge capacity, and almost all of it is being spent on memorized facts from the training set. That’s the wrong thing to spend it on.
The Three-Cost Trap
Today’s frontier models are three things welded into one: a reasoning engine, an encyclopedia, and a calculator. That design forces three costs onto the world.
The monetary cost is obvious: you need enormous GPU clusters to run these things, so the only people who can afford to operate them become your intermediaries. The privacy cost is less discussed: every query goes to someone else’s server. The access cost is the quietest: most of the world can’t pay what the cloud vendors charge.
The knowledge stored in the weights is mostly geography, history, trivia, and encyclopedic content that any external database could store for a fraction of the cost. The actual reasoning circuits, the part that figures out how to use information, are a smaller core buried inside. Nobody separates them because training them jointly is easier. But joint training is not a law of physics.
A Different Architecture
The alternative isn’t a smaller version of the same thing. It’s a different architecture.
Pull the encyclopedia out of the weights and put it on disk, where storage is cheap and you can read, update, and audit it. Pull the calculator out and make it an actual calculator that doesn’t approximate. Leave the reasoning circuits resident in RAM, small and fast. Add a coordination layer that knows when to consult each piece.
The result: a system that runs on a $5/month VPS. That runs on a laptop. That runs offline. The knowledge is a file on disk that you can swap, version, and trust. The same binary becomes a medical assistant or a legal assistant or a coding assistant by changing the library.
The Research That Makes It Feasible
The research already validated each part independently.
RETRO showed that a 7.5B model with retrieval matches GPT-3 at 175B on language modeling benchmarks. Atlas showed that an 11B model beats PaLM at 540B on knowledge-heavy tasks. TinyAgent showed that a 1.1B model matches GPT-4-Turbo on tool-use benchmarks by retrieving the relevant tool schemas per query rather than loading all of them into context.
Nobody had assembled all three pieces into one portable system that fits on hardware you already own. That’s what we’re building.
What Comes Next
We’re calling it Metis after the Greek Titaness of practical wisdom, the one whose counsel even Zeus sought. The idea is the same: win by cunning, not by size.
The next question is whether the constraints of cheap hardware force an architecture clean enough to actually work. That requires getting honest about the physics first.
Key takeaways:
- Frontier models fuse reasoning, memory, and arithmetic into one expensive, opaque blob
- Separating these concerns lets you run a capable AI on hardware you already own
- The research validates each piece independently; Metis assembles them into a working system
Follow the Metis journey to see how each piece comes together.