The Tie Broke (and We Found Something Better)
We built a harder benchmark to test whether 0.6B really equals 1.7B. It doesn't. But the bigger surprise was an architecture bug that was quietly costing the larger model nine right answers.
The last entry ended with a promise and a caveat. The promise: a 0.6B model matched a 1.7B on grounded question answering. The caveat: the benchmark was 14 questions over one document, mostly answerable by copying a sentence out of the text. We said the honest next step was to make the benchmark hard enough to find the floor, and that this was exactly where a smaller model should start to crack.
So we did that. And it cracked. But not before showing us something we weren’t looking for.
A Harder Benchmark
We wrote four fictional documents that reference each other: a technical spec, a governance document naming who maintains what, a changelog with version history, and an unrelated biology document to force retrieval to tell domains apart. Every fact is invented, so neither model could have seen it in training. Then 42 questions, sorted into five tiers by difficulty.
The crucial design decision was separating two kinds of hard. Some questions need reasoning over facts the system can pull in a single retrieval — comparing two memory budgets, adding two numbers, picking the largest. That isolates the model: the evidence is right there, can it reason over it? Other questions need chaining across documents — find the component with the largest budget, then find the working group that maintains it, then find who chairs that group. That tests the whole system, because no single retrieval gathers the full chain.
If we mixed those two together, a failure would be uninterpretable: is the model too small, or did retrieval just not fetch the right pieces? Kept apart, the numbers actually mean something.
What We Expected to Find
We expected the tie to hold on easy questions and break on hard ones. That’s roughly what happened — but the first run was muddied by something else entirely, and chasing that down turned out to be the real result.
On the first run, both models scored terribly on the synthesis questions. The 1.7B got 3 out of 10. That made no sense; these are questions like “which component has the smallest memory budget,” answerable by reading four numbers that all sit in the same chunk. A 1.7B model does not fail that by being too small.
Looking at the per-question output, almost every synthesis failure had the same signature: the answer came back in a tenth of a second, as a raw slab of the source document, with no reasoning in it. The system never asked the model anything. It had taken a shortcut.
The Shortcut That Was Lying
Metis has a fast path. Before involving the language model at all, it checks whether any retrieved chunk is very similar to the question — cosine similarity above a threshold. If so, it returns that chunk directly. The idea is that for a simple lookup, you don’t need to spend a model call; the answer is sitting in the text.
The problem is that “similar to the question” is not the same as “answers the question.” Ask “which has a larger budget, Aster or Quill” and the chunk describing Quill scores very high on similarity — it contains both the word budget and the word Quill. So the fast path returned that chunk and never did the comparison. Worse, on questions whose answer isn’t in the corpus at all, a topically-near chunk could still clear the threshold, and the system would hand it back as an answer — fabricating, while bypassing the very verification step that’s supposed to make Metis trustworthy.
So we made the threshold configurable and turned the fast path off, sending every question through the full generate-and-verify loop. The 1.7B went from 19 correct out of 30 to 28 out of 30. Fabrications dropped from 2 to 0.
| fast path on | fast path off | |
|---|---|---|
| 1.7B answerable (of 30) | 19 | 28 |
| 1.7B fabrications | 2 | 0 |
That swing — nine answers and two fabrications — is larger than the entire difference between the two model sizes. The single biggest lever on quality wasn’t the model at all. It was a latency optimization that had quietly been trading away correctness, and trading away the one property the whole project is built on: not making things up.
The fix isn’t to delete the fast path. It’s to only take it for genuine single-fact lookups, never for comparisons, arithmetic, or absent facts. That’s the next piece of work.
And the Tie? It Broke.
With the fast path off — the fair comparison, both models reasoning on every question — here is where the 0.6B stands against the 1.7B:
| tier | 1.7B | 0.6B |
|---|---|---|
| extractive | 12 / 12 | 10 / 12 |
| synthesis (single-doc reasoning) | 9 / 10 | 7 / 10 |
| multi-hop (chain across docs) | 7 / 8 | 3 / 8 |
| unanswerable (should abstain) | 8 / 8 | 5 / 8 |
| general + math | 4 / 4 | 3 / 4 |
The 1.7B wins, 28 to 20. Phase 2’s “they’re equal” was an artifact of an easy benchmark, exactly as we worried. But the gap isn’t spread evenly — it lives in two specific places.
The first is multi-hop chaining: 7 versus 3. The 0.6B cannot reliably hold a chain of facts together across documents. When the chain gets long, it gives up and abstains. The 1.7B keeps the thread.
The second is knowing when to shut up: 8 versus 5 on the unanswerable questions, with three fabrications from the smaller model and zero from the larger. Recognizing that an answer simply isn’t in the evidence turns out to be its own skill, and it scales with size.
Where they stay close is extraction and single-document reasoning — looking something up, comparing two numbers that sit side by side. For that, the small model is genuinely fine.
What This Establishes
It corrects the record. The 0.6B is not a free replacement for the 1.7B; on hard questions the larger model is clearly better, and we can now name precisely where: multi-hop reasoning and abstention discipline. That’s a more useful result than the tie was, because it tells us what a smaller model actually buys and what it costs.
It also turned up the more important finding by accident. The fast path — a piece of the architecture we’d never questioned — was the dominant quality bug, costing more than model size and undermining the system’s central promise. We only saw it because the benchmark was hard enough to make it fail loudly. The easy benchmark had been hiding it the whole time.
That’s the argument for harder benchmarks in one picture: the easy one told us a flattering thing that was wrong, and hid a real bug. The hard one corrected the flattering claim and surfaced the bug. We’re keeping the hard one.
Key takeaways:
- The 0.6B does not equal the 1.7B on a hard benchmark; the larger model wins 28 to 20, with the gap concentrated in multi-hop reasoning and knowing when to abstain
- The biggest quality lever was not model size but a cosine-similarity fast path that returned raw chunks without reasoning or verification, costing the 1.7B nine right answers and two fabrications
- A good benchmark earns its keep by falsifying your flattering results and exposing the bugs the easy one hid
Follow the Metis journey for the fast-path fix and multi-hop retrieval.