Checking Is Cheap. Saying No Is Not.
We tried to push Metis's founding bet — that verifying is cheaper than generating — to its logical extreme, and measured it. The bet is real, but it splits in two: recognizing a true answer is free, rejecting a plausible false one is not, and the second half doesn't scale with model size.
A few entries ago we leaned on a single idea: checking an answer is cheaper than producing one, so a small model that can’t reliably generate a correct answer can still reliably recognize one. That asymmetry is the whole reason Metis verifies instead of trusts. Today we tried to push it as far as it goes — and found it isn’t one asymmetry. It’s two, and only one of them is on our side.
The Bet We Were Testing
Retrieval taught us that knowledge doesn’t have to live in a model’s weights — you can keep it on disk and fetch it. The natural next question: does reasoning have to live in the weights either? What if the language model isn’t “the reasoner” but a cheap part that only ever does one tiny thing — check a single fact against the evidence — while an external engine handles the chaining, the planning, the backtracking?
That only works if a small model can verify atomic steps reliably, even when it can’t chain them. The promise is that you decompose a hard multi-hop question into single hops, verify each one, and compose the answer outside the model — so error never compounds, because every step is gated. Before building any of that, we wrote down the condition that would kill it: if the 0.6B can’t verify a one-hop fact with at least ~90% balanced accuracy, the whole engine is moot. Then we measured.
The Experiment
We held the evidence fixed — the relation tables from our fictional corpus: which component has which codename, which working group maintains it, who chairs that group, what each one’s memory budget is. Then we varied only the depth of the claim we asked the verifier to check: one hop (“the Tessera group is chaired by Dr. Solvang”), two hops (“the group that maintains Lumen is chaired by…”), three hops (“the group that maintains the component codenamed Falconer is chaired by…”). For each depth we mixed true claims with plausibly false ones, where the wrong answer is another real value pulled from the same evidence — the kind of mistake that actually looks like an answer. We ran the real production verifier across three model sizes.
We tracked two things separately, because they turn out to be completely different problems: recall (does it confirm a claim that’s actually true?) and precision (does it reject one that’s false?).
| model | hops | recall | precision | bal. acc |
|---|---|---|---|---|
| 0.6B | 1 | 1.00 | 0.38 | 0.69 |
| 0.6B | 2 | 0.75 | 0.50 | 0.63 |
| 0.6B | 3 | 0.25 | 0.75 | 0.50 |
| 1.7B | 1 | 1.00 | 0.75 | 0.88 |
| 4B | 1 | 1.00 | 0.38 | 0.69 |
| 4B | 3 | 1.00 | 0.50 | 0.75 |
The 0.6B’s balanced accuracy on one-hop claims came out at 0.69 — well under the 0.90 we’d set. By the rule we wrote down in advance, the simple version of the thesis is falsified. Atomic granularity alone does not make a small model a reliable verifier. We’re not going to move the line we drew. But which way it failed is the whole story.
The Half That Held: Recognizing Truth Is Free
Look at the recall column at one hop: 1.00 for every model size. The 0.6B confirms every true atomic fact, exactly as well as the 4B. And recall is precisely what falls apart with depth — the 0.6B drops from 1.00 to 0.75 to 0.25 as the chain grows to three hops, while the 4B holds a flat 1.00 all the way down. That gap at three hops, 1.00 versus 0.25, is the reasoning-depth advantage of a bigger model, isolated cleanly.
But it’s an artifact of asking the small model to verify the whole chain at once. Break that three-hop claim into three one-hop claims and the 0.6B checks each at 1.00. So on this axis, the architecture works: an engine that decomposes can confirm correct steps with a tiny model as reliably as a large one. The expensive part — holding a long chain together — is exactly what you can move out of the weights. This half of “checking is cheaper” is real.
The Half That Broke: Saying No Doesn’t Scale
Now the precision column. At one hop, the 0.6B rejects only 38% of false claims — and the 4B, seven times the size, rejects the same 38%. Precision does not improve with scale. The small model waves through five of eight plausible falsehoods; the big one is less confident about it but still doesn’t catch them. Recognizing that a fluent, on-topic, evidence-shaped sentence is actually wrong turns out to be roughly as hard as generating the right one — and you don’t buy your way out of it with parameters.
This reframes a problem we’d misdiagnosed. Two entries back, the smaller model fabricated more on unanswerable questions, and we filed it under “abstention is a capability that scales with size.” This says otherwise. The fabrication isn’t mainly about size; it’s about verifier precision, and precision is flat across every size we can run. The thing Metis exists to do — say nothing rather than make something up — runs straight into the one axis that scale doesn’t fix.
Two Failures That Are the Same Failure
Back in Phase 2 we tried a dedicated NLI model as a verifier, and it failed by the opposite mistake: it called correct-but-reworded answers “neutral” and over-abstained. The LLM judge fails the other way — it rubber-stamps plausible wrong ones. One has poor recall, the other poor precision. Neither is calibrated; they’re mirror images on the same trade-off. Which finally makes the target precise instead of vague: not “a better verifier,” but one specific number to move — keep recall at 1.00, and drag one-hop precision from 0.38 toward 1.00, on a sub-1B model. The recall half is already free. The entire remaining problem is learning to say no to a plausible lie.
One Bug, Found Because a Number Looked Wrong
The first run had the 4B rejecting everything, at every depth — which made no sense for a model that can clearly read. It was a parsing bug, not behavior: Qwen3 thinks out loud first, and its thinking echoes the instruction back, including the words “SUPPORTED or UNSUPPORTED.” Our parser scanned the whole reply and caught the echoed “unsupported” before the model’s real verdict, flipping every answer to a rejection. In production, that quietly turns the judge into a machine that refuses everything whenever it runs a reasoning model. Fixed: read the verdict from after the thinking block, and a test to keep it fixed.
Key takeaways:
- “Verifying is cheaper than generating” is really two claims: recognizing a true answer is cheap and works at any model size; rejecting a plausible false one is not, and doesn’t scale with size at all.
- The good half is the load-bearing one: decomposing a multi-hop question into single hops lets a 0.6B verify each step as reliably as a 4B, so chaining depth can move out of the weights — recall at one hop was a perfect 1.00 for every size we tested.
- The hard half is now a single measurable target: lift one-hop precision on a sub-1B model from 0.38 toward 1.00 while keeping recall at 1.00 — which is exactly where a verifier that isn’t just another generalist LLM would earn its place.
Follow the Metis journey as we try to teach a tiny model to say no.