How Small Can the Model Go?
If the architecture does the work, the model should be replaceable. We swapped the 1.7B generator for a 0.6B one, 62% less model, and measured what broke.
The benchmark from the last entry made a claim: the architecture is the differentiator, not the model. If that’s true, there’s an obvious test. Make the model smaller and see if the result survives. If quality holds when you shrink the generator, the architecture really was carrying the weight. If it collapses, the model mattered more than we said.
Three Configurations
We ran three configurations on the same 14 questions, the same private corpus, the same hardware. The only variable was which model generates and which model verifies.
- E0: qwen3:1.7b generates, the same 1.7b verifies. The baseline from last entry. 1.36 GB on disk.
- E1: qwen3:0.6b generates, the same 0.6b verifies. 0.52 GB on disk. A 62% smaller model doing both jobs.
- E2: qwen3:0.6b generates, a dedicated 67M NLI model verifies. The “specialist” idea: a model trained only on entailment should verify better than a generalist.
The hypothesis going in was that E1 would degrade (the 0.6B is a much weaker model) and E2 would recover the loss (the specialist verifier compensates). Both predictions were wrong, and the way they were wrong is the interesting part.
E1: The Smaller Model Did Not Degrade
| grounded facts (8) | fabrications (lower is better) | avg latency | |
|---|---|---|---|
| E0 – 1.7B | 8 / 8 | 0 / 4 | 0.57s |
| E1 – 0.6B | 8 / 8 | 1 / 4 | 0.58s |
On the grounded surface, the 0.6B scored identically to the 1.7B. Not close. Identical: 8/8, through the exact same internal paths (four answered by the extractive fast-path, four by the generate-and-verify loop).
E1 loses exactly one point overall, and it’s worth looking at which one. The question was “In which city is the Edge Compute Consortium headquartered?” – a fact that is in no document. With no evidence to retrieve, the verification loop never engages; there’s nothing to check an answer against. So the model answers from its raw weights, and the 0.6B fabricated “San Francisco” where the 1.7B happened to hedge.
That failure is real, but it’s not on the surface the system is built for. It happens in the one path where the architecture is switched off. Where the architecture is on, the two models tie.
This is the result. A model 62% smaller, costing a fraction of the RAM, matches the larger one on grounded question answering because the model barely runs. Of the 14 questions, the generator only produces an answer on four of them. The rest are handled by cosine similarity over the corpus, by rule-based abstention, or by a tool. You can’t lose much quality by shrinking a component that’s idle most of the time.
E2: The Specialist Verifier Failed
The clever idea was supposed to be E2. Instead of using the language model as its own fact-checker, use a small model trained specifically on Natural Language Inference. A 67M-parameter specialist, the reasoning went, should beat a generalist at its one job.
It scored 4/8 on the grounded facts. It got worse, badly.
The failure was systematic, not random. The NLI model kept rejecting answers that were correct. Here’s the shape of it:
- Evidence: “The Zephyrian reference implementation, codenamed Marlowe, targets…”
- Answer (from the 0.6B): “The reference implementation is codenamed Marlowe [1].”
- NLI verdict: NEUTRAL – not entailed.
The answer is correct. But it’s a paraphrase, with a citation marker bolted on, and a model trained on clean NLI sentence pairs reads that as “I cannot confirm strict entailment.” So the system abstained on four facts it actually had right.
The lesson is sharper than “the specialist didn’t work.” It’s that verifying a RAG answer is not the same task as textual entailment. Off-the-shelf NLI models are trained to be strict about sentence-to-sentence logical entailment. RAG verification needs something looser and more semantic: does this answer, phrasing and citations and all, reflect what the evidence says? The language model in judge mode already does this well, because it understands paraphrase. The 67M specialist doesn’t, because it was trained for a subtly different question.
A dedicated RAG verifier could still be the right call, but it would have to be trained on pairs of (evidence, answer), not on generic NLI data. That’s a different project, and not one we need to start, because the generalist-as-judge already works.
What This Establishes
It establishes that shrinking the generator from 1.7B to 0.6B costs nothing measurable on this grounded surface, which supports the thesis that the architecture, not the model, is doing the work. And it establishes a clean negative result: generic NLI models do not transfer to RAG verification.
It does not establish that the 0.6B is a free lunch in general. The same caveat as last time applies, only harder. Fourteen questions, one document, one corpus that happens to be answerable mostly by extraction. We have not tested the floor – the questions where the answer can’t be copied out and the generator actually has to reason. That’s exactly where a smaller model should start to crack, and our current benchmark is too easy to find it.
So the honest status: the cheap result is in, and it’s encouraging. The expensive result – does the tie survive a harder, larger benchmark – is the next thing to measure, and the one that would turn this from an encouraging anecdote into something worth standing behind.
Key takeaways:
- A 0.6B generator matches a 1.7B on grounded facts because the architecture handles most queries before the model ever runs
- Generic NLI models fail at RAG verification because paraphrase plus citation markers breaks strict entailment checking
- The honest status: promising on this surface, not yet tested on harder reasoning where the gap should appear
Follow the Metis journey to see the next benchmark.