The Benchmark
We ran qwen3:1.7b bare vs inside Metis on the same 14 questions. Same weights, same hardware, different architecture.
The thesis needed a number, not a story. So we ran one: take one model, run it two ways, measure what changes. The only variable was the architecture.
Setup
The model: qwen3:1.7b at 4-bit quantization. The two conditions: call it directly through ollama with no retrieval and no verification loop, then run it through Metis with a local corpus it has never trained on. Same weights, same hardware, same questions.
We used a private document (sample-docs/zephyr.md) with invented facts: specific memory limits, a mascot named Pippa, a ratification year, a parent organization. Nothing the model could have memorized during training. Then we added four questions with no answer in any document (to test whether the system fabricates) and two general questions including one exact arithmetic problem.
Bare Model Results
For every corpus fact question, it invented an answer. A few examples:
- The ratification year: “2015” (wrong, invented)
- The mascot: “a winged horse” (wrong, the document says a blue heron named Pippa)
- A chairperson who doesn’t exist: “John W. Lott”
- A headquarters location: “San Francisco”
- The multiplication 84937 x 2261: “191,737,397” (off by about 300,000)
On the four unanswerable questions, it fabricated confident answers to every one. It invented a person, a location, a programming language (“Scala”), and a certification price.
Metis Results (Same Weights)
8 out of 8 corpus facts answered correctly, each with a citation to the source document. All four unanswerable questions returned “I don’t have reliable information about that” with empty sources. The multiplication returned the correct answer via the calc tool.
| metric | bare | Metis |
|---|---|---|
| answerable facts correct | 0 / 8 | 8 / 8 |
| fabrications on unanswerable | 4 / 4 | 0 / 4 |
| general (incl. exact math) | 1 / 2 | 2 / 2 |
Zero datasets. Zero retraining. The same model.
The result reproduced identically on the live Railway deployment, with average latency around 3-5 seconds for verified answers and up to 21 seconds on the search path (when the first candidate fails verification and the system tries more candidates).
Lessons and Caveats
The decimal question is worth calling out separately. The initial extractive fast-path was getting it wrong: it retrieved “caps resident memory at 1” instead of “1.84 GB”. The sentence splitter was treating the period inside the decimal as a sentence boundary. The fix was to only split on punctuation followed by whitespace. Decimals never have whitespace after the dot. One line change, one regression test added. These are the kinds of bugs that only show up on real data.
The architecture is the differentiator. Not the model. The model in isolation is about as reliable as a confident person who hasn’t checked their sources and won’t admit uncertainty. The same model inside Metis checks its sources, cites them, and stays quiet when it can’t.
Whether this generalizes beyond our test corpus is an open question. The result is on 14 questions over one document on one model. We’re not claiming it’s a comprehensive study. We’re claiming the mechanism works on the test case it was designed for, and the mechanism is sound.
The next question is whether the same result holds on open-domain questions answered from web search rather than a local document. So far: yes, with the same caveat that source quality matters. The verifier checks whether the answer is entailed by the source, not whether the source is trustworthy. Junk sources produce answers that pass the entailment check but might still be wrong. That’s the next problem to work on.
Key takeaways:
- Same weights, same hardware: 0/8 correct bare vs 8/8 inside Metis on grounded facts; 4 fabrications vs 0
- The architecture is the differentiator, not the model; retrieval and verification do the work the weights can’t
- Source quality is the next unsolved problem: the verifier checks entailment, not trustworthiness