Teaching the Shortcut Some Restraint
The fast path was answering questions it had no business answering. The fix wasn't to remove it — it was to teach it which questions are actually shortcuts.
The last entry found a bug worth more than model size. Metis has a fast path: when a retrieved chunk looks very similar to the question, it returns that chunk directly without involving the language model, saving a couple of seconds. The trouble was it fired on questions that needed reasoning — “which component has the smallest budget” matches the chunk listing the budgets, so it returned that chunk and never did the comparison. Turning the fast path off entirely fixed the quality but cost the latency on every simple lookup, which is the whole reason the fast path exists.
So the real fix isn’t on/off. It’s teaching the shortcut to recognize which questions are genuinely shortcuts.
A Cheap, Precise Guard
We added a small check that runs before the fast path, on the question text alone. If the question carries a marker of reasoning, the fast path steps aside and the question goes through the full generate-and-verify loop. Three kinds of marker:
- Comparison and superlatives — larger, smallest, strictest, longer, between, than. These need at least two facts weighed against each other.
- Aggregation — combined, total, sum, difference. These need arithmetic the copied sentence won’t contain.
- Chaining — maintained by, codenamed, chaired by. These pivot from one entity to ask about another, which a single sentence rarely answers.
The check is deterministic, costs nothing, and is deliberately high-precision: it only steps aside when a marker is clearly present, so a plain lookup like “what is the codename of the Aster component” keeps its tenth-of-a-second path. We unit-tested it in both directions — it must fire on the reasoning questions and stay quiet on the lookups.
What It Bought
Measured on the 1.7B against the hard benchmark, comparing the naive always-on fast path, the new type-gated fast path, and the fast-path-off ceiling:
| tier | always-on | type-gated | off (ceiling) |
|---|---|---|---|
| synthesis | 3 / 10 | 9 / 10 | 9 / 10 |
| multi-hop | 6 / 8 | 7 / 8 | 7 / 8 |
| total correct (of 42) | 29 | 36 | 40 |
| avg latency | 1.27s | 1.53s | 1.8s |
The guard recovers seven of the eleven points the naive fast path was losing — the entire synthesis collapse, fixed — while keeping the fast latency on real lookups. It pays a quarter-second over the always-on path and saves a quarter-second against running the full loop on everything. That’s the trade we wanted: pay for reasoning only when the question needs it.
The Part We Didn’t Fix, and Won’t Pretend We Did
The type-gated path scores 36, not the 40 ceiling. The missing four points are two separate, smaller bugs, and naming them precisely is more useful than papering over them.
Two are genuine lookups where the extractor picks a bad sentence — a truncated span, a section header instead of the line below it. The guard correctly leaves these alone, because they really are lookups; the problem is in how the sentence gets chosen, which is a different repair.
The other two are the more uncomfortable ones. Ask for a fact that isn’t in the corpus at all — “who is the treasurer of the foundation” — and a topically-near sentence can still clear the similarity bar. The fast path returns it, bypassing the abstention the rest of the system is built to enforce. No keyword guard catches this, because the question looks exactly like a normal lookup. The honest fix is a cheap verification step on fast-path answers, not a longer list of trigger words. That’s the next piece.
So: one bug closed cleanly and measured, and the remaining gap narrowed from one diffuse problem into two specific ones. That’s the shape of progress we’re after — not a single number going up, but the unknowns getting smaller and better named.
Key takeaways:
- The fix for an over-eager optimization is usually a precondition, not a removal: teach the fast path which questions are actually shortcuts
- A zero-cost, high-precision text guard recovered seven of eleven lost points and the full synthesis collapse, while keeping fast-path latency on real lookups
- The remaining gap is now two named bugs — bad span selection and unverified fast-path answers on absent facts — instead of one vague one
Follow the Metis journey for verified fast-path answers and multi-hop retrieval.