Two-Model Cross-Checks in RAG: What They Catch

The failure that finally gets a retrieval system taken seriously is rarely an invented document. It’s quieter than that. The answer comes back fluent and confident, with a citation; someone checks, because someone eventually does, and the passage is real. The clause exists, it says roughly what the answer claimed, and the answer is still wrong. The model read the right page and took the wrong thing from it — or read a page about something adjacent and found what it needed — and you find out in a review meeting, or worse, from someone outside your team.

The natural response is to check the answer before it goes out. A better prompt won’t do the checking — the model that misread the passage will misread it again with more encouragement. The useful question is what does the checking. One pattern we’ve built: leave retrieval exactly as it is, then have two models answer the same question from the same retrieved evidence, independently, and compare their answers before anything is returned or cited.

Two readers, not a vote

The gain comes from a single property: the two models’ errors are imperfectly correlated. One reader catches the unsupported claim the other let through — the assertion no retrieved passage actually backs. One notices the passage the other skipped, the qualifier that changes the answer. One spots the reasoning slip: right passages, wrong inference. A single model won’t flag any of these itself; it has no vantage point on its own reading.

None of this is averaging, and it isn’t a vote. The answers aren’t blended into a compromise; the comparison is a check, not a synthesis. If both models failed in exactly the same places, the second reader would add nothing at all.

Which is why independence has to be real. Show the second model the first one’s answer and you haven’t built a second reader, you’ve built a reviewer with a prior — it anchors on what it was shown and nods. Running the same model twice and hoping for a different opinion buys much less, for the same reason: a model’s mistakes correlate strongly with its own. That is still a better signal than asking a model how confident it is, which is close to useless — but the distance between two samples of one model is small, and the distance between two different models is where the check gets its value.

It’s worth separating the cross-check from a pattern it gets conflated with. Hybrid retrieval is two retrievers over one corpus — dense, meaning semantic similarity, and keyword — whose results are merged to improve what gets found. The cross-check is one retrieval, then two evaluators reading what was found: a verification layer above retrieval, not a claim about retrieval.

We’ve built both, at different layers, because they fix different failures: private retrieval-augmented generation (RAG) over 24,000-plus documents in 1,100-plus vector collections, dense and keyword retrieval together inside the compliance boundary, and then two models answering from the same retrieved evidence with the answers compared.

Disagreement is the signal

Comparing the answers is harder than it looks. Two answers can agree in substance and differ in wording — “the clause permits this” against “this is allowed under section four” — or agree in wording while differing in substance, both citing the same passage while one quietly goes further than the passage does. A naive text diff can’t tell those apart: it flags the first as a conflict and waves the second through.

The workable shape is to have both models emit claims tied to passage identifiers — structured assertions, each pinned to the passage it rests on — and compare those. You get something genuinely diffable. As a side effect, every claim in the returned answer carries its own citation, which is the format an auditor wants anyway.

Then the point the whole mechanism hangs on: agreement is cheap. Two models will agree most of the time, and that agreement tells you very little. The value sits entirely in what happens when they differ, and there are four honest options:

retry retrieval with a different query, on the assumption the evidence is incomplete;
escalate to a stronger model for a third reading;
return the answer marked low-confidence, with the disagreement left visible;
route it to a person.

A system that records an agreement rate and does none of these has thrown the mechanism away. It’s paying for two inferences and keeping one opinion. The comparison was the cheap half of the design; the routing is the mechanism.

The human route is the one teams underprice. An escalation queue isn’t a feature flag; it’s a queue, an owner and a response time, indefinitely. Someone has to clear it while the asker waits, or it becomes a compost heap of unanswered escalations. Those are salaries, not tokens, and they want a line in the budget before the build, not after the first incident.

What the cross-check cannot see

Now the caveat, stated as plainly as the benefit, because it matters as much. The cross-check does nothing when the retrieved evidence is itself missing, stale, irrelevant or misleading. Two models reading the same bad context will agree on the wrong answer, with matched confidence, and the mechanism reports health. It checks the answer against the evidence. Nothing in it checks the evidence against the world.

So there are three outcomes, not two: agree, disagree, and agree-on-thin-evidence. The third is the one the pattern cannot see. If the corpus is out of date, both readers are corroboratively out of date, and the dashboard glows green. Coverage, freshness and retrieval quality remain their own disciplines, handled by source curation, refresh cycles and the retrieval layer itself — and the cross-check absolves none of them.

The economics are straightforward. The check costs roughly a second inference on every question answered, plus whatever the escalation path costs. Latency needn’t double — the two readers can run in parallel — but the spend does. The obvious cheaper variant is to run the cross-check only above a risk threshold: two readers where a wrong answer is expensive, one where it isn’t. That turns “twice the cost of everything” into “twice the cost where being wrong costs more than the inference.”

What a team running this pattern measures is not an accuracy score. It’s the disagreement rate, what the escalations turn up, and how often the human route overturns both models. Those are the numbers that say whether the mechanism is earning its spend, and unlike an accuracy figure they can be read straight off the system you are actually running.

Where it earns its place: regulated or auditable work, where a wrong answer is expensive and a citation has to hold up under a hostile reading months later. Not general chat, and not low-stakes internal tooling, where a wrong answer gets noticed and corrected in the flow of work and costs someone a mildly wasted afternoon. In the first setting the escalation queue pays for itself; in the second it’s overhead dressed as rigour.

The decision this leaves isn’t whether two readers beat one. It’s what a wrong answer costs you, and whether disagreement has somewhere to go. If the honest answers are “not much” and “nowhere”, one reader and a clear low-confidence flag will do. If a citation has to survive an audit, the second inference is the smallest part of the bill — the queue is the commitment.