AI evaluation and model routing: test before you trust
How to test an AI system before it carries real work: golden sets, eval harnesses, a second reader, and the cheapest model that clears your bar. Results come with the cost per thousand decisions.
Jev 1.13 Tested: Its Router’s Accuracy for 2 Cents per 1,000 Decisions
TypeSafe’s Jev 1.13 only decides; it never writes. Picking AI model sizes for 94 of our tasks, it scored 73 to its own router’s 72, in a fifth of a second and for about 2 cents per 1,000 decisions.
A Second Reader Catches What the First One Missed. Not What Retrieval Missed.
Two models answering independently from the same retrieved passages catch each other’s misreadings, because their mistakes are not the same mistakes. What neither of them can catch is a passage that should never have been retrieved — and that is the outcome the mechanism reports as healthy.
The Cheapest Model That Clears Your Eval Bar Is the Correct Model
Most teams pick a frontier model in the kickoff meeting and never revisit it. That is not a technology decision — it is the absence of one. Here is what a routing harness actually looks like, and the three places a cascade quietly breaks.
AI Integration Projects Need a Check on the Output, Not a Better Model
AI integration projects mostly stall on workflow, not on the model, according to MIT’s 2025 study. Before scaling one, build a check that catches a wrong answer before it reaches a customer, a regulator or print.