Routing products tend to be compared against the most expensive baseline. We ran the one that actually threatens us, and reported what it did.
The flattering way to benchmark a router is against the most expensive thing a customer might otherwise do. Always use the premium model, show that routing is cheaper, quote the percentage. Every number in that comparison is true and the comparison is close to worthless, because almost nobody actually runs every request through the most expensive model available.
The baseline that threatens a router is the boring one: pick a good mid-tier model and always use it.
32 graded cases across three models, seven trials per cell, pooled across six repeat runs. Always-premium: 99.1% success at $0.001535 per success. A hand-rolled task-tier heuristic: 98.7% at $0.001318. Always-balanced: 98.7% at $0.000641. Our router at a 0.75 quality floor: 95.1% at $0.000350. At a 0.85 floor: 98.7% at $0.000674.
So the router has the best cost-per-success of anything tested, and it does not dominate always-balanced. At 0.75 it trades 3.6 points of success for roughly half the cost. At 0.85 it matches on success at about 5% more. That is a dial, not a free lunch.
The hand-rolled heuristic is itself dominated by simply using one balanced model — worth knowing before anyone builds one.
Three structural reasons, none of them a bug in the harness. One model in the pool is near Pareto-optimal on its own: Sonnet scored 32/32 at 2.6 times less than Opus, and when a single model dominates, routing has little room to add value.
Measurements are also sparse — one or two trials per endpoint and task type — so the quality posterior stays prior-dominated and a cheap model cannot yet earn a high enough lower bound to be selected for easy work even where it is in fact reliable. And the economy-tier prior is deliberately pessimistic, which locks the cheapest model out above roughly a 0.65 floor regardless of evidence.
A later run put all six providers through the same suite — 32 graded cases per model, five repeat runs, roughly 3,900 graded trials. Nine of the nineteen routable models scored 98% or better.
That is a problem, not a triumph. A benchmark almost everything passes has stopped separating the top of the field while still looking like evidence, and any routing comparison run on it is measuring noise at the high end. The honest next step is harder cases, not a better headline — so treat every quality figure here as a floor on what these models can do rather than a ranking of them.
With no quality floor set, the router correctly reduces to "cheapest" and matches the always-economy baseline exactly. Any comparison at that setting is vacuous. The floor is the dial, and quoting a single point on that curve without the rest of it would be cherry-picking — so the full sweep is published alongside.
What would change the verdict: many more trials per cell, so a cheap model's bound can rise where it genuinely performs, and a more heterogeneous pool where no single model dominates. Until then this is what we can show.
Do not quote a savings figure against always-premium without also showing always-balanced. Omitting the strongest simple baseline is how routing products flatter themselves.
Four semantic approaches, measured against a deterministic ranker on the same catalogue and the same tasks. None of them won, and one of them — our own first instinct — was materially worse.
We nearly deleted a working component because it benchmarked badly. The defect was one layer below it, in code nobody was looking at.
We split the value of tool governance into the part the policy filter does and the part the ranker does. They are not close.