Benchmarks

Best cost-per-success of any strategy, but it does NOT dominate always_balanced — at 0.75 it trades 3.6 points of success for half the cost, and at 0.85 it matches on success at roughly 5% more. What it does beat outright is always_premium and the hand-rolled heuristic, and that heuristic is itself dominated by simply using one balanced model.

With no floor the router correctly reduces to 'cheapest' and matches always_economy exactly — any comparison at that setting is vacuous. The floor is the dial, and quoting a single point on this curve without the rest of it would be cherry-picking.

798 is 21 dangerous tools exposed on all 38 tasks — the naive baseline sends every money-moving and destructive tool on every request. Every policy-respecting strategy is zero structurally, because the filter runs before ranking. Value splits as policy 3775 → 809 tokens, then ranking 809 → 292: most of the benefit is the filter, not the ranker.

At this size lexical and keyword selected identical sets on all 38 tasks — verified by diff, not inferred from equal metrics. The sophisticated ranker bought nothing here, and at 60 tools it initially did worse than substring matching. The cause turned out to be a missing stemmer one layer below, not the ranker.

Semantic features must demonstrate repeatable benefit or be disabled. Three independent measurements failed to beat deterministic ranking, so the infrastructure stays built, tested and off by default. MMR was our own first instinct and it was the worst option of the five.

A cap that can be exceeded under concurrency is not a cap. The check and the reservation share one transaction behind a lock on the budget scope, and the comparison runs in SQL numeric rather than JavaScript floats — money comparisons in float64 are how an off-by-a-fraction breach goes unnoticed.

Read the saturation before the spread. Nine of these score 98% or better, which means starter-v2 no longer separates the top of the field — a benchmark almost everything passes measures very little while still looking like evidence, and the honest next step is harder cases, not a louder claim. What the suite still supports is the cost axis, where the spread is real: gemini-3.7-flash passed every case at roughly a ninth of Sonnet's cost per success, and glm-5.3-flash at about a twenty-third. That is an argument for routing which did not exist while the pool was effectively one vendor — measured in August, routing was Pareto-efficient but did not beat simply always using one good mid-tier model. Three caveats travel with these numbers: this is our suite and not your workload, so a task type absent from it is unmeasured rather than proven; the two Ollama models carry no pricing row, so they have no cost/success figure and would fail closed under a cost ceiling whatever their quality; and glm-4.5-air threw 14 infrastructure errors, correctly excluded from the pass rate above but a reliability signal that a quality column is the wrong place to look for.