Every result below comes from a benchmark in our own repository, re-runnable against the same fixtures. Each table states what was run, how many trials, and when — a figure without those three is an assertion rather than a result.
32 graded cases × 3 models, 7 trials per cell, pooled across 6 repeat runs.
| Strategy | Success | Total cost | Cost / success |
|---|---|---|---|
| always_premium | 99.1% | $0.0487 | $0.001535 |
| task_tier_heuristic | 98.7% | $0.0416 | $0.001318 |
| always_balanced | 98.7% | $0.0202 | $0.000641 |
| planverity_router @0.75 | 95.1% | $0.0107 | $0.000350 |
| planverity_router @0.85 | 98.7% | $0.0213 | $0.000674 |
Best cost-per-success of any strategy, but it does NOT dominate always_balanced — at 0.75 it trades 3.6 points of success for half the cost, and at 0.85 it matches on success at roughly 5% more. What it does beat outright is always_premium and the hand-rolled heuristic, and that heuristic is itself dominated by simply using one balanced model.
Single run, suite starter-v2, 32 cases. Router behaviour as the floor rises.
| Floor | Router success | Router cost | always_balanced | always_economy |
|---|---|---|---|---|
| none | 84.4% | $0.0056 | 100% / $0.0192 | 84.4% / $0.0056 |
| 0.60 | 90.6% | $0.0083 | 100% / $0.0192 | 84.4% / $0.0056 |
| 0.65 | 93.8% | $0.0097 | 100% / $0.0192 | 84.4% / $0.0056 |
| 0.70 | 93.8% | $0.0106 | 100% / $0.0192 | 84.4% / $0.0056 |
| 0.75 | 100% | $0.0220 | 100% / $0.0192 | 84.4% / $0.0056 |
With no floor the router correctly reduces to 'cheapest' and matches always_economy exactly — any comparison at that setting is vacuous. The floor is the dial, and quoting a single point on this curve without the rest of it would be cherry-picking.
38 tasks × 60 deliberately confusable tools, 21 of them in policy-denied classes. Selection only, no model calls.
| Strategy | Recall | Precision | Forbidden exposures | Schema tokens |
|---|---|---|---|---|
| all_tools (no policy) | 100% | 1% | 798 | 3775 |
| all_permitted | 100% | 6% | 0 | 809 |
| keyword | 91% | 15% | 0 | 285 |
| keyword_stemmed | 91% | 15% | 0 | 291 |
| lexical | 96% | 16% | 0 | 292 |
| semantic | 80% | 13% | 0 | 321 |
| hybrid_rrf | 93% | 16% | 0 | 293 |
798 is 21 dangerous tools exposed on all 38 tasks — the naive baseline sends every money-moving and destructive tool on every request. Every policy-respecting strategy is zero structurally, because the filter runs before ranking. Value splits as policy 3775 → 809 tokens, then ranking 809 → 292: most of the benefit is the filter, not the ranker.
Same 38 tasks against the smaller catalogue, before stemming was added.
| Strategy | Recall | Precision | Forbidden exposures | Schema tokens |
|---|---|---|---|---|
| all_tools (no policy) | 100% | 4% | 152 | 1153 |
| all_permitted | 93% | 12% | 0 | 385 |
| keyword | 93% | 17% | 0 | 273 |
| lexical | 93% | 17% | 0 | 273 |
| semantic | 93% | 14% | 0 | 326 |
| hybrid_rrf | 87% | 16% | 0 | 272 |
At this size lexical and keyword selected identical sets on all 38 tasks — verified by diff, not inferred from equal metrics. The sophisticated ranker bought nothing here, and at 60 tools it initially did worse than substring matching. The cause turned out to be a missing stemmer one layer below, not the ranker.
Real text-embedding-3-small vectors over policy-permitted tools, K=5.
| Ranker | Recall @5 | Verdict |
|---|---|---|
| lexical (default) | 3/4 | kept |
| semantic (cosine) | 3/4 | no gain, more tokens |
| hybrid RRF | 3/4 | below its better input |
| MMR λ=0.5 | 1/4 | materially worse |
| MMR λ=0.3 | 1/4 | materially worse |
Semantic features must demonstrate repeatable benefit or be disabled. Three independent measurements failed to beat deterministic ranking, so the infrastructure stays built, tested and off by default. MMR was our own first instinct and it was the worst option of the five.
Live against Postgres. Concurrent reservations issued against a known remaining budget.
| Property | Result |
|---|---|
| 6 concurrent reservations vs remaining budget | 2 granted, 4 refused, cap not breached |
| Exact boundary (4 × 0.25 into a 1.00 cap) | 4 granted, 5th refused |
| Reserve then settle | reserved 0.0000830, settled 0.0000790, headroom returned |
| Rewrite a settled row | refused by trigger |
| Delete a reservation row | refused by trigger |
| Capped request through the gateway | HTTP 402, no ledger row, nothing billed |
A cap that can be exceeded under concurrency is not a cap. The check and the reservation share one transaction behind a lock on the budget scope, and the comparison runs in SQL numeric rather than JavaScript floats — money comparisons in float64 are how an off-by-a-fraction breach goes unnoticed.