Four semantic approaches, measured against a deterministic ranker on the same catalogue and the same tasks. None of them won, and one of them — our own first instinct — was materially worse.
Tool selection is a ranking problem. A model is offered a set of tools; if the set is too large it wastes context and makes worse choices, so something has to decide which ones are relevant to the request in hand. Embeddings are the obvious answer, and we had the infrastructure for them from the beginning — pgvector enabled, a provider-neutral embedder, vectors precomputed per tool version, the request vector cached by digest.
The obvious answer lost.
Real text-embedding-3-small vectors over the policy-permitted tools for one request, needing four specific tools at a cutoff of five. The deterministic ranker — term frequency with IDF weighting and BM25 length normalisation — recalled three of the four. Pure cosine similarity recalled three of four. Hybrid reciprocal-rank fusion recalled three of four, which is to say it matched its better input without improving on it.
Maximal marginal relevance, the technique specifically designed to stop a result set collapsing onto one theme, recalled one of four at λ=0.5 and one of four at λ=0.3.
MMR was our own first instinct for this problem, and it was the worst option tested.
The request was a refund-eligibility question. Its vocabulary is dominated by refund language, so every refund-shaped tool scores highly and the needed tools cluster together. MMR is built to break exactly that clustering — so it reached past the cluster and pulled in genuinely unrelated tools. The property that makes it valuable in document retrieval is the property that made it wrong here.
That is worth stating plainly because it was not predicted. Reasoning about which technique should work produced the wrong answer; measuring produced the right one.
A reasonable objection is that four tools and one request is too small a test, and that semantic ranking should pay off as the catalogue grows. We ran the same comparison across 38 tasks against a 60-tool catalogue of deliberately confusable tools — seven near-identical refund verbs across three risk classes, among others.
The lexical ranker recalled 96% of required tools. Semantic recalled 80%, while exposing more tools and spending more schema tokens to do it. Hybrid fusion landed at 93%, again below its better input. Scale made the gap wider, not narrower.
The specification this system is built to says that a semantic feature which does not produce repeatable benefit should be disabled. Three independent measurements failed to beat deterministic ranking, so semantic ranking is off by default. The infrastructure stays — it is built, tested, and costs nothing when unused — but the claim that it helps is unsupported, and we will not make it.
There is a real caveat. Everything above concerns short, one-or-two-sentence requests against catalogues of tens of tools. Nothing in it speaks to multi-paragraph requests or catalogues of five hundred. If that changes the result, we will publish that too.
A semantic feature has to beat the boring alternative on your own data before it ships, and the honest outcome of that test is sometimes that it does not.
We nearly deleted a working component because it benchmarked badly. The defect was one layer below it, in code nobody was looking at.
Routing products tend to be compared against the most expensive baseline. We ran the one that actually threatens us, and reported what it did.
We split the value of tool governance into the part the policy filter does and the part the ranker does. They are not close.