We had built a term-weighted ranker for tool selection: inverse document frequency so that a term appearing in most tools counts for little, BM25 pivoted length normalisation so a well-described tool is not penalised for being well described. On an 18-tool catalogue it selected identical sets to naive substring matching on all 38 tasks — verified by diffing the selections, not inferred from equal scores.
At 60 tools it did worse. Lexical ranking recalled 89%; plain substring matching recalled 91%. The sophisticated thing lost to the simple thing, at the scale where sophistication was supposed to start paying.
The honest reading at the time
We wrote it up as a refutation. The IDF and BM25 work had been tuned against a single four-tool case, and measured at scale it bought nothing — worse, it cost two points against a one-line alternative. That was the conclusion, and it was published.
It was wrong.
What the misses had in common
Three of the four missed tools shared a shape. The request said "what they actually ordered" and the tool was get_order. It said "what this customer was invoiced" and the tool was get_invoice. Both substring matching and term matching failed on the inflection: ordered does not contain order as a token, and invoiced is not invoice.
The tokeniser had no stemming. Every ranker above it was scoring against words that did not match the words in the tools, and no amount of clever weighting fixes a term that is never compared.
A limit had been attributed to the benchmark that belonged to the code.
What changed when it was fixed
Stemming is conservative and inflectional only — plurals, -ed, -ing — applied inside the shared tokeniser so the request and the tool text cannot disagree. With it, lexical ranking recalls 96% where stemmed substring matching recalls 91%. The stemmed keyword arm exists precisely so the comparison is not stacked: both arms get the same tokens, and only the term-weighted ranker converts them into recall.
Substring matching gains nothing from stemming at all — 91% either way. So the ranker was always doing real work; it had simply never been given correct input.
The correction we had to publish
The earlier write-up also claimed that the shared 93% ceiling across every strategy was a property of the benchmark's ground truth rather than of the ranker. After stemming, the unranked policy-permitted baseline reaches 100%, and a direct check found zero tasks with an unreachable required tool. The ground truth had always been fully achievable.
Two smaller things fell out of the same fix. The hybrid fusion arm had been building its request terms with an inline split, applying neither stopwords nor stemming — so it was fusing the lexical ranking with a lexical ranking the lexical arm never produced. And a test had the same bug, hand-building a term set instead of tokenising. The tokeniser is now exported so every caller shares one function.