Each of these is a write-up of a real benchmark run in this repository, with the figures taken from the result rather than remembered. Three of the four describe something that went against us — a finding that confirmed what we already believed would teach you nothing about whether to trust the rest.
Four semantic approaches, measured against a deterministic ranker on the same catalogue and the same tasks. None of them won, and one of them — our own first instinct — was materially worse.
A semantic feature has to beat the boring alternative on your own data before it ships, and the honest outcome of that test is sometimes that it does not.
We nearly deleted a working component because it benchmarked badly. The defect was one layer below it, in code nobody was looking at.
Before deleting a component that measures badly, check whether its inputs are correct. A ranker can only be as good as its tokeniser.
Routing products tend to be compared against the most expensive baseline. We ran the one that actually threatens us, and reported what it did.
Do not quote a savings figure against always-premium without also showing always-balanced. Omitting the strongest simple baseline is how routing products flatter themselves.
We split the value of tool governance into the part the policy filter does and the part the ranker does. They are not close.
Most of the value of tool governance is the filter, not the ranker — and the filter's guarantee is structural rather than statistical.
Every figure quoted in these posts comes from a benchmark table with its method, trial count and date beside it. The tables are the source; the posts are the explanation.