Planverity started from a specification, not a demo. The parts that survived are the parts that held up when we tested them.
Three times we built a semantic feature, measured it against the deterministic one it was meant to beat, and turned it off. Embeddings, hybrid fusion and MMR diversity are all implemented, tested, and disabled by default — because none of them produced a repeatable improvement. That rule is written into the system, not just the culture: a semantic feature must demonstrate benefit or be removed.
We tuned a ranking algorithm against a single test case and it looked like a clear win. Measured across 38 tasks, it selected identical results to naive substring matching — the sophistication bought nothing. Later, at 60 tools, it performed worse than the naive version, and the cause turned out to be a missing stemmer one layer below. The component we were ready to delete was fine; its input was broken.
We also once concluded from a live run that the strongest model available was the weakest of three. It was not: a token budget too small for a model that reasons before answering produced empty responses that graded as wrong answers. A confident, entirely false measurement — and the reason unmeasured values in this product never render as zero.
Planverity does not dominate a single good mid-tier model on general traffic. It is Pareto-efficient and has the best cost-per-success of any strategy we tested, but against always-Sonnet it offers a dial rather than a free lunch. We publish that alongside the wins, because a routing product that reports only favourable comparisons is not giving you evidence.