Theo built an 80-case eval from his own agent rage moments
A week of writing failure cases turned into a tool schema rewrite that Theo measures at 4x better tool use on 79% less context.
Theo spent about a week building an eval for the tools in T3 Code, then used it to rewrite the tool schemas. He posted the result as day 23 of his build log, saying agents now use the tools 4x better while using 79% less context. Both numbers are his own, measured on his own harness.
My eval had over 80 cases, and Opus failed over half of them.
Amazed at how many problems you can kinda just turn into evals
The interesting part is where the eval came from. He took the ways his agents used the tools in day to day work, plus the failures and what he calls his rage moments, and turned them into a generic eval he can rerun against any change to a tool or its description. He expected to be mostly trimming prose. Instead he found error cases. The eval ended up with over 80 of them, and Opus failed more than half.
The hill climb
From there it was ordinary optimization. He benched Sonnet 5.5 first and adjusted tool descriptions until it scored perfectly, then added Sol and repeated, then added Opus and repeated again. Once error rates were better than the starting point he merged. The context reduction came last and separately, in a fresh thread where he told Opus to trim until the schema used 50% less context with no regressions on the eval.
The bloat had a cause. Theo says T3 Code started as a wrapper so thin the agent did not know what it was, and that Orchestrator V2, which lets harnesses call each other, forced a full overhaul. Built-in browser, a device manager for mobile, subagents and delegates, send to other thread and inline HTML visualizations all added surface area. One specific bug is now fixed. The schema listed every model available for subagents, which was harmless for most people and bad for OpenCode and Pi users with hundreds of models loaded.
What it cost
He puts the token spend for the whole exercise at $1500. That is the number worth sitting with. A week of authoring cases and $1500 of tokens bought a schema rewrite with a regression check attached, on one codebase, with one person's definition of correct behaviour.
For people building on it, the copyable piece is the source of the eval, not the hill climb. Writing down the failures that annoyed you during real use gets you cases a synthetic suite will not produce, and the fact that Opus failed over half of them on the first run says more about how untested most tool descriptions are than it does about Opus. The 4x and the 79% are a single run on a single harness, not a benchmark anyone else can reproduce yet.
Know what Jev is good at before you bet on it
One email a day. Under three minutes. What builders found out about Jev yesterday.