Open SWE router numbers land, 973 threads, merge rate flat

The numbers behind LangChain's coding agent router are out. NeilXbt's writeup says Open SWE, LangChain's open source coding agent, was tested on 973 threads, half of them routed to the cheapest model likely to finish the job and half always on GPT-6 Astra. Median cost per thread was $0.94 for the routed half against $2.61 for the control, which is the 64% figure that circulated earlier.

The quality number is the one to read

Threads that ended in a merged PR came in at 29.2% for routed and 27.3% for Astra. LangChain's own p-value for that gap is 0.49, so the honest reading is that the router did not measurably change merge rate, not that it improved it. The routed half sent 56% of threads to the balanced model, 34% to the fast one and 10% to the performance one, which is where the cost difference comes from.

There is a second result in the writeup that is easy to skip. LangChain also ran a configuration that always used the fast model. That test was ended within a day because engineers reported low output quality. So the routing is doing work that a blanket downgrade does not do, and the tier assignment is the product, not the cheap model.

Caveats, from LangChain

NeilXbt lists two caveats that LangChain states itself. The control arm was the most expensive model available, which makes the 64% a comparison against the worst case for spend rather than against a sensible default. And the router is described as a proof of concept, not a shipped, tuned system.

Harrison Chase added the design argument for where the routing decision belongs.

router lives in the harness, not a gateway, bc the harness knows the task

That is the structural claim worth arguing with. A gateway sees a request. A harness sees the thread, the repo state and how far the agent has got, which is the context you would want when deciding whether a task needs the expensive model.

For people building on this, the transferable result is not the 64%. It is that a three tier split held merge rate roughly flat while an always fast setup failed inside 24 hours on engineer judgement rather than on a metric. If you copy the pattern, copy the control too, because measuring a router against your priciest model will always flatter it. And one cost run on one agent across 973 threads is one team's result, not a benchmark.

Harrison Chase

@hwchase17

X

router lives in the harness, not a gateway, bc the harness knows the task