Theo spent $1,000 benchmarking Jev Router, found it 5x slower
An overnight run against a single model, not against the Auto Router that OpenRouter measured.
Theo stayed up until 2am and spent $1,000 benchmarking Jev Router, OpenRouter's new cache aware router that uses Jev to pick a model and reasoning effort per turn. His result on DeepSWE was performance roughly the same as GPT-6 Astra on low, at slightly higher cost, and almost 5x longer to run.
Performance on DeepSWE was roughly the same as GPT-6 Astra on low. It costs slightly more, and it took almost 5x longer to run.
Reminder that I love Jev! I use it for a ton of things and I'm super happy with it.
That is one person's run on one benchmark, and it is not the comparison OpenRouter published. OpenRouter's own numbers are against its Auto Router, where it says Jev Router solved 82% more tasks across four agent benchmarks, 237 of 423 versus 130. It also says that on five agent benchmarks Jev Router had a faster median time to first token than every other router it tested. Theo measured against a single model at a fixed effort setting. Neither number settles the other, and nobody has posted the two side by side on the same tasks.
What the router actually does
Per OpenRouter, Jev reads the conversation text before each turn and scores it on difficulty and precision, then checks whether a bigger model or more effort would help, whether a cheaper model is enough, and whether the task changed. It keeps a model for the rest of the session and switches only when the expected gain beats the cost of losing the cached chat. It runs under zero data retention terms, attachments are never sent to Jev, and if the Jev call times out or returns invalid output the request fails rather than falling back to another router.
Theo's objection to the design is the same one he made earlier about Jev and validation. Jev sees the prompt and nothing else, so it cannot tell whether "port this to Rust" means a 100 line TypeScript file or a million line application. Before the benchmark finished he said the router looked like "basically just a DeepSeek 4.1 flash router".
For people building on it, the interesting figure here is the runtime, not the score. A router that matches a single model's quality at slightly higher cost is a wash, but 5x wall clock on an agent benchmark is the kind of thing that shows up in your users' patience before it shows up in your bill. Theo was clear that he still uses Jev elsewhere.
It is nearly impossible to know how "complex" a task is purely from a prompt.