Nilforoshan's Jev benchmark still stands after OpenAI beta
The only published comparison of OpenAI's Decisions API against Jev is one private ranking task on a service with 2.5 million users.
OpenAI's Decisions API is now in public beta for all developers, and the only head to head anyone has published still goes Jev's way. Hamed Nilforoshan benchmarked the two on predicting how relevant a user query or resume is to a job description, scored 1 to 10, on a service serving 2.5 million users. His result, posted earlier and unchallenged since, is that OpenAI is 2x more expensive and 5 to 10% worse.
The announcement it is measured against is OpenAI's own. OpenAI Developers said the Decisions API lets an app choose the right model, tool, or action in near real time, and that it makes decisions up to 10x faster than GPT-6 Luna through the Responses API. That claim is about speed. Nilforoshan's numbers are about price and accuracy on one ranking task, so the two are not in contradiction, and nobody has published a latency comparison on his workload.
What the number is and is not
This is one private benchmark on one domain, run by one person, on a task that nobody outside his company can reproduce. Diogo Almeida liked the methodology, posting "yay to domain specific private benchmarks!" That is a compliment about the approach, not an endorsement of the result, and a private benchmark is harder to game and impossible to audit. Treat the 2x cost gap as a fact about relevance scoring at that company, not a general ranking of the two APIs.
yay to domain specific private benchmarks!
TypeSafe AI, which makes Jev, reposted the thread with a one line comment, "The rumors of our death have been greatly exaggerated". That is the vendor taking a victory lap on a customer's number, which is worth exactly what a vendor repost is usually worth.
The rumors of our death have been greatly exaggerated
For people building on it
The useful part of Nilforoshan's setup is not the verdict, it is the shape. He picked a scalar scoring task with a clear target, ran both models over his own traffic, and compared cost and accuracy together rather than one at a time. Cost gaps between decision models move with token counts, prompt length and how much output you constrain, all of which are specific to your schema. If you are deciding whether to move a routing or ranking step off Jev, the honest read of this week is that the public evidence points one way and the sample size is one. Run your own queries before you move anything.
Know what Jev is good at before you bet on it
One email a day. Under three minutes. What builders found out about Jev yesterday.

