Jev Daily

Nilforoshan's Jev benchmark still stands after OpenAI beta

The only published comparison of OpenAI's Decisions API against Jev is one private ranking task on a service with 2.5 million users.

OpenAI's Decisions API is now in public beta for all developers, and the only head to head anyone has published still goes Jev's way. Hamed Nilforoshan benchmarked the two on predicting how relevant a user query or resume is to a job description, scored 1 to 10, on a service serving 2.5 million users. His result, posted earlier and unchallenged since, is that OpenAI is 2x more expensive and 5 to 10% worse.

The announcement it is measured against is OpenAI's own. OpenAI Developers said the Decisions API lets an app choose the right model, tool, or action in near real time, and that it makes decisions up to 10x faster than GPT-6 Luna through the Responses API. That claim is about speed. Nilforoshan's numbers are about price and accuracy on one ranking task, so the two are not in contradiction, and nobody has published a latency comparison on his workload.

What the number is and is not

This is one private benchmark on one domain, run by one person, on a task that nobody outside his company can reproduce. Diogo Almeida liked the methodology, posting "yay to domain specific private benchmarks!" That is a compliment about the approach, not an endorsement of the result, and a private benchmark is harder to game and impossible to audit. Treat the 2x cost gap as a fact about relevance scoring at that company, not a general ranking of the two APIs.

Diogo Almeida
@CompleteSkeptic
X
yay to domain specific private benchmarks!
Oct 7, 2026 · View on X

TypeSafe AI, which makes Jev, reposted the thread with a one line comment, "The rumors of our death have been greatly exaggerated". That is the vendor taking a victory lap on a customer's number, which is worth exactly what a vendor repost is usually worth.

TypeSafe AI
@typesafeai
X
The rumors of our death have been greatly exaggerated
Oct 7, 2026 · View on X

For people building on it

The useful part of Nilforoshan's setup is not the verdict, it is the shape. He picked a scalar scoring task with a clear target, ran both models over his own traffic, and compared cost and accuracy together rather than one at a time. Cost gaps between decision models move with token counts, prompt length and how much output you constrain, all of which are specific to your schema. If you are deciding whether to move a routing or ranking step off Jev, the honest read of this week is that the public evidence points one way and the sample size is one. Run your own queries before you move anything.

Know what Jev is good at before you bet on it

One email a day. Under three minutes. What builders found out about Jev yesterday.

Get the next one by email

Jev, read daily so you do not have to. The builds, the benchmarks, the criteria that worked and the cases where it lost, from the people shipping on TypeSafe AI's System One model.