OpenAI's Decisions API lost Nilforoshan's benchmark
Hamed Nilforoshan ran OpenAI's Decisions API against Jev on a production relevance task and found OpenAI 2x more expensive and 5 to 10% worse. Every number that landed this week came from a private benchmark nobody else can rerun.
Built with it
vogel gave agents their own email addresses, with Jev doing the filtering
vogel shipped an email service for agents where the agent claims its own address, you approve it in one click, and it sends, receives and replies over MCP or API. He says Jev handles the filtering and moderation, and that serving something like this to thousands of people was hard before. No volume, latency or cost numbers were posted, and Google sign in landed the same day.
before Jev it was kinda hard, but now it's almost second nature
Someone measured it
Nilforoshan's benchmark holds: Decisions API 2x pricier, 5 to 10% worse
A week after OpenAI opened its Decisions API beta, the only head to head anyone published still goes Jev's way. Hamed Nilforoshan ran both on predicting how relevant a user query or resume is to a job description, on a service serving 2.5 million users, and found OpenAI 2x more expensive and 5 to 10% worse.
The rumors of our death have been greatly exaggerated
Vals AI says Jev matched GPT-6 Astra accuracy at 1/500th the cost
Vals AI says it independently tested Jev against 11 other models and, on 400 claim-verification questions, it matched GPT-6 Astra's 97.5% accuracy at roughly 1/500th the cost. Diogo Almeida passed the result around with the argument that cheaper intelligence means intelligence everywhere. The framing in the test is blunt: can a model that does not generate text compete with frontier LLMs.
Harrison Chase says Open SWE routing cut median cost per task 64%
Harrison Chase says Open SWE moved model choice into the harness, so each task goes to the cheapest model that still does the job, and median cost per task dropped 64%. He posted it replying to a developer arguing that deepseek v4.1 flash removes the case for running a frontier model on everything. He says the routing was tested against quality, but no quality number came with it.
Writing criteria
Mike Taylor's eval recipe: train a judge until it agrees 90% with humans
Mike Taylor posted a full eval recipe: save past tasks, replay them against new models or prompts, and measure whether anything improved. His judge trick is to rewrite the AI judge's prompt until it agrees 90% with human judgements, then grade with yes/no questions scored 1 or 0 so a grade cannot be misread.
Measuring "if it improved" is 90% of the work and 10% is infra.
Also on the timeline
Ramp data has Jev taking a full point of market adoption in a month
Ramp's October vendor data says Jev captured an entire percentage point of market adoption within a month, growth Ara Kharazian calls unheard of. TypeSafe AI reposted it.
Max Blade raced 100 Haiku agents against one Opus 5.5 agent
Max Blade put 100 communicating Haiku agents against a single Opus 5.5 agent in a death maze.
Frequently asked questions
What is Jev and what does it do?
Jev is a decision-making service that helps filter and moderate data. It powers email services for agents, runs comparisons against other AI models, and handles filtering at scale.
How does Jev's email service work?
Agents claim their own email address, get approval in one click, and can send, receive and reply through MCP (Model Context Protocol) or API connections. Jev handles all the filtering and moderation of those emails.
How much cheaper is Jev than other decision models?
Benchmarks vary by task: Jev was found 2x cheaper than OpenAI's Decisions API on resume relevance scoring, and roughly 1/500th the cost of GPT-6 Astra on claim verification questions. These numbers are specific to those individual tests.
What is a decision model and why does it matter?
A decision model makes yes-or-no choices about data instead of generating text, like ranking whether a resume matches a job or verifying if a claim is true. They can be cheaper than text-generating models for tasks that only need a choice.
How quickly is Jev being adopted?
According to Ramp's October vendor data, Jev captured an entire percentage point of market adoption within a month, which was called unheard of growth.