Jev Daily

Ulmer adds Gemini Decisions to HotdogBench, says Jev still wins

Claude Fable 5.2 made the fewest mistakes at the highest cost, but Ulmer published the ranking without the underlying numbers.

Hamilton Ulmer added the Gemini Decisions API to HotdogBench and reported that it came out expensive next to Perplexity Decider, with Jev still ahead of it on his results. Claude Fable 5.2 landed at the other end of the table, fewest mistakes at the highest cost.

Hamilton Ulmer
@hamiltonulmer
X
surprised at how expensive it is vs Perplexity Decider, and still ~ Jev beats it.
Nov 12, 2026 · View on X

That is a quick turnaround on a model Google announced the same week. Sundar Pichai launched Gemini Decision 1 in AI Studio at $0.10 per million input tokens with output billed at zero, which he put at roughly 5x cheaper than Flash-Lite, and said it leads on several benchmark evals with the lowest calibration error. The pitch is closed set choices rather than token by token generation, scoring up to 1,000 questions across text, images, audio, video and PDFs in a single pass, with yes/no, multiple choice up to 255 options, and rubric scoring.

The gap between list price and bench price

Ulmer's complaint is not about the list price, it is about what a decision actually costs once you run it through a benchmark. Input tokens are the only thing being billed here, so a model that needs more context per question, or more questions per answer, gets expensive at $0.10 per million just like anything else. Perplexity Decider came out cheaper on his run, and Jev came out ahead of Gemini on whatever HotdogBench is measuring.

The honest caveat is that he posted a read, not a table. There are no accuracy figures, no cost per thousand decisions, and no description of the task mix in the post itself. Fable 5.2 having the fewest mistakes at the highest cost is the shape every cost accuracy chart has, and without the axes it does not tell you where your workload sits.

For people building on it, this is one person's result on one benchmark published within days of the model shipping, which is exactly the point at which a vendor's calibration claim and a practitioner's bench run are most likely to disagree. If you are deciding whether to move judgment calls off Jev to save money, the thing to measure is your own input token count per decision, because that is the only number Google is charging you for.

Sundar Pichai
@sundarpichai
X
$0.10 / 1M input tokens with $0 billed for output tokens (~5x cheaper than Flash-Lite)
Nov 11, 2026 · View on X

Know what Jev is good at before you bet on it

One email a day. Under three minutes. What builders found out about Jev yesterday.

Get the next one by email

Jev, read daily so you do not have to. The builds, the benchmarks, the criteria that worked and the cases where it lost, from the people shipping on TypeSafe AI's System One model.