Archive
- Prism beats Jev by two emails on a 200-email test
Arushi Gandhi ran 200 of her own emails through four decision models and says Jev came second by a margin of two emails, while being the fastest. On a sample that small, that is a tie.
- Vogel's Jev spam ranker was too costly to ship
vogel built an X reply filter that scores every reply for spamminess with Jev and auto-mutes the account, then said he cannot release it because it is expensive. He swapped it for a Grok bot keeping a public blocklist instead.
- Jev picked login fields in 0.53s, Haiku took 7.83s
Lightpanda clocked Jev at 0.53s and 2 calls against Claude Haiku 5.5 at 7.83s and 8 tool calls on the same login form task. The week's other lesson is cheaper: Mike Taylor says Jev as a judge keeps picking whatever option you listed first.
- AI shoppers skipped the 50% off ad, Berman says
Matthew Berman's Jev agent ran 100 synthetic shoppers over 3,092 real Meta ads and says only 2 of the top 100 picks carried a 50% discount. If you are about to cut prices for Black Friday, that is one person's result worth replicating before you copy it.
- Mike Taylor: Jev-as-a-judge favors whatever comes first
Mike Taylor warns that Jev used as a judge is strongly biased to position, often picking the first option it is shown. If your eval compares two candidates in a fixed order, that bias is in your numbers.
- Illyism's agent closed a support ticket and shipped the fix
ILIAS ISM pasted a support screenshot into Cursor with zero words of prompt and says the agent read the Gmail thread, found the root cause and deployed a page change in 60 seconds. He also says an AI agent spent six hours probing his admin panel two nights earlier, so he deleted the panel.
- Shengkun Ye's people search runs at $0.0024 a lead
A second open source people search on Jev landed, this one quoting $0.0024 per lead by waterfalling six data providers and paying for enrichment only on the leads Jev keeps. Mike Taylor, separately, says short-lived skills are an eval failure, not a skill problem.
- Lightpanda: Jev picks login fields in 0.53s
Lightpanda put Jev on the login step of its browser agent and posted 0.53 seconds and 2 calls against 7.83 seconds and 8 tool calls for Claude Haiku 5.5. One vendor's own build, one task, but it is the first number anyone has posted on form filling.
- Sentry spends $263 a day on Jev for 10M events
Sentry screens 10 million events a day through Jev at $263 a day to catch prompt injection, and is now testing its own classifier to hit the same accuracy for 90% less. The cheap model is cheap enough to meter, and still worth replacing.
- Kush pulled Jev out of Puffle's harness
Kush says Jev kept making wrong calls on whether a file was worth reading, so Puffle went back to Qwen 3.7 flash. Where a false negative is expensive, a cheap classifier's error rate is the whole product.
- Moritz Kremb ships a Grok bot for X bookmarks
Moritz Kremb published a Grok bot template that syncs X bookmarks daily and lets another bot query them as a knowledge base. No numbers, no cost, no eval, just a template you can run.
- TypeSafe discloses leak of all customer noul outputs
Diogo Almeida says TypeSafe AI became aware of a massive unauthorized leak of all customer noul outputs at 6:49PM Friday, and at least one builder has already found his own data in it.
- Composio puts Jev in front of every agent action
Composio now runs every action an agent takes through Jev and its own tuned small models before it executes, and published no latency or cost figures for the check. Sanity is using Jev as a router in its internal agent harness.
- Lightpanda put Jev in its binary for page triage
Lightpanda compiled Jev into the Lightpanda binary and built judge() on it, one call at roughly 0.2s to tell a real article from a cookie wall. Separately, UiPath shipped a Jev connector, with the cost claim coming from the vendor.
- Mike Taylor: Gemini beat stacked Jev on image detection
Nathan LeClaire stacked Jev on itself to find bounding boxes in an image, and Mike Taylor says Gemini did a better job in one call, for less money. Separately, Harrison Chase wrote down the loop that turns a calibrated judge into an eval pipeline.
- Zachi: Cognition's Devin Fusion chart is "a crime"
Cognition says Devin Fusion scores 68 on the Coding Agent Index at 31% lower cost per task than Claude Code with Opus 5.5. Zachi says the chart behind it drops every model under $5, which is most of the field.
- Haiku 5.5 takes the WebMCP crown from Jev
Idan Levin's benchmark now puts Haiku 5.5 with WebMCP ahead of previous winner Jev on combined success, speed and cost, while OpenRouter shipped a production tool call guard running on Jev 1.13. Two results, opposite directions, same week.
- Vercel says Jev was September's most-requested model
Vercel says Jev became AI Gateway's fastest-adopted model ever, with no adoption numbers attached. Also today: Lee Knowlton's split between work an AI can verify and work it cannot.
- Jev images in MotherDuck: Sonnet quality, 1/25th cost
Hamilton Ulmer says the image benchmark he ran scores worse than MotherDuck's hot dog bench, and that the confidence scores are what rescue it. If you are classifying images at analytics scale, the gating matters more than the headline accuracy.
- Haiku 5.5 knocks Jev off the WebMCP top spot
Idan Levin's WebMCP benchmark has a new winner, and it is not Jev: Haiku 5.5 took first on the combined success, speed and cost score. On the same day OpenRouter shipped Armor, a tool call guard built on Jev 1.13.
- Jev 1.14 can see, and Clef claims 3 of 4 against it
TypeSafe says Jev was an image model the whole time and they had it switched off. Same day, Merge says Cloudflare's own tests put Clef ahead of Jev on three of four workflow evals.
- ScaleDown claims 77.2 vs Jev's 72 on DecisionBench
Neal Patel launched a three model family and put Jev's DecisionBench score in the launch post, 77.2 against 72. The number comes from the people selling the model, and nobody outside has run it yet.
- SubQ Snap says it beats Jev on Decision Index
Alexander Whedon launched SubQ Snap claiming 61.7 on Decision Index v0.2.1 against Jev's 57.9, with 384K tokens of context. Every number so far comes from the vendor and two people it gave early access to.
- OpenAI's Decisions API lost Nilforoshan's benchmark
Hamed Nilforoshan ran OpenAI's Decisions API against Jev on a production relevance task and found OpenAI 2x more expensive and 5 to 10% worse. Every number that landed this week came from a private benchmark nobody else can rerun.