Jev picked login fields in 0.53s, Haiku took 7.83s
Lightpanda clocked Jev at 0.53s and 2 calls against Claude Haiku 5.5 at 7.83s and 8 tool calls on the same login form task. The week's other lesson is cheaper: Mike Taylor says Jev as a judge keeps picking whatever option you listed first.
Someone measured it
Lightpanda clocks Jev at 0.53s on logins, Haiku 5.5 at 7.83s
Lightpanda now has Jev picking the username field, password field and submit button inside its agent's login flow. It posted 0.53s and 2 calls for Jev against 7.83s and 8 tool calls for Claude Haiku 5.5 with browser tools. Lightpanda did not say how many pages it ran or how often either one picked the wrong field.
Jev now handles logins in Lightpanda's agent. It picks the username field, password field and submit button from the page.
Berman's 100 AI shoppers rated 3,092 Black Friday ads for $1.59
Matthew Berman ran 100 simulated shoppers over 3,092 real US Meta ads from 669 brands with Jev, and says the whole experiment cost $1.59 and took three minutes. Of the top 100 ads the shoppers picked, only 2 carried a 50% discount; 32 of the 100 shoppers picked an ad with no discount at all, against 12 for the 50% off ad.
32 of 100 AI shoppers picked a black friday ad with no discount.
Writing criteria
Allie the Icon's line-by-line search: nouls for one answer, scores to rank
Allie the Icon's line-by-line search pattern puts a whole document in state and asks the question once per line or paragraph, instead of chunking and embedding it. Her rule for picking an output type: nouls (a single named choice) when you expect one dominant answer, scores when you want every relevant passage highlighted by relevance. Diogo Almeida has been drawing visual explainers of the pattern.
Use nouls when you expect one dominate answer, use scores when you want to highlight lines by overall relevance.
Theo turned a week of agent rage moments into an 80-case tool eval
Theo spent a week building an eval out of how his agents actually used T3 Code's tools plus every failure that annoyed him, then hill-climbed tool descriptions against it: Sonnet 5.5 to a perfect score, then Sol, then Opus, then a final pass told to cut context 50% with no regressions. The eval had over 80 cases and Opus failed more than half on the first run.
My eval had over 80 cases, and Opus failed over half of them.
The case against
Mike Taylor says Jev as a judge keeps picking whatever comes first
Mike Taylor warned against using Jev as a judge, saying it is strongly biased to position and often picks the option listed first. He did not publish the eval behind it or say how large the skew was. Nobody has posted a counter-test, and TypeSafe has not responded.
Word of warning on Jev-as-a-judge, it is strongly biased to position (it often picks what comes first)
Frequently asked questions
What's the difference between nouls and scores in line-by-line search?
Nouls are used when you expect one dominant answer from a document, while scores are used when you want every relevant passage highlighted by how relevant it is. The choice depends on what kind of output you need from your search.
Why does line-by-line search cost more than chunking?
Line-by-line search puts a whole document in state and asks the question once per line or paragraph, so the cost scales with how many lines or paragraphs you have. You should price a long document before you use this pattern.
How did Theo optimize his agent's tool use?
Theo built an evaluation from how his agents actually used tools plus every failure that bothered him, then tested tool descriptions against it starting with the cheapest model until it got a perfect score, then moved to more expensive models. The approach was to go cheapest first, then add expensive models without going backwards.
What bias does Jev show when used as a judge?
Jev is strongly biased to pick whichever option is listed first when comparing two things. If you use any model as a judge for scoring A against B, you should shuffle the order and run it both ways to make sure the result is real.