Mike Taylor: Jev-as-a-judge favors whatever comes first
Mike Taylor warns that Jev used as a judge is strongly biased to position, often picking the first option it is shown. If your eval compares two candidates in a fixed order, that bias is in your numbers.
Writing criteria
Berman ranks nine agents to build, Jev shows up at number five
Matthew Berman posted the order he would build nine agents in if he had to make money with them starting Monday, with speed-to-lead first and the weekly report last. Jev appears at number five, the competitor ad breakdown, where he says it broke down 724 live ads from 37 brands in 40 seconds for 9 cents.
pick the channel already making you money. build one agent for its most boring hour.
Mike Taylor pairs Claude with Jev to do napkin math
Mike Taylor ran Fermi problems (estimate an unknown by splitting it into parts) through Claude plus Jev, with Claude breaking down the question and Jev estimating the probability on each piece.
claude breaks down the questions and jev estimates the probability
The case against
Mike Taylor says Jev-as-a-judge is strongly biased to position
Mike Taylor posted a warning about using Jev as a judge, saying it is strongly biased to position and often picks whatever option comes first. He did not publish the setup or a hit rate in the post, so this is one person's finding, not a measured rate you can plan around.
it is strongly biased to position (it often picks what comes first)