Mike Taylor warns Jev-as-a-judge favors whatever comes first

Mike Taylor posted a word of warning about using Jev as a judge, saying it is strongly biased to position and often picks whatever option comes first.

Mike Taylor

@hammer_mt

X

it is strongly biased to position (it often picks what comes first)

That is the whole claim. He did not publish the prompt, the task, the number of pairs, or a rate at which the first option won. So this is one practitioner's finding from his own use, not a measured bias you can budget for. Treat it as a reason to run a check, not as a number to quote in a design doc.

Why position bias matters more than it sounds

Jev-as-a-judge means handing the model two candidate outputs and asking which is better, then using that verdict to rank, to gate, or to score an eval. The failure mode Taylor describes is the nastiest kind, because it does not look like a failure. The judge returns a confident winner every time. If the winner correlates with slot order rather than quality, every downstream metric built on it drifts in the same direction, and nothing in the output tells you.

The cheap test is a swap. Score each pair twice with the order reversed and compare. Pairs where the verdict flips are pairs the judge did not actually decide on merit, and the flip rate is the number Taylor's post does not give you. If you are already running a judge over production traffic, that is a doubling of judge calls, which at Jev prices is usually the cheap part of the experiment.

Context on the source

Taylor is a repeat critic rather than a drive-by one. He previously reported that Gemini beat a stacked Jev setup on image bounding boxes, and cost less, and he has argued that a short skill half-life is an eval failure. His track record is opinions from someone who ran the thing, which is the better kind, but it is still a single post with no setup attached.

For people building on it, the order you feed candidates in is now a variable in your eval, not a formatting detail. Randomize it, or swap and rerun, before a pairwise score decides anything that ships.