Mike Taylor: Gemini beat stacked Jev on image detection
Nathan LeClaire stacked Jev on itself to find bounding boxes in an image, and Mike Taylor says Gemini did a better job in one call, for less money. Separately, Harrison Chase wrote down the loop that turns a calibrated judge into an eval pipeline.
Writing criteria
Harrison Chase's loop for a judge running on production traffic
Harrison Chase posted the sequence he thinks makes a judge useful in production: calibrate it on human labels, flag low scorers, confirm the real failures, add them to evals, fix, rerun. He was quoting a PyTorch post on Shopify's continual learning loop, where a calibrated judge flags low-scoring outputs, critics diagnose each failure, and accepted repairs expand the training data.
prod finds the failures your offline dataset never had
The case against
Mike Taylor says Gemini beat stacked Jev on image detection, cheaper
Nathan LeClaire showed Jev stacked on Jev, dividing an image up to find bounding boxes, and Mike Taylor replied that Gemini just does a better job and costs less because it only makes one call.
Because gemini only has to make one call it actually ends up cheaper