Harrison Chase says a production judge is only worth the flags you chase
His six step loop sends low scoring production outputs back into the eval set, and nobody has put a number on what it recovers.
Harrison Chase posted a six step loop for running a judge on live traffic. Calibrate it on human labels, flag the low scorers, confirm which of those are real failures, add them to evals, fix, rerun. His framing is that the judge itself is not the deliverable, the queue of flagged outputs is.
prod finds the failures your offline dataset never had
He was quoting a PyTorch post from PyTorchCon 2026, where Cody Mazza-Anthony presented Shopify's continual learning loop built with PyTorch and vLLM. In that setup, using Buyer Profile as the example, a calibrated judge flags low scoring outputs in production, critics diagnose each failure, a replay with a critic hint gets scored again by the judge, and accepted repairs expand the training data. PyTorch says Cody also showed a small specialist model outperforming a frontier model on that task, then moved on to prompt compression for inference.
What the claim actually covers
The two loops are not the same shape. Shopify's version closes back into training data. Chase's version closes back into an eval set, with a human confirming the failure before it counts. The shared part is the front half, a judge you have calibrated against human labels pointed at traffic you did not write test cases for.
Neither post carries a number. There is no recovery rate, no count of failures surfaced, no measure of how many flags turned out to be real. Chase is describing a sequence he thinks works, not reporting a run. Treat it as process advice from someone who builds eval tooling, not as a result.
For people building on it
The calibration step is the one that decides whether the rest is worth doing. An uncalibrated judge gives you a flag queue that is mostly noise, and the confirm step then eats the time you were trying to save. Mike Taylor's version of that threshold, training a judge until it agrees with humans about 90% of the time, is the closest thing to a published bar anyone in these threads has offered.
The other thing to note is that Chase's argument is about where eval cases come from, not about how accurate your judge gets. The line he is selling is that offline datasets are written by people who already know the failure modes they are testing for. Production traffic is not. If your eval suite has stopped finding bugs, that is the claim he is making about why.
Critics diagnose each failure, and a replay with a critic hint is scored again by the judge.
Know what Jev is good at before you bet on it
One email a day. Under three minutes. What builders found out about Jev yesterday.