LangChain tested Jev as a judge against LLM judges
The posts announce the comparison but publish none of the numbers, so the four axes are only as good as the write-up behind them.
LangChain says it tested Jev against LLM judges on accuracy, repeatability, latency and cost, framing it as a check on whether System One models offer a different approach to agent evaluation. The announcement post links out to a guide and carries no figures of its own.
Sydney Runkle, sharing the same work, credits the guide to Sean and Daniel and describes it as covering why Jev suits evals, an experiment against other LLMs, and how to run it on your own agents. Her summary is the only verdict either post gives.
What is actually claimed
Two things, and only two. First, that the four axes were measured. Second, Runkle's line that Jev as a judge is cheaper and more precise than LLM as a judge for online evals. No accuracy percentage, no latency figure, no cost per judgement, no description of the task set or how many runs the repeatability measurement covers. Anyone quoting a number from this has taken it from the write-up, not from the posts.
Repeatability is the axis worth reading for. LLM judges drift between runs on identical inputs, which is what makes them awkward for online evaluation where you are scoring production traffic continuously rather than a frozen test set. If Jev's variance is lower, that is a structural argument rather than a quality argument, and it would hold even where an LLM judge scores better on a single pass. Neither post states what the drift was in either direction.
For people building on it, treat this as a vendor adjacent claim until you have read the guide. LangChain sells evaluation tooling, Runkle is summarizing her colleagues' experiment, and "cheaper and more precise" is doing a lot of work without a table behind it. The interesting comparison is not Jev versus an LLM judge in the abstract, it is Jev versus whatever judge you currently run, on your own rubric, with the disagreement rate against human labels written down. That number is the one that decides whether you swap.
jev as a Judge proves to be a cheaper and more precise alternative to LLM as a judge for online evals.