Mike Taylor checked Jev against the probability words chart
An eyeball check against a well known survey chart, with no per-word numbers and no error measure posted.
Mike Taylor ran Jev against the chart of what probabilities people actually mean when they say words like "impossible" or "likely", and reported that it is pretty well calibrated. He built the comparison as an artifact with Claude and linked it in the same post. He did not publish per-word numbers, a spread, or any error measure.
Wanted to see how well calibrated Jev by @typesafeai is to this famous chart about what probabilities people mean by specific words... and it's pretty well calibrated!
The chart itself is the one that circulates periodically because the spread is funny. @koreindian posted a version of it the same week, noting the people who hear "impossible" and read it as roughly 10 percent.
What this actually tests
This is a check on whether Jev's probability outputs line up with a human survey of vague language, not a check on whether those probabilities predict Jev's own accuracy. Those are different properties, and only the second one is what you want if you are gating on confidence. A model can map "likely" to the same number a person does and still be wrong about how often it is right when it says 0.8.
So treat this as one person's eyeball comparison, posted as a screenshot and an artifact, rather than an eval. There is no count of words tested, no reference to the underlying survey's sample, and no residual reported. Taylor's own framing is a single judgement about the fit.
For people building on it, the reliability number you care about is the one you measure on your labels. If you are routing on Jev's confidence, the way Hassan did when he sent low confidence calls to a second model, the question is what fraction of items in each confidence bucket Jev gets right on your data, not whether its sense of the word "probable" matches a survey of humans. Bucket your own outputs, score them, and look at the gap between stated confidence and observed hit rate. That takes an afternoon and it is the only version of this check that tells you where to put the threshold.