Ulmer says Jev images match Sonnet at a 25th of the cost
MotherDuck's own benchmark also includes one where Jev does worse than its hot dog test, until the confidence scores get used.
Jev image support is now in MotherDuck SQL, and Hamilton Ulmer says it reaches comparable quality to Sonnet at ten times the speed and a twenty fifth of the cost. The feature covers classification, scoring and bounding boxes on images at what Ulmer calls analytics scale, called from SQL the same way as text, as in `SELECT prompt_jev(image, 'Is this a hot dog?') FROM food`. The numbers come from MotherDuck's own benchmark, which Ulmer describes as actually-rigorous, and the underlying accuracy figures sit in the linked write up rather than in the post.
The benchmark that lost
The more useful half of the post is the part where it did not work. Ulmer says one of the benchmarks performs worse than the hot dog test, which is about as low a bar as image classification offers. His fix is the confidence score that Jev returns with each call, which he says can be used to drastically improve performance on that benchmark. That is the familiar classical ML move of thresholding on confidence and routing or dropping the low scores, and Ulmer is explicit that this is what is happening.
What is not in the post
No per task accuracy, no latency figure in milliseconds, no dollar cost per thousand images, and no statement of which Sonnet version the comparison ran against. The speed and cost multiples are relative claims from a vendor benchmarking its own integration, so they are a starting point for your own run, not a result you can cite. The hot dog example is a joke, but it is also the only workload named in the post.
For people building on it, treat the confidence score as the actual product here. The headline multiples describe an average over MotherDuck's set, while the one benchmark Ulmer singles out needed the confidence filter to become usable at all. If you are running Jev over images in bulk, budget for the fraction you will have to discard or escalate, and measure that fraction before you price the job. Jev picked up image support and a bounding box type in the 1.14 preview.
We have a benchmark here that actually performs WORSE than our hotdog bench.
Comparable quality to Sonnet at 10x the speed and 1/25th the cost.
Know what Jev is good at before you bet on it
One email a day. Under three minutes. What builders found out about Jev yesterday.