Theo re-ran a coding model on CLI, still calls it garbage
A broken harness inflated the first result, and fixing it moved the number without moving the conclusion.
Theo re-tested a coding model after finding his harness was broken, and the verdict did not change. He says the ACP bindings (Agent Client Protocol, the layer connecting the model to his editor) were "kind of broken" during the original test, so he ran the model again through the CLI instead. False reports went down by about 5 percent.
Better, but still utter garbage for real world work
That follows an earlier and much harsher call. Theo had said the model is incapable of working in a real code base without hallucinating in egregious ways, and that it is not 5 percent worse but 10x worse. The re-test narrows the gap by a few points and leaves the shape of the claim intact. Theo does not name the model in either post.
For people building on it, this is one person's run on his own codebases, not a published eval, and there is no task count, pass rate or baseline model given. If you are comparing models through an agent protocol layer, Theo's experience is a reminder to also run the same cases through a plain CLI before you trust the numbers.
Know what Jev is good at before you bet on it
One email a day. Under three minutes. What builders found out about Jev yesterday.
