Kush pulled Jev out of Puffle's harness over wrong file calls

Kush says the Puffle team has removed Jev from its agent harness and gone back to Qwen 3.7 flash. The opening line of the post is "beware of jev".

Kush

@kushbhuwalka

X

jev constantly makes wrong assessments which leads the agent astray

Kush

@kushbhuwalka

X

the punishment of wrongly classifying a relevant file is much worse than the reward of correctly classifying a irrelevant file

The job Jev was given is the one a lot of teams are trying right now. Kush describes two uses for decision models in a harness. First, as scouts for the agent, checking whether a file is worth reading at all, which he says massively saves cost and latency. Second, solving precision in RAG, the retrieval step that decides which chunks get into the prompt. He says he stuck Jev in immediately when it came out.

Why it broke the benchmarks

The failure was not cost or speed, it was correctness. Kush says Jev constantly makes wrong assessments, which leads the agent astray, and his verdict is that it is just not smart enough.

The interesting part is why that showed up so sharply in Puffle's numbers. Kush says the punishment of wrongly classifying a relevant file is much worse than the reward of correctly classifying an irrelevant file, and that the same holds for RAG. A file the scout skips is a file the agent never sees, and nothing downstream recovers it. A file the scout waves through costs some tokens and some latency. The two errors are not priced the same, so an average accuracy figure hides the thing that actually hurts.

He is not writing off the category. Kush says he does believe decision models will be big for harnesses, and that we are early, with the gap being decision models that can reason like a small LLM can.

For people building on it

This is one team on one workload, with no published eval numbers attached, so treat it as a report rather than a benchmark. But the asymmetry is worth checking against your own setup before you put a cheap classifier in front of retrieval. If a false negative silently removes evidence from the agent's context and a false positive only costs you a few thousand tokens, then the number you want is the miss rate on relevant items, not overall accuracy. A model can look fine on a balanced test set and still be the wrong shape for a gate where one side of the error is unrecoverable.

The practical read is that scout style gating is a place to measure before you ship, and that the fallback, in Puffle's case a small flash model doing the same job, is still a live option.