Kush pulled Jev out of Puffle's harness
Kush says Jev kept making wrong calls on whether a file was worth reading, so Puffle went back to Qwen 3.7 flash. Where a false negative is expensive, a cheap classifier's error rate is the whole product.
Built with it
Lightpanda put Jev in its binary to spot cookie walls in 0.2s
Lightpanda compiled Jev into its binary and built judge() on top, so a crawler stops sending a whole page to an LLM just to learn it is a cookie banner. judge() asks a set of predefined questions and returns whether the page has real content and what kind: article, listing, product or docs. One call, about 0.2 seconds, and the demo run covers 30 sites in roughly 14 seconds.
Someone measured it
Runkle says context, then harness, then weights is the ROI order
Sydney Runkle posted eight takeaways from a conversation with Harrison Chase and Harry Stebbings, built on agent equals model plus context plus harness. She says optimizing context is the highest ROI move, and that model routing, the cost lever most people reach for first, works best in the harness where the routing context already lives.
Writing criteria
Harrison Chase's loop for a judge that runs on production traffic
Harrison Chase laid out the sequence: calibrate the judge on human labels, flag low scorers, confirm the real failures, add them to evals, fix, rerun. He was reacting to Shopify's Cody Mazza-Anthony at PyTorchCon, where critics diagnose each flagged output, a replay with a critic hint is scored again, and accepted repairs expand the training data.
prod finds the failures your offline dataset never had
The case against
Kush ripped Jev out of Puffle's harness over wrong file calls
Kush pulled Jev out of the Puffle harness and went back to Qwen 3.7 flash. He wanted it as a scout, checking whether a file is worth reading and tightening RAG precision, but says it kept making wrong assessments and leading the agent astray. He adds that wrongly classifying a relevant file is punished far harder than correctly skipping an irrelevant one, which is why it hurt their benchmarks.
jev constantly makes wrong assessments which leads the agent astray
Also on the timeline
Composio Firewall gates every agent action with Jev
Composio Firewall evaluates every action an agent takes using Jev plus SLMs Composio tuned on usage data. Karan Vaidya says it checks both directions, actions out and information coming back.
TypeSafe discloses a leak of all customer noul outputs
Diogo Almeida posted a disclosure saying TypeSafe became aware Friday evening of a massive unauthorized leak of all customer noul outputs. He linked it himself, noting the unfortunate timing.
Frequently asked questions
What is a cookie wall and how quickly can it be detected?
A cookie wall is a banner or prompt asking users to accept cookies before accessing page content. Lightpanda's judge() function can detect whether a page has a cookie wall and identify the actual content type in about 0.2 seconds.
What order should I optimize my AI agent for best return on investment?
According to Sydney Runkle, the highest ROI order is: first optimize context (the information fed to the model), then optimize the harness (the tools and structure controlling the model), then optimize weights (the model routing decisions). Model routing, which many teams try first, works best after context optimization is already in place.
How do you calibrate a judge to work reliably in production?
Harrison Chase describes a loop where you calibrate the judge against human labels, flag outputs with low confidence scores, confirm which ones are real failures, add those to your evaluation dataset, fix the judge, and rerun. The calibration step against human labels is critical because an uncalibrated judge will just reproduce its own mistakes.
Why does misclassifying relevant information hurt more than skipping irrelevant information?
When a judge incorrectly identifies relevant information as not worth reading, it causes an agent to miss important context, which leads to wrong decisions downstream. Correctly skipping irrelevant information has minimal negative impact by comparison, so the cost of false negatives typically outweighs the cost of false positives.