Lee Knowlton sorts work by whether an AI can verify it

Lee Knowlton posted a sorting rule for deciding how much AI to point at a piece of work. Most work, he says, falls into two buckets, AI-verifiable and not, and how you should work depends on which bucket you are in. His definition of verifiable is narrow and useful. It comes down to whether an AI can independently tell if work that it completes is good or not.

Lee Knowlton

@leeknowlton

X

sometimes a good judge is easy to build, sometimes it's so hard that you may as well not bother.

If the answer is yes, his advice is to design a verifier or a system of verifiers and then get out of the way. The reasoning is about attention, not capability. Once the verifier exists you are no longer the bottleneck, so no marginal attention is required as tasks scale. That is the case where throwing more model at the problem actually buys you something.

If the answer is no, the rule flips hard. Someone has to be the judge, and that someone is you. Knowlton's prescription there is deliberately unglamorous, one or two tasks at a time, avoid context switching, protect your focus. He calls it treating the work like the olden days but with AI help.

The middle bucket is where the work is

The interesting case is the maybe. For work that might be verifiable, Knowlton says you can try building an AI judge, a model that scores another model's output, but you need evals to test the judge's competence before you lean on it. He names @hammer_mt as his go-to on how to do that well, which lines up with Mike Taylor's recipe for training a judge until it agrees with humans. Get the judge good enough and the work moves into the verifiable bucket, and you are back to scaling.

He does not pretend that always works. Sometimes a good judge is easy to build and sometimes it is hard enough that you may as well not bother. That is the line most of these frameworks skip over, and it is the one that decides whether a weekend spent on an eval harness pays for itself.

Taylor, separately, framed the same gap as the opportunity. He posted that the most valuable thing you can do right now is find a way to make an unverifiable domain verifiable by AI and let it rip.

Mike Taylor

@hammer_mt

X

The most valuable thing you can do right now is find a way to make an unverifiable domain verifiable by AI and let it rip.

For people building on it, the fractal bit is the part worth stealing. Knowlton notes that larger pieces of work often contain verifiable sub-tasks, and the same judgment and eval-building process applies to those. You do not have to classify a whole job as automatable or not. You carve off the parts a verifier can hold and keep the rest on your own desk. This is one practitioner's heuristic, posted as notes, not a measured result.