Mike Taylor's eval recipe trains a judge to 90% human agreement

Mike Taylor posted a full walkthrough of how he builds evals, and the load bearing number in it is 90%. That is the agreement threshold he pushes an AI judge to before he trusts it. The method is to collect feedback from human domain experts, then rewrite the judge's prompt until it agrees 90% with those human judgements. After that you can run A/B tests and let the model grade prompt variants until something performs better or cheaper.

Mike Taylor

@hammer_mt

X

Measuring "if it improved" is 90% of the work and 10% is infra.

The grading format matters as much as the agreement rate. Taylor says simple yes/no questions work best, with 1 or 0 assigned for each specific thing that could go right or wrong, so the grade is impossible to misinterpret. His examples are concrete. Did this article have a specific AI tell, one judgement per type of tell. Did the powerpoint use our brand colors. Is it following our coding conventions.

The infra part is the small part

Taylor's framing is that evals are saving past work and replaying it with new models or prompts to see if it improved. He puts that to three uses, seeing where it fails so you can tell your boss or users, optimizing until it improves, and making it work on a cheaper or open source model.

For storage he suggests telling your agent to store tasks in Cloudflare R2 in Harbor task format, saving both the instructions and how the folder looked before the task started. Runs happen in a docker container so the agent cannot peek at the answer, with your own auth credentials copied across so you burn a Codex or Claude subscription instead of API tokens. The one extra he recommends is mocking any tools the agent uses, Slack channels, Google search, document search, so a replay does not come back with something new every time.

Where humans sit

Taylor rates humans as the slowest and most expensive graders but usually the most accurate when they are domain experts, and his advice is to bring the AI output to where they already work. Edits as comments in Google Docs for an editor, actual PRs for engineers, Notion copy for marketers. Get them talking out loud on a call and let AI turn the transcript into structured data later.

He also says the human is not the ceiling, because domain experts get things wrong often enough that the data points at better methods. For people building on it, the binary yes/no grader is the same shape as a decision model choice set, which is our read rather than his, so the loop ports onto judging a Jev call without much translation. Taylor's next step, deterministic code tests closer to an RL environment, is deferred to a later post.