Mike Taylor calls a skill's short half-life an eval failure

Mike Taylor's position on agent skills is that every common complaint about them is really a complaint about missing evals. He was responding to a summary posted by Khe Hy of Felix from Cowork, who warned about being careful with skills because of their short half life and the need to update them with each model release, and who argued for driving more with outcomes.

Taylor took each point in turn. A short half life means you need to be able to recompile your skills for each new model. Skills that are often not necessary mean you should test against a baseline to check the skill is still needed. Superficial role personas mean you never tested which instructions actually produce what you want. And focusing on outcomes, in his framing, is just a description of what evals are.

The claim

The strong version of the argument is that the skill file stops being an artifact you maintain by hand at all. Taylor wrote that with good evals you do not even need to look at your skills, and can recompile them on your tasks every time a new model comes out, or when you want to move to a cheaper one.

Mike Taylor

@hammer_mt

X

If you have good evals you don't even need to look at your skills and can recompile them on your tasks every time a new model comes out

Mike Taylor

@hammer_mt

X

These are all failures to have proper evals

That last clause is the part worth noticing for anyone running a mixed model stack. If the prompt layer is regenerated against a task set rather than hand tuned, swapping in a cheaper model is a rerun rather than a rewrite.

What is not here

This is an argument, not a result. Neither post includes a task set, a model, a pass rate or a cost. Nobody in this exchange has shown the recompile-on-every-release loop running on real work, and Khe Hy's own note is about personal usage, saying he is using skills way less and barely uses projects or folders, while still seeing them as important for orgs with less power users who need repeatable workflows.

For people building on it, treat this as a prompt for your own eval harness rather than a method you can copy. Taylor has posted measured results before, including a case where Gemini beat a stacked Jev setup on image detection, and this is not one of those.