Jev Daily

Mike Taylor says a clone needs 400 shots to hit 95%

Three accuracy figures from one subject, and the question of whether the clone reasons like the team is still unanswered.

Mike Taylor posted accuracy numbers for predicting one person's decisions, measured in testing on Dan. One shot gets 60% accuracy, ten shot gets 85%, and it takes 400 shots to reach 95%.

Mike Taylor
@hammer_mt
X
In testing on Dan we can predict 60% accuracy one shot, or 85% accuracy 10 shot. You need 400 shots to get to 95%
Nov 6, 2026 路 View on X

That is the whole curve he gave, and the shape of it is the useful part. The jump from one example to ten buys 25 points. The next ten points cost roughly forty times more examples. If you are planning to clone a colleague's judgment from a handful of past decisions, the first number is the one you will actually hit, and it is not close to good enough to run unattended.

The question Taylor has not answered yet

The figures came in reply to Alexander, who asked whether the clones were reproducing anything more than style. His suggestion was to hold out real decisions and check the reasoning rather than the prose. Taylor's answer was three words long.

"That's phase 2", he wrote.

So the accuracy numbers describe how much data the approach needs, not whether the clone arrives at the right call for the right reason. A model that has read 400 of someone's decisions can match their output distribution and still be pattern matching on how they phrase things. That is exactly the failure mode Alexander flagged, and it is still open.

For people building on it

This is one subject. Taylor did not describe the task, the dataset, the baseline or what counts as a correct prediction, so 95% is not a benchmark figure you can compare against anything. Treat it as a cost signal rather than a quality one. If your plan involves cloning a decision maker from their Slack history, budget for hundreds of labelled past decisions before the thing is reliable, and assume you will need the held out test Alexander described before you trust it with a decision nobody reviews.

The sample size here is one person and the evaluator is the person building the system. Both of those should move how much weight you put on the curve.

Alexander 馃
@Alexandheir
X
would be interesting to hold out a few real decisions and see whether the clones reproduce the team's reasoning, vs just how everyone writes
Nov 6, 2026 路 View on X

Know what Jev is good at before you bet on it

One email a day. Under three minutes. What builders found out about Jev yesterday.

Get the next one by email

Jev, read daily so you do not have to. The builds, the benchmarks, the criteria that worked and the cases where it lost, from the people shipping on TypeSafe AI's System One model.