Jev counts letters at 70%, perfect on character lists
Paolo Rosson found the strawberry question is a coin flip for Jev until you split the word into characters first.
Paolo Rosson asked Jev to count the r's in strawberry and got a coin flip. Across repeated asks, 47% of answers said three and 47% said two, which he described as the same as every LLM. Over a set of 168 test words he measured 70% accuracy overall, with the errors concentrated in doubled letters, which Jev undercounts.
168/168. same model, same question, 260ms
47% says 3, 47% says 2. a coin flip, same as every LLM.
Then he changed the input shape and nothing else. Instead of the word strawberry, he passed the letters as a list, `["s","t","r","a","w","b","e","r","r","y"]`. Same model, same question. He reports 168 of 168 correct at 260ms.
What moved the number
The interesting part is what did not change. Rosson's second run used the same model and the same question, so the fix was not a different constrained choice set or a better prompt about counting carefully. It was giving the model input that already had the boundaries the task asks about. A word arrives as tokens that do not line up with letters. A JSON array of single characters does.
This is one person's measurement on one class of task, not a published eval, and 168 words is a small set. It is also the kind of task where a single sequence of identical failures is the whole point, because the failure is systematic rather than random. Undercounting doubled letters is the signature of a model that never saw the individual letters.
Rosson tagged Diogo Almeida asking whether this was expected. Almeida replied that at this point nothing is expected and told everyone to keep cooking, which does not explain the gap.
For people building on it
If any part of your pipeline depends on character level facts, spelling checks, letter frequency, string length, position of a substring, do the splitting in your own code before the call and hand Jev the array. The 70% figure is what you get if you pass the raw string and trust the answer. The 260ms at 168 of 168 is what the same model does when the preprocessing is already done. Confidence scores are not the lever here, and neither is a tighter prompt. The input representation is.