Muse Wonder scores 52% on Dan Shipper's rewrite benchmark

Dan Shipper ran Muse Wonder on what he calls his senior engineer benchmark and got 52%, against GPT-Sol 6.1 at 71%.

The benchmark has one task. It asks the model to rewrite a vibe coded slop codebase from scratch, which Shipper describes as a good measure of a model's usefulness for real-world engineering. That is a single long-horizon job rather than a suite of isolated problems, so a score is closer to a verdict on one kind of work than a general capability number.

He ran it twice

The first version of the result did not survive. Shipper took the post down, said he had found a problem in the run, and restarted it. When the rerun finished he put the post back with the same conclusion. That detail is worth more than the score itself, because a benchmark number that holds through a restart by the person who owns the harness is harder to wave away as a bad run.

Dan Shipper

@danshipper

X

and it's back up! still bad

The variant that was not tested

The 52% is Muse Wonder Standard. Shipper notes that Wonder Full is not available on the API, which he calls inexplicable, and says a run will follow when he can get to it. So the number currently on the board is for the smaller configuration. Anyone reading 52% as the ceiling for the Wonder family is reading something Shipper did not measure.

For people building on it, treat this as one person's result on one task, not a published eval. There is no breakdown of where Wonder lost the 48%, no cost or latency figure alongside the accuracy, and no third model in the comparison beyond GPT-Sol 6.1. What you can take from it is that on a from-scratch rewrite of a messy codebase, the model that has been getting attention came in 19 points behind an incumbent, and the person running it tried twice.

The Wonder Full run is the one to watch for.

Dan Shipper

@danshipper

X

Wonder Full is, inexplicably, not available on the API