Anatomy of a Broken Benchmark Runner: How Seven AI Models Fixed (or Didn't Fix) run-code.sh

There is a question quietly buried inside every task handed to a language model, and it rarely gets asked out loud. Do you trust one model? Do you go looking for the best model? Do you run several models side by side and keep whatever each of them happens to catch? Or — one turn further still — do you hand that whole pile of partial answers to yet another model and ask it to combine them into one? Each answer sounds reasonable on its own. Each is also, on its own, incomplete — and the only way to find out which incompleteness actually hurts you is to run the experiment rather than assume the answer.
This article runs it, in four steps that build on each other.
One model. A single broken bash script, with seven distinct bugs of varying severity, was handed to four models — Sonnet 5, HY3, Qwen3-Max, DeepSeek-V4-Flash — each working alone, each with no knowledge of what the others were doing. That's the first framing: one model, unaided, fixing only what it personally noticed.
Best model. Comparing those four independent attempts side by side produces the second framing almost automatically: if you had to pick just one of these four scripts to actually use, which is best? Table 1 answers that, and the answer is Qwen3-Max — but "best" here means "most complete," not "complete," a distinction the table is careful to preserve rather than paper over. Even the single best model in this test left real bugs unresolved.
Multi models, combined. That incompleteness is precisely why a third framing exists: instead of picking one winner and discarding the other three, what if a model's job is to read all four and assemble a single script that keeps every good idea, regardless of which of the four originally had it? Seven more models — ling-3.0-flash, Mistral-Medium-3.5, Nemotron-3-Super-120B, Qwen3-Max, DeepSeek-V4-Flash, Gemini Pro, dots-studio-3-note — were given exactly that job.

















