All streams

A test taking 20 seconds does not necessarily cost a developer only 20 seconds. At some point during the wait, staying in the IDE becomes less attractive than opening a browser, checking documentation, reading a message, or starting another small task. I built a simulation of 300 code → test → fix cycles to estimate what happens when test latency begins to trigger context switching, and why cutting a test from 20 seconds to 10 may be much more valuable than cutting it from 5 seconds to 1.

There is a question quietly buried inside every task handed to a language model, and it rarely gets asked out loud. Do you trust one model? Do you go looking for the best model? Do you run several models side by side and keep whatever each of them happens to catch? Or — one turn further still — do you hand that whole pile of partial answers to yet another model and ask it to combine them into one? Each answer sounds reasonable on its own. Each is also, on its own, incomplete.

The goal here is to actually run that experiment rather than assume how it comes out. A single broken bash script, with seven distinct bugs of varying severity, is the test case, and the research runs in two rounds.

Round one: one model, unaided. Four models — Sonnet 5, HY3, Qwen3-Max, DeepSeek-V4-Flash — each fix the script working alone, with no knowledge of what the other three are doing. That's the first framing: one model, fixing only what it personally notices.

Best of the four. Once all four independent attempts exist side by side, the next question follows on its own: if you had to ship just one of these four scripts, which is best? Table 1 works that out below — though "best" here will turn out to mean "most complete," not "complete." Even the strongest single script among the four leaves real bugs unresolved.

Round two: each model selects the best from all four. That incompleteness is why a second round exists at all. Seven more models — ling-3.0-flash, Mistral-Medium-3.5, Nemotron-3-Super-120B, Qwen3-Max, DeepSeek-V4-Flash, Gemini Pro, dots-studio-3-note — are each shown the same four fixed scripts from round one and given the same job: pick out whichever fix, from whichever of the four sources, is genuinely the best solution to each bug, and assemble those choices into a single script.

Who actually combined best. All seven believe they've picked the best pieces. That's exactly the open question this round is built to test: does the model whose combination is objectively the strongest also turn out to be the one whose combination reads as the strongest — or does a different one of the seven, working from the exact same four inputs, put together something better than the obvious pick? Table 2 is where that gets settled, not here.