I Simulated 300 Code → Test → Fix Cycles to Find Out When Test Latency Starts Breaking Developer Focus

A test taking 20 seconds does not necessarily cost a developer only 20 seconds. At some point during the wait, staying in the IDE becomes less attractive than opening a browser, checking documentation, reading a message, or starting another small task. I built a simulation of 300 code → test → fix cycles to estimate what happens when test latency begins to trigger context switching, and why cutting a test from 20 seconds to 10 may be much more valuable than cutting it from 5 seconds to 1.

Introduction

Half a second does not feel like waiting.

I change a condition, run the test, and by the time my eyes move from the code to the test panel, the result is already there. The entire sequence feels like one action.

At five seconds, something changes. I notice the pause.

At twenty seconds, I have enough time to do something else.

That difference bothered me because most discussions about test performance stop at wall-clock time. A test used to take 35 seconds, now it takes 12. We saved 23 seconds. Multiply that by the number of runs and the optimization looks easy to quantify.

But that assumes the developer behaves identically during both waits.

I do not think that assumption is correct.

A sufficiently short delay stays inside the current cognitive loop. A sufficiently long one creates an opportunity to leave it. Once I leave, the cost of the test is no longer equal to the test runtime. I also need to notice that the test has finished, return to the IDE, locate the previous execution point in my head, reconstruct the hypothesis I was checking, and only then make the next change.

I wanted to see what that would look like as a system rather than as a vague productivity argument.

I did not have telemetry from hundreds of real developers, so I did not try to manufacture an empirical result. Instead, I built a small simulation of 300 feedback cycles and deliberately treated switching behavior as a parameter.

The question was not whether exactly 36 percent of developers leave the IDE after ten seconds.

The question was more interesting:

What happens to the economics of the development loop once the probability of leaving the task starts growing quickly?

1. The Model: 300 Feedback Cycles and Six Latency Levels

I reduced development to the smallest loop that still resembles real programming:

code change → test launch → result → next change.

One pass is one cycle.

The simulated session contains 300 cycles divided equally between six latency levels:

• 0.5 seconds

• 2 seconds

• 5 seconds

• 10 seconds

• 20 seconds

• 40 seconds

Each latency level gets 50 cycles.

I intentionally do not model the test itself as computationally expensive. The test can be thought of as finishing instantly internally, followed by an artificial delay before the result becomes available.

That separation matters.

If a 40-second test also consumes more CPU, generates more logs, performs I/O, and touches a database, I am no longer measuring latency alone. I am mixing several variables into one experiment.

For each simulated cycle I keep four quantities.

The first is test latency t.

The second is whether a context switch happened before the result became available.

The third is detection delay: the amount of time between test completion and the developer noticing that it has completed.

The fourth is reconstruction delay: the time required after returning to the editor before meaningful work on the original task continues.

Those last two quantities are where the model becomes more interesting than a stopwatch.

Suppose I run a test with a 20-second delay.

After six seconds I open a browser.

At second fourteen I am already reading something unrelated.

The test completes at second twenty.

I return to the IDE at second twenty-six.

Then I spend another four seconds figuring out which branch of the condition I was trying to fix.

The computer took 20 seconds.

The feedback loop took 30.

That difference does not appear in a test report.

Why I use a probability instead of a hard threshold

My first instinct was to define a threshold.

Perhaps developers stay focused below eight seconds and switch above eight seconds.

That produces a neat model, but I do not believe human behavior works that way.

At seven seconds I might stay in the editor.

At the next seven-second wait I may open documentation.

At twelve seconds I might still stare at the test runner because the bug is interesting enough.

The transition should therefore be probabilistic.

I model switching probability using a logistic function:

P(switch | t) = 1 / (1 + exp(-(a + b × ln(1 + t))))

The exact coefficients are configurable.

The important part is the shape.

At very low latency, switching probability remains close to a baseline. In the middle region it rises quickly. At high latency it approaches saturation because almost every wait creates enough empty time for another activity.

I use ln(1 + t) rather than t directly because perceived waiting time is unlikely to scale linearly.

The difference between one second and five seconds is enormous inside an interactive loop.

The difference between 101 and 105 seconds is practically irrelevant. By that point the interaction has already stopped being interactive.

For one baseline run I used approximate switching probabilities of 0.02, 0.04, 0.12, 0.36, 0.68 and 0.88 across the six latency levels.

These are not measurements.

They are assumptions.

That distinction is important because the simulation is not trying to prove that developers have a universal ten-second attention threshold. It is trying to show how the system behaves if context switching follows a transition curve of this general shape.

And that is enough to expose a surprisingly large hidden cost.

2. Runtime Is Not Feedback Time

The obvious metric for a test is duration.

I think that is the wrong metric for interactive development.

What I actually care about is the distance between forming a hypothesis and being ready to act on the answer.

Imagine that I change one line because I believe a boundary condition is wrong.

The hypothesis exists before the test starts.

The feedback loop ends only when the test result has been incorporated into my next decision.

If the test finishes while I am reading another application, the loop is not over.

If I return to the editor but need ten seconds to understand why I changed the code, the loop is still not over.

So I define effective feedback latency as:

T_effective = T_test + T_detection + T_reconstruction

For cycles without a context switch, both additional terms are close to zero.

For cycles with a switch, they can become substantial.

At the aggregate level this can be approximated as:

E[T_effective] = T_test + P_switch × E[C_return]

where C_return represents the combined detection and reconstruction cost.

Take a five-second test.

Suppose the context-switch probability at this latency is 0.12 and a switch adds an average of 1.1 seconds.

Then:

T_effective = 5 + 0.12 × 1.1

T_effective ≈ 5.13 seconds

The difference is negligible.

Now take a 20-second test.

Suppose switching probability is 0.68 and an interruption adds an average of 9.7 seconds before meaningful work resumes.

Then:

T_effective = 20 + 0.68 × 9.7

T_effective ≈ 26.6 seconds

The test still reports 20 seconds.

The modeled developer experiences something closer to 27.

That is a 33 percent increase without making the test itself one millisecond slower.

This is the first important result of the model: wall-clock runtime underestimates latency once the developer starts leaving the feedback loop.

But there is a second effect that I find more damaging.

Long waits change how I write code.

If feedback is nearly instantaneous, I am comfortable making very small changes.

I change one condition.

Run the test.

Change another condition.

Run it again.

Each failed run leaves me with a tiny search space.

But imagine every run costs 40 seconds.

After a while, running the test after each small modification starts to feel wasteful. I make three changes before testing. Then perhaps five.

This is rational local behavior.

It also makes debugging worse.

If a test fails after one modification, I have one primary suspect.

If it fails after five modifications, I first need to determine which of the five invalidated my assumption.

The slow feedback loop therefore modifies its own workload.

The sequence becomes:

slow feedback → fewer runs → larger batches of changes → larger failure search space → more expensive debugging → stronger incentive to avoid frequent runs.

This is a positive feedback mechanism in the unpleasant sense.

The latency does not merely waste seconds.

It changes the granularity of development.

A small Monte Carlo simulation

The model itself does not require much code. A stripped-down version looks roughly like this:

import math
import random
from statistics import mean

LATENCIES = [0.5, 2, 5, 10, 20, 40]

SWITCH_PROB = {
    0.5: 0.02,
    2: 0.04,
    5: 0.12,
    10: 0.36,
    20: 0.68,
    40: 0.88,
}

RETURN_COST = {
    0.5: 0.3,
    2: 0.6,
    5: 1.1,
    10: 3.8,
    20: 9.7,
    40: 18.0,
}

def simulate_cycle(latency):
    switched = random.random() < SWITCH_PROB[latency]

    if not switched:
        return {
            "latency": latency,
            "switched": False,
            "return_cost": 0.0,
            "effective_latency": latency,
        }

    avg_cost = RETURN_COST[latency]

    # Positive skew: most interruptions are short,
    # but a few become significantly longer.
    sigma = 0.45
    mu = math.log(avg_cost) - sigma * sigma / 2

    return_cost = random.lognormvariate(mu, sigma)

    return {
        "latency": latency,
        "switched": True,
        "return_cost": return_cost,
        "effective_latency": latency + return_cost,
    }


cycles = []

for latency in LATENCIES:
    for _ in range(50):
        cycles.append(simulate_cycle(latency))

random.shuffle(cycles)

for latency in LATENCIES:
    group = [x for x in cycles if x["latency"] == latency]

    switch_rate = mean(x["switched"] for x in group)
    effective = mean(x["effective_latency"] for x in group)

    print(
        latency,
        round(switch_rate, 3),
        round(effective, 2),
    )

There is one deliberate detail here.

Return cost is not constant.

I generate it from a log-normal distribution.

That matches the kind of process I am trying to represent better than a normal distribution. Interruption costs cannot go below zero, most of them should remain relatively small, but occasionally one should become much larger.

A developer opens documentation.

Then notices a message.

Then answers it.

Then reads another message.

The resulting interruption has a long right tail.

Using a fixed return penalty would hide exactly that behavior.

And once this tail exists, average test latency becomes an even weaker description of the experience.

3. The Most Valuable Optimization May Be Near the Transition Region

Suppose I have two optimization opportunities.

The first reduces a test from five seconds to one.

The second reduces another test from twenty seconds to ten.

If I look only at runtime, the first saves four seconds and the second saves ten.

Nothing surprising there.

But suppose the five-second test was already below the behavioral transition region. Developers almost always stayed in the same context.

Going from five to one second feels nicer, but it mostly produces a direct four-second saving.

Now look at the twenty-second test.

At twenty seconds, context switching may be common.

At ten seconds, it may happen much less often.

Reducing the runtime therefore creates two gains simultaneously.

The test finishes sooner.

And the probability that the developer is still mentally attached to the original problem when it finishes becomes higher.

This makes optimization value nonlinear.

A ten-second reduction around the transition region can be more valuable than ten seconds removed elsewhere.

It also means percentage-based performance improvements can be misleading.

Reducing two seconds to one is a 50 percent improvement.

Reducing twenty seconds to twelve is only a 40 percent improvement.

Yet the second optimization may have a much larger effect on the development process.

This led me to think about test latency less as a performance metric and more as an interaction-design metric.

An IDE action that responds in 100 ms, one second, ten seconds and two minutes does not merely differ in speed.

Each duration belongs to a different interaction regime.

At 100 ms, I perceive a direct response.

At one second, I wait.

At ten seconds, I negotiate with myself about whether waiting is worth it.

At two minutes, I start planning another activity.

The system has changed modes even if nobody explicitly designed those modes.

The t50 value

A useful derived metric is t50.

I define t50 as the latency at which the fitted switching model predicts a 50 percent probability of leaving the current context.

For the logistic model:

P(switch | t50) = 0.5

At this point the exponent becomes zero, so:

a + b × ln(1 + t50) = 0

Therefore:

t50 = exp(-a / b) - 1

I would not interpret t50 as an attention-span constant.

It is not a biological property of a programmer.

It is a property of a particular person, environment, task type, toolchain, notification environment and probably even time of day.

But as an engineering metric it could still be useful.

Imagine measuring the same developer workflow before and after changing the test infrastructure.

The exact t50 does not need to generalize to everyone.

What matters is whether the feedback system moves a large percentage of daily interactions below or above that transition.

That is a much more interesting optimization target than shaving milliseconds from already-fast operations.

4. What I Would Measure in a Real IDE

The simulation exposes the mechanism, but the obvious next step is replacing assumptions with telemetry.

Most of the required events already exist somewhere inside an IDE, test runner or operating system. The difficult part is joining them into a single feedback-loop trace.

For each run I would want to collect:

• timestamp of the last meaningful code edit before test launch

• test launch timestamp

• test completion timestamp

• IDE focus loss and focus regain

• active application category during the interruption

• first editor interaction after returning

• first actual code modification after returning

• number of files visited before the next change

• number of code edits accumulated before the next test run

• whether the next run passed or failed

I would deliberately avoid recording actual source code, browser URLs, message contents or keystrokes. None of them are necessary for the measurement, and collecting them would turn a simple latency experiment into a privacy problem.

From those events I would calculate several metrics.

Raw test latency is still useful.

But I would add feedback completion latency, switching probability, detection delay, reconstruction delay and edit batch size.

Edit batch size is particularly interesting.

If slow tests really change development strategy, the median number of edits between test runs should increase as feedback becomes slower.

That would give an observable behavioral consequence rather than merely an assumption about attention.

Another useful metric is failure recovery depth.

After a failed test, how many files does the developer inspect before the next meaningful edit?

If slower feedback encourages larger batches, failed runs may require wider navigation through the codebase.

At that point test latency begins interacting with code comprehension cost.

And then the interesting question stops being how fast are our tests.

It becomes how much additional cognitive search does our feedback system create.

5. Predictability May Matter More Than the Average

While building the model, I ran into a variant of the problem that I now find more interesting than the original one.

Imagine two test runners.

Runner A always responds in ten seconds.

Runner B responds in one second half the time and nineteen seconds the other half.

Both have the same average latency.

From a conventional performance dashboard they are equivalent.

From a developer perspective I doubt they are.

With Runner A, I learn the timing.

I know that waiting is probably reasonable.

With Runner B, I do not know which world I am currently in.

After one second without a result, I still do not know whether the test is about to finish or whether eighteen more seconds remain.

That uncertainty changes the decision.

The choice to stay or switch now depends on the conditional probability of completion.

If the test has already been running for t seconds, what is the probability it will finish within the next Δt?

In reliability terms, this is related to the hazard function:

h(t) = f(t) / S(t)

where f(t) is the probability density of completion time and S(t) is the probability that the test is still running at time t.

A nearly deterministic ten-second test has a very different hazard profile from a highly variable test with the same mean.

Near the expected completion time, the deterministic process becomes increasingly worth waiting for.

A heavy-tailed process may not.

This suggests another optimization target: latency variance.

A test suite with a slightly worse mean but tight timing may create a better interactive experience than a faster suite with unpredictable long tails.

The usual p95 and p99 metrics already expose some of this, but they are normally interpreted as infrastructure reliability numbers.

I think they also have a human consequence.

A long-tail test runner teaches the developer not to trust waiting.

Once that habit forms, even fast runs may trigger switching because the developer no longer expects the result to arrive soon.

In other words, history may matter.

The switching probability should probably not be modeled only as P(switch | t).

A more realistic model might be:

P(switch | t, H)

where H represents recent latency history.

If the last five runs completed in two seconds, I may tolerate the sixth for longer.

If the previous run unexpectedly took 45 seconds, I may leave after three.

The feedback system therefore has memory even if the test runner does not.

Conclusion

I started with a simple question: after how many seconds does a developer stop waiting for a test and start doing something else?

I no longer think a single threshold is the right answer.

The useful concept is a transition region.

Below it, test latency remains part of the coding action.

Inside it, context switching becomes increasingly likely.

Above it, leaving the feedback loop becomes normal behavior, and raw runtime stops representing the actual cost of the test.

The 300-cycle simulation does not tell me where that region is for every developer. It is not supposed to.

What it shows is why the region matters.

Once switching probability starts rising, one additional second of test latency can cost more than one second of development time. It can create completion-detection delay, context reconstruction, larger edit batches and a wider debugging search space.

That changes how I think about test optimization.

The real target is not minimum runtime.

It is minimum hypothesis-to-feedback latency while the original hypothesis is still active in the developer's head.

And I suspect the next experiment should not focus on average latency at all.

I would keep the mean constant and vary only predictability.

Five seconds every time.

Then a distribution that averages five seconds but occasionally jumps to twenty or thirty.

My guess is that the second system will produce more context switches even if both look identical in an average-duration dashboard.

If that turns out to be true, the worst property of a slow development tool may not be that it makes us wait.

It may be that it teaches us that waiting is irrational.