The Philosophy of Automated Tests: Management, Maintenance and Flakiness

[Ariless]

Agreed with the core idea "tests exist not to pass, but to fail." But there's a nuance around flaky tests.

The article suggests disabling an unstable test and investigating later. The problem is that sometimes the instability itself is the finding.

One case: a test was failing during BrowserStack integration with an error that looked like a connectivity issue. Classic candidate for "disable and investigate later." But the real cause was different: safari is not a valid browser name in the Playwright SDK - you need webkit. A misleading error was hiding a fundamental configuration problem.

If we had disabled the test as "flaky due to BrowserStack", we would have only found this when the entire Safari run started failing in production.

A flaky test with an unclear cause is not technical debt. It's a signal. Disabling it means losing the only indicator of the problem.

[sva_work]

Absolutely agree. I raise this topic in the paragraph on removing old tests. Unfortunately, in conditions of time deficits and business priorities, many teams ignore this matter. In some cases, this might be justified, but in others, it increases risks.

[Ariless]

Time pressure is real, and I'm not arguing every flaky test deserves immediate investigation.

The distinction I'd make: "we know this is environment noise, accepting the risk" vs "unclear root cause, we'll look later." The first is a conscious tradeoff. The second is where the signal quietly disappears.

The BrowserStack case fit the second pattern exactly - it looked like infrastructure noise, which is precisely why it was close to being filed and forgotten. The config problem only surfaced because I kept digging instead of labeling it flaky-infrastructure and moving on.