IT systems testing *

After my previous experiments with test latency, I started wondering whether I was still looking at the wrong statistic.

Mean latency is obviously incomplete.

p95 is better.

p99 is useful when rare slow runs matter.

But all of these measurements have one property that is easy to overlook:

They do not care about order.

If I take 300 test durations and randomly rearrange them, the mean remains identical.

So do p50, p95 and p99.

The total waiting time is identical too.

Yet from a developer perspective, ten slow runs scattered across an afternoon do not necessarily feel like ten slow runs arriving almost back to back.

That gave me a very specific experiment.

I generated one set of 300 test durations.

Then I created two timelines from exactly the same values.

In the first timeline, durations were randomly ordered.

In the second, slow runs were deliberately clustered.

Nothing else changed.

The result surprised me more than changing the latency distribution itself.

Both timelines had:

Can you train a cyberattack detector without ever showing it a real cyberattack?

It sounds like a contradiction. If you want a neural network to detect lateral movement, you would expect to show it lateral movement. I did the opposite: I generated an entire corporate network with its login history, staged an attack inside that artificial world, and trained networks on it. Not a single real row in the training data. The whole world is a 135-line config; each network has four thousand parameters and trains in seconds on a laptop, and the best result came from six of them, trained on six different invented worlds.

Then I pointed them at real data: the authentication logs of Los Alamos National Laboratory, 1.65 billion events, with red-team exercises labelled in them.

And it worked. The networks rank 3.6 million windows by suspicion, and the top twenty-three rows of that list hold sixteen real attacks and seven false alarms: all the analyst has to do is open those rows. A threshold counter on the same data needs a hundred and sixty-one thousand false alarms to reach the sixteenth attack. By AUC the synthetic training landed inside the range of published research trained on real labelled data, although the two cannot be compared head-on, and I will explain why.

I benchmark models on a repo of my own. This round I stopped testing whether they can fix a bug, and tested whether they can refuse to.

Eleven models got a ticket. Fifteen of its sixteen items were already fixed — decoys, to see who checks before patching. The last item asked them to document an invariant, and I stated that invariant as settled fact with three bullets of evidence.

All eleven agreed with me. The invariant was false — I had written the premise myself, and it took three lines of Python to break it.

Here is what they produced instead of catching it, what it cost in tokens, and the one model that came within ten lines of the answer and walked past.

Two test suites can have exactly the same average runtime and still create completely different development workflows. I modeled 300 code → test → fix cycles while keeping mean feedback latency close to ten seconds and changing only its variance. One runner behaved predictably. The other was usually fast but occasionally became extremely slow. The average stayed almost the same. The cost of waiting did not.

My Go API benchmark once showed an almost 7x throughput improvement after I moved a read-heavy endpoint from PostgreSQL to a local in-memory cache. The result looked obvious: PostgreSQL was the bottleneck, the cache removed it, and the API became much faster. Later I realized that this interpretation was too simple. ApacheBench had measured an entire request path consisting of HTTP handling, connection management, pgxpool waiting, SQL execution, decoding, serialization, and response writing. I decided to decompose that path and understand what the 7x result actually meant.

Daily tracking of job openings across 1,500+ tech companies reveals an emerging upward trend in AI testing roles.

A test taking 20 seconds does not necessarily cost a developer only 20 seconds. At some point during the wait, staying in the IDE becomes less attractive than opening a browser, checking documentation, reading a message, or starting another small task. I built a simulation of 300 code → test → fix cycles to estimate what happens when test latency begins to trigger context switching, and why cutting a test from 20 seconds to 10 may be much more valuable than cutting it from 5 seconds to 1.

A normal code review comment usually sounds harmless.

Rename this variable. Move this method. Add a test. Remove the duplicate condition. But sometimes an experienced developer leaves a much shorter comment: No.

Not maybe. Not could we simplify this. Just a clear rejection of the entire approach.

For a junior developer, this can feel strange. The code compiles, tests pass, the implementation is readable, and the ticket requirements seem complete. Why throw it away?

I started saving such cases after one of my pull requests was rejected for the third time in a week. The goal was not to prove that the reviewer was wrong. I simply wanted to understand what experienced engineers noticed before everyone else.

Over several months, I collected review discussions from backend services, internal tools, queue consumers, APIs, and data-processing jobs. I removed comments about formatting and naming. Only full design-level rejections remained.

The result was a small catalogue of professional paranoia.

And honestly, most of it was useful.

Deleting code sounds easy until the code belongs to a running product.

A function may have no direct callers but still be loaded through a plugin registry. A serializer may look duplicated but quietly preserve an old field name used by one customer. A command may not appear in analytics because it runs from cron at 3:10 a.m. on the first Sunday of each month. Python makes this even more fun because imports, decorators, entry points, reflection, monkey patches, and strings can all become hidden edges in the dependency graph.

The project in this story was a multi-tenant reporting backend written in Python. It accepted events, stored normalized records, generated reports, exported CSV and JSON files, and delivered them through HTTP, email, and object storage. Nothing huge. Around 140 API endpoints, 46 background tasks, PostgreSQL, Redis, and a queue.

The repository contained 62,914 lines of Python excluding tests and migrations.

That number was not the actual problem. The problem was that a small change in report filtering could require edits in the API schema, a service class, a repository, a filter translator, a query builder, an export adapter, and several nearly identical tests. The system had layers, but the layers did not reduce complexity. They distributed it.

The first plan was a rewrite. Fortunately, that plan died before production did.

Instead, the question became much simpler: How much code can disappear while externally observable behavior remains unchanged?

That wording changed the whole project.

At some point, ChatGPT stopped feeling like a tool and started feeling like a reflex.

Need a parser? Ask ChatGPT.
Strange exception in a background worker? Ask ChatGPT.
Forgot how a lock behaves under contention? Ask ChatGPT.
Need to rename a method? Apparently, that also required artificial intelligence.

Nothing looked wrong on the surface. Tasks were moving. Pull requests were getting merged. The code usually worked. Sometimes it even looked cleaner than what I would have written from scratch. But there was a small problem. A few days later, I often could not explain why a certain solution was built that way. I remembered the task, the final code, and maybe the prompt. The reasoning in between was missing.

That bothered me more than I expected.

So I set a simple rule: for six months, no ChatGPT, no Copilot Chat, no AI-generated code pasted into production. Documentation, source code, issue trackers, books, debuggers, profilers, and search engines were allowed. AI assistants were not.

The experiment started as a way to test my own dependence. It ended up changing how I design APIs, debug systems, read unfamiliar code, and even write comments.

Despite the annual summer lull, July unexpectedly saw a rise in the percentage of companies with open job listings—even for roles like Manual QA.

Unfortunately, the Tech Hiring Activity Index for June doesn’t bring any particularly good news.

However, there is a compelling trend: just two months ago, roughly three times as many US tech companies were hiring for classic manual QA as for dedicated AI testing roles. That gap has now narrowed to two to one.
But don’t let the growth rates fool you. While dedicated AI roles – those where AI is the deliverable, whether that means building, testing, and evaluating models or developing applications on top of ready-made ones, rather than merely using AI tools to support other work – are emerging, they are doing so from a very low baseline and still represent a tiny fraction of the overall tech job market.

Why I built Requester, a real-time HTTP load testing app for macOS, and what Swift structured concurrency taught me about telling the truth under load.

I wanted to hammer an HTTP endpoint and see what happened. Not read a summary report three minutes later — watch it, live, the way you watch a profiler.

The existing options are great but they all live in the terminal: wrk, hey, k6. I love them, but I kept wishing for a native window with a chart that moved. So I built one for macOS, in Swift and SwiftUI, and called it Requester.

This post is less “here are the features” and more “here are the three things I made building it.” The most interesting one: making the tool honest about backpressure turned out to be a design decision, not an accident.

My name is Vladimir Smirnov, and I am responsible for testing the trading backend at EXANTE. Development moves fast. Regression suites grow. With them come the chaos and inconsistency of test environments, and a steady rise in unstable failures, known as flakes. Real problems hide behind those flakes. How do we keep our automated tests in acceptable shape without spending too much time on it? That is what this article is about.

Our test framework broke as services grew. We redesigned it with clear layers to scale tests, reduce coupling, and support SDKs and AI.

Every prompt is a gamble. Your AI writes code that compiles, looks right, and silently breaks things you already built. Tests catch it before your users do. The best part? Your AI can write the tests for you.

~206 символов. Добавил "compiles, looks right" из A для конкретики, убрал "And here's the thing" на более кликбейтное "The best part?".

Добавить в начало статьи?

Regular expressions (also called regexp, or regex) are a mechanism for finding and replacing text. In a string, a file, multiple files... They are used by developers in application code, testers in autotests, and even just when working on the command line!

Why is this better than a simple search? Because it allows you to specify a pattern.

For example, you receive a date of birth in the DD.MM.YYYYY format. You need to pass it on, but in the YYYY-MM-DD format. How can you do this with a simple search? You don't know in advance what the exact date will be.

My name is Anatoly Bobunov, and I work as a Software Development Engineer in Test - or SDET for short - at EXANTE. When I joined one of our projects, I discovered that several of our test suites took more than an hour to run - painfully slow, to the point where running them for every merge request was simply unrealistic. We wanted fast feedback on each commit, but at that speed, it just wasn’t going to happen.

Eventually, through a series of small but precise improvements, I managed to speed things up to 8.5× faster, without rewriting the tests from scratch. In this article, I’ll walk through the bottlenecks we found and how we fixed them.

From outsourcing to product: a QA engineer’s honest journey to better releases, healthier work culture & real impact on the product.

During load testing of Tantor Postgres databases or other PostgreSQL-based databases using the standard tool pgbench, specialists often encounter non-representative results and the need for repeated tests due to the fact that details of the environment (such as DBMS configuration, server characteristics, PostgreSQL versions) are not recorded. In this article we are going to review author's pg_perfbench, which is designed to address this issue. It ensures that scenarios are repeatable, prevents the loss of important data, and streamlines result comparison by registering all parameters in a single template. It also automatically launches pgbench with TPC-B load generation, collects all metadata on the testing environment, and generates a structured report.