There is a performance graph that can make a backend look much healthier than it really is: requests per second. Imagine a service that keeps processing roughly the same amount of traffic for twenty minutes. PostgreSQL CPU is nowhere near 100 percent, the number of completed queries per second barely changes, and average latency moves only a little. Nothing looks broken enough to start an incident. Then p99 starts climbing, first slowly, then much faster, while throughput remains almost perfectly flat. That was the part that bothered me, because at first it looked contradictory. If the database still completes the same number of requests, why are some of them suddenly taking so much longer?
The obvious suspects are familiar: a query plan changed, the cache got colder, autovacuum arrived at the wrong time, an index stopped fitting into memory, or storage became slower. All of those are real problems, but there is another failure mode that is easier to miss because PostgreSQL itself may still be doing its job quite well. Requests can spend more and more time waiting for a free connection before the SQL is even sent to the database. PostgreSQL sees only the second part of the story. From its point of view, a query that waited 80 ms in the application and then executed in 2 ms is still a 2 ms query.
That distinction is easy to forget because application monitoring often wraps everything into one database span. From the application side, a request may look like one database operation, but internally it is closer to this:
request latency = pool wait + query execution + application work
Throughput does not tell us how those parts are distributed. If one request finishes in 3 ms and another in 120 ms, both still count as one completed request. Even average latency can hide the problem for a while. If 99 requests take 3 ms and one takes 150 ms, the mean is still only around 4.5 ms. That number does not look dramatic, even though one percent of users are seeing something completely different from the rest. This is exactly where p99 becomes useful. It is not a magic metric, but it asks a more interesting question: how bad is the experience near the slow end of the distribution? Once the connection pool becomes the queue, the slow end usually moves first. The system can keep producing the same RPS for quite a while because requests eventually finish. They just spend more time waiting before they get their turn. A rough utilization estimate already gives some intuition. If requests arrive at rate λ, each database operation holds a connection for an average time S, and the pool has m connections, then utilization is roughly:
ρ ≈ λ × E[S] / m
Suppose the service performs 700 database operations per second and each connection is occupied for about 5 ms on average. With eight connections, average utilization is around 44 percent. Cut the pool to four connections and the same workload moves close to 88 percent. The service may still have enough theoretical capacity to keep throughput almost unchanged, but there is much less space for bursts, slower queries, scheduler noise, garbage collection pauses, or any other small disturbance. Now add variance. Most queries take 1 or 2 ms, but a few occasionally hold a connection for 40 or 50 ms. In a four-connection pool, one such query temporarily consumes 25 percent of the available database concurrency. Two overlapping slow queries take half of it. Fast requests arriving behind them are still fast from PostgreSQL's point of view, but they may spend tens of milliseconds waiting before they are even allowed to run. That is the part that makes this behavior so deceptive. Throughput can look normal, SQL execution can look normal, and the application can still be getting slower.
A small experiment with an intentionally boring workload
For the test, I wanted to remove as many distractions as possible. A complicated SQL query would create too many possible explanations. If p99 moved, it would be easy to blame the optimizer, indexes, joins, cache state, data distribution, or something else. So the workload is deliberately boring: one indexed lookup against a table with one million rows and a controlled small percentage of slower operations. The slow path uses pg_sleep. Obviously nobody is suggesting that production code should do this. It is just a simple way to create repeatable service-time variance without changing the query shape, schema, indexes, or application logic between runs. The useful variable is the pool size. Everything else stays the same: same table, same query, same offered request rate, same percentage of slow operations. If PostgreSQL execution time remains roughly stable while application p99 starts rising, the pool becomes a much more believable explanation than the query plan.
DROP TABLE IF EXISTS bench_items; CREATE TABLE bench_items ( id bigint PRIMARY KEY, payload text NOT NULL ); INSERT INTO bench_items SELECT g, repeat(md5(g::text), 4) FROM generate_series(1, 1000000) AS g; ANALYZE bench_items; EXPLAIN (ANALYZE, BUFFERS) SELECT payload FROM bench_items WHERE id = 500000; WITH delay AS ( SELECT pg_sleep(0.050) ) SELECT b.payload FROM bench_items AS b CROSS JOIN delay WHERE b.id = 500000; SELECT state, wait_event_type, wait_event, count(*) AS connections FROM pg_stat_activity WHERE datname = current_database() GROUP BY state, wait_event_type, wait_event ORDER BY connections DESC;
The table size is not sacred. One million rows is just enough to make the test feel like a real database rather than a toy with ten records. The important part is consistency. If every run changes both pool size and workload, the result becomes hard to interpret. Here the schema stays the same, the query stays the same, the request rate stays the same, and the only thing that changes is how many database connections the application is allowed to use at once. The application side is where things become more interesting, because one timer is not enough. Measuring only total request latency tells us that something is getting slower, but not where. Measuring only query duration tells us what PostgreSQL did after it received the request. For this experiment, three timings matter: total request latency, time spent waiting for Acquire, and query execution after a connection has already been obtained.
I also wanted a fixed offered load instead of a classic closed-loop benchmark. This matters more than it first appears. In a closed-loop test, a worker sends the next request only after the previous one finishes. If latency starts increasing, the workers naturally generate fewer requests. The benchmark reduces pressure exactly when the tested system starts struggling. That can produce a very stable-looking test because the load generator quietly stops asking for as much work. The following Go program keeps generating requests at a target rate and records the queueing time separately from SQL execution. Five percent of operations hold a connection for an additional 50 ms. The rest perform the same indexed lookup.
package main import ( "context" "flag" "fmt" "log" "os" "sort" "sync" "sync/atomic" "time" "github.com/jackc/pgx/v5/pgxpool" ) type sample struct { total time.Duration acquire time.Duration query time.Duration } var counter atomic.Uint64 const query = ` WITH delay AS ( SELECT pg_sleep($2::double precision) ) SELECT b.payload FROM bench_items b CROSS JOIN delay WHERE b.id = $1 ` func main() { poolSize := flag.Int("pool", 8, "max connections") rps := flag.Int("rps", 700, "requests per second") duration := flag.Duration("duration", 45*time.Second, "test duration") flag.Parse() cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL")) if err != nil { log.Fatal(err) } cfg.MaxConns = int32(*poolSize) cfg.MinConns = int32(*poolSize) pool, err := pgxpool.NewWithConfig(context.Background(), cfg) if err != nil { log.Fatal(err) } defer pool.Close() interval := time.Second / time.Duration(*rps) ticker := time.NewTicker(interval) defer ticker.Stop() var wg sync.WaitGroup results := make(chan sample, *rps*int(duration.Seconds())) end := time.Now().Add(*duration) for time.Now().Before(end) { <-ticker.C wg.Add(1) go func() { defer wg.Done() start := time.Now() ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() acquireStart := time.Now() conn, err := pool.Acquire(ctx) if err != nil { return } acquire := time.Since(acquireStart) defer conn.Release() n := counter.Add(1) delay := 0.0 if n%20 == 0 { delay = 0.050 } queryStart := time.Now() var payload string err = conn.QueryRow( ctx, query, int64((n%1_000_000)+1), delay, ).Scan(&payload) if err == nil { results <- sample{ total: time.Since(start), acquire: acquire, query: time.Since(queryStart), } } }() } wg.Wait() close(results) var total, acquire, queryTimes []time.Duration for s := range results { total = append(total, s.total) acquire = append(acquire, s.acquire) queryTimes = append(queryTimes, s.query) } printP99("total", total) printP99("acquire", acquire) printP99("query", queryTimes) stats := pool.Stat() fmt.Printf( "empty_acquires=%d empty_wait=%s\n", stats.EmptyAcquireCount(), stats.EmptyAcquireWaitTime(), ) } func printP99(name string, values []time.Duration) { sort.Slice(values, func(i, j int) bool { return values[i] < values[j] }) if len(values) == 0 { return } i := int(float64(len(values)) * 0.99) if i >= len(values) { i = len(values) - 1 } fmt.Printf("%s p99=%s\n", name, values[i]) }
The runs themselves are simple. Keep the request rate unchanged and repeat the same test with pool sizes such as 32, 16, 8, 6, 4 and 3. There is no need to restart PostgreSQL before every run because that would also change cache state and introduce another variable. The point is not to create a perfect academic laboratory. It is to keep the surrounding workload boring enough that a change in queueing behavior is easy to notice.
A run can look like this:
DATABASE_URL=postgres://... go run . -pool 8 -rps 700 -duration 45s
Then change only the pool size. The graph worth watching is not pool size versus throughput. That one may stay surprisingly flat for quite a while. The more useful comparison is pool size versus total p99, acquire p99 and query p99.
The queue moves before throughput breaks
With a large enough pool, most requests acquire a connection almost immediately. Total latency is then mostly determined by query execution, and the intentionally slow five percent are visible without causing much secondary queueing. As the pool becomes smaller, query execution does not necessarily get slower. PostgreSQL can continue doing roughly the same work at roughly the same speed once a request reaches it. What changes is the time before that point.
This is where EmptyAcquireCount becomes more interesting than CPU usage. It tells us that an acquisition happened while no connection was immediately available. EmptyAcquireWaitTime adds the missing part: how expensive those waits became. If query p99 stays stable while acquire p99 grows from almost nothing to tens or hundreds of milliseconds, SQL optimization is probably not the first thing to work on. This explains a monitoring pattern that can look impossible at first. PostgreSQL dashboards show no dramatic increase in query execution time, while traces from the API clearly show database operations becoming slower. Both can be correct. PostgreSQL measures work after the query arrives. The application span may include the wait for a connection before anything is sent to the server.
That becomes even easier to miss behind an ORM or database abstraction. Application code may contain one harmless-looking query call, while underneath it the request first waits for a pool slot, then waits on the network, then executes SQL, then decodes the result. When those stages are collapsed into one timer, every slowdown looks like a slow database even when the database is not where the queue is forming. The slow-path part of the experiment also shows another useful effect. A few slow queries can damage unrelated fast ones. Imagine a pool with four connections. One slow operation occupies one connection for 50 ms, leaving three. Another slow operation overlaps with it, and now half of the available concurrency is gone. A fast indexed lookup arriving at that moment may still need only 2 ms inside PostgreSQL, but it can wait 20, 40 or 80 ms before getting a connection.
So one query class can increase the latency of another query class without making the second SQL statement slower at all. This matters in real services because very different workloads often share the same pool. A normal API request, report generation, scheduled job, health check and background worker can all compete for the same connections. The database does not care that developers mentally classify them as separate features. If they share one pool, they share one queue. That does not mean every service needs separate pools for every endpoint. Usually that would be overkill. But it does mean that a small number of slow background operations can affect latency-sensitive traffic in a way that is not obvious from SQL execution metrics alone.
Increasing the pool may fix the symptom or just move the queue
At this point the tempting solution is obvious: increase MaxConns. Sometimes that is exactly the right thing to do. If the pool is unnecessarily small and PostgreSQL still has plenty of spare capacity, allowing more concurrent work can reduce application-side waiting. The dangerous part is turning that into the general rule that more connections always mean less latency.
A PostgreSQL connection is not a free token. More application connections mean more concurrent work entering the database. At some point those queries start competing for CPU, memory bandwidth, storage, locks, cache pages and internal resources. A small pool can create a queue in the application. A huge pool can move the same queue into PostgreSQL. The queue did not disappear. It changed address. This is why pool tuning cannot be done by looking at one side only. The useful question is not how many connections the application is capable of opening. The useful question is how much concurrent database work the whole system can sustain while keeping latency acceptable. For simple indexed reads, the useful concurrency level may be quite high. For heavy analytical queries, lock-heavy transactions, or storage-bound workloads, it may be much lower than expected.
There is also a deployment-level trap. A pool size is configured per application process, but PostgreSQL sees the sum of all of them. Ten service instances with 50 connections each can potentially create 500 database connections. Autoscaling makes this even more interesting. Application capacity increases, new replicas start, every replica creates another pool, and database concurrency rises even though nobody changed a PostgreSQL setting. That makes connection-pool size part of system capacity planning, not just an application detail.
p99 has its own traps too. Percentiles are not additive. If acquire p99 is 40 ms and query p99 is 30 ms, total p99 is not automatically 70 ms. The slowest pool wait and the slowest query may belong to different requests. That is why per-request timings are more useful than trying to reconstruct end-to-end latency from separate percentile graphs later. Sample size matters as well. With only 100 requests, p99 is basically one sample near the edge. With tens of thousands of requests, it starts becoming much more informative. A benchmark that runs for three seconds and prints p99 to three decimal places may look precise, but the precision is mostly cosmetic.
The same warning applies when aggregating across application instances. Averaging p99 values from five replicas does not give the p99 of all combined traffic. Histograms that can be merged are much more useful for that. And then there is coordinated omission. If your load generator waits for one response before sending the next request, latency growth automatically lowers the offered load. A struggling system gets less pressure exactly when it starts struggling. The test may look stable partly because the generator quietly stopped asking for the original amount of work. That is why I care more about the point where p99 begins bending upward than the exact point where throughput finally collapses. Maximum throughput is interesting for a benchmark. Production usually wants to operate before that edge.
What I would watch in production
After this experiment, RPS would still stay on the dashboard. It is useful. The mistake is expecting it to answer questions it was never designed to answer. For a PostgreSQL-backed backend, the useful picture is wider: end-to-end request latency, preferably as a histogram; pool size; connections in use; idle connections; connection acquisition time; acquisition timeouts; query execution time; PostgreSQL active backends; wait events; and error rate. The relationships between those metrics are more useful than any one of them alone. If request p99 rises together with query execution time, then the database workload itself probably deserves attention. If request p99 rises while query time stays stable and pool acquisition gets slower, the queue is probably forming in the application. If a larger pool reduces acquire time but PostgreSQL wait events and query latency suddenly increase, the bottleneck has simply moved deeper into the database.
One metric I would add almost everywhere is the fraction of requests that had to wait for a connection at all. Average acquisition time can stay tiny while a small percentage of requests wait much longer. It has the same weakness as average HTTP latency: one number can compress a very uneven distribution into something that looks harmless. The uncomfortable part is that PostgreSQL can still look completely healthy during this stage. CPU has room. Queries are still fast. Transactions per second are flat. There may be no dramatic lock contention and no obvious storage problem. Meanwhile the application has started building a queue, and the unlucky requests near the tail are paying for it first.
That is probably the most useful conclusion from the whole experiment. Throughput is usually one of the last metrics to admit that a system is running too close to its limit. Tail latency complains earlier. So when RPS looks perfectly stable and everyone is happy with the graph, there is one question worth asking before closing the dashboard: How long did the request wait before PostgreSQL even knew it existed?