Size an application thread pool from how long requests spend working versus waiting.
Every thread that touches the database holds a connection while it waits, so the pool here sets the floor for your connection pool.
Why the right thread pool is smaller than you think, until suddenly it isn't
A thread pool has one job: keep the CPU busy without letting work pile up inside the process. The size that achieves this depends almost entirely on one ratio — how long a request spends waiting on something else versus how long it spends actually computing. This tool applies the standard formula for that ratio, checks it against Little's Law, and tells you when the answer means you need a different concurrency model rather than a bigger number.
How it works
- Applies the classic pool-sizing formula: cores × target utilisation × (1 + wait ÷ service).
- Uses Little's Law to work out the concurrency your target throughput actually implies.
- Reports the throughput the recommended pool can sustain, so you can see whether it clears your target.
- Warns when the arithmetic has run past what a thread-per-request model can sensibly do.
threads = cores x utilisation x (1 + wait_time / service_time) Little's Law concurrency = arrival_rate x latency throughput ceiling max_rps = threads / (service_time + wait_time)
Worked example
A service on 8 cores, targeting 80% CPU, where a request spends 20 ms computing and 80 ms waiting on the database, at a peak of 300 requests per second.
- wait ÷ service = 80 ÷ 20 = 4
- threads = 8 × 0.8 × (1 + 4) = 32
- latency = 20 + 80 = 100 ms
- Little's Law: 300 requests/s × 0.1 s = 30 concurrent requests needed
- 32 threads ≥ 30, so the pool clears the target
- ceiling: 32 ÷ 100 ms = 320 requests per second
Thirty-two threads on eight cores. The pool is four times the core count not because threads are free, but because each thread is idle 80% of the time — it is holding a place in a queue at the database, not using a CPU. Change the wait to zero and the same formula returns 6, which is the right answer for work that never blocks.
Reading the result
- The formula is only as good as the two times you feed it. Service time is CPU time, not wall-clock time, and the two are easy to confuse — wall-clock includes the waiting, which is the other variable. A profiler separates them; a stopwatch does not.
- Every thread that queries the database holds a connection while it waits. That makes the thread pool a floor for the connection pool, and it is the usual reason a database connection limit is hit: nobody sized the two together.
- Past a few hundred threads the model itself is the problem. Each platform thread costs around a megabyte of stack and a scheduler slot, so a pool of 1,300 spends more time context-switching than working. At that ratio the answer is non-blocking I/O or virtual threads, which is why the tool says so rather than printing the number and leaving you to it.
- A queue in front of the pool is not extra capacity. If arrivals exceed what the pool can retire, the queue grows without bound and latency grows with it — the requests still fail, only later and after consuming memory. Bound the queue and shed load rather than pretending depth is throughput.
- Utilisation targets above about 80% stop being safe. Queueing theory is unkind near saturation: as utilisation approaches 100%, waiting time rises without limit, so the last 20% of a core costs far more latency than it returns in throughput.
Common questions
- Where does the (1 + wait/service) formula come from?
- It is the standard result for keeping N cores busy when each task only occupies a core for part of its lifetime, popularised by Brian Goetz in Java Concurrency in Practice. With no waiting it reduces to cores × utilisation, which is the correct answer for CPU-bound work; the ratio simply accounts for the threads parked on I/O at any moment.
- Should the thread pool match the connection pool?
- The connection pool should be at least as large as the number of threads that can be inside a database call at once, but no larger than the database's own limit will bear across your whole fleet. If the two conflict, the resolution is fewer threads or more application instances — not a bigger connection pool, which just moves the contention into the database.
- Do virtual threads make this obsolete?
- They change which number matters. Virtual threads are cheap enough that you can have one per request without sizing a pool, so the wait ratio stops constraining you. What remains is the downstream limit: the database connection pool, the rate limit on an upstream API, the CPU itself. You still have to size those, and this calculation still tells you where they bind.
- How do I measure service time and wait time?
- Take service time from CPU profiling or from the difference between wall-clock and the sum of your instrumented external calls. Take wait time from the spans your tracing already emits for database and HTTP calls. If you have neither, start from an educated guess and check the prediction: if the pool sustains far less throughput than this formula says, your wait estimate was low.