How Many Threads? Sizing a Thread Pool for CPU and IO Work

"How many threads should this pool have?" has no universal number, which is exactly why it is asked. The answer depends on what the threads spend their time doing, and a good candidate reasons from that instead of quoting 10, 100 or "twice the cores". There are three limits to reason about: the cores, the waiting, and the work itself.

The problem it solves

A thread pool caps how much runs at once. Set the cap too low and requests queue while cores or network sit idle. Set it too high and the extra threads cost memory for their stacks, time for context switches, and they push more concurrent load onto whatever is downstream, often a database that was already the bottleneck. The size is a bet about where the bottleneck is, so the first question is always: what does a task spend its time on?

• CPU-bound tasks compute the whole time: parsing, compressing, resizing images.
• IO-bound tasks mostly wait: an HTTP call, a disk read, a database query.
• Most real tasks are a mix, and the mix decides the size.

The intuition

CPU-bound: a core runs one thread at a time. With 8 cores, the ninth busy thread can only take turns, so throughput stops rising at about one thread per core. In CPython the global interpreter lock means threads do not run Python bytecode in parallel on the standard build (as of September 2026, free-threaded builds exist but are not the default), so CPU-bound Python work goes to a process pool, sized the same way; see threads vs processes.

IO-bound: a waiting thread wants no core at all. If each task computes for C ms and waits for W ms, a core is busy only C / (C + W) of the time one thread holds it, so it takes 1 + W / C threads to keep that core fed. The common rule of thumb is therefore threads ≈ cores × (1 + W / C). With 4 cores and tasks that wait three times as long as they compute, that is 16.

Little's law gives the same answer from the outside: the average number of tasks in progress equals the arrival rate times the time each one spends inside. At 400 requests per second, each holding a thread for 125 ms, 50 threads are busy on average; a pool much smaller than that builds a queue that grows until requests time out.

The floor: no pool finishes a batch faster than its longest single task, and no pool of n workers beats total work divided by n. Once the pool is bigger than the number of tasks available, more threads change nothing.

Watch it run

The animation deals the lesson's eight tasks, four long and four short, onto one, two, four and then eight workers. Twenty units of work, and the longest single task is four. One worker runs them end to end: twenty units, and nothing else is happening. A second worker halves it to ten, the part everyone expects. Four workers take the four long tasks: five units, and every lane is doing something. Add four more and the finish line barely moves, from five to four, because the longest task is a floor nothing gets under. Past that, extra threads are memory and context switches bought for nothing. That is the CPU-bound shape, where the cores are the wall, so about one thread per core. IO-bound work is the opposite: most of each lane is waiting, and a waiting thread wants no core, so many more threads fit. The closing frame: the size is a bet about the bottleneck, so measure queue length and latency instead of guessing the constant.

The code

The lesson's longest-task-first schedule with the two floors next to it, then a toy model of threads that alternate between computing (which needs one of 4 cores) and waiting (which needs none), with the rule of thumb and Little's law:

def makespan(tasks, workers):
    """Longest task first, each onto the worker that frees up soonest."""
    free = [0] * workers
    for cost in sorted(tasks, reverse=True):
        i = free.index(min(free))
        free[i] += cost
    return max(free)

jobs = [4, 4, 4, 4, 1, 1, 1, 1]                     # the lesson's eight tasks
for n in (1, 2, 4, 8, 16):
    floor = max(max(jobs), -(-sum(jobs) // n))       # no schedule beats either limit
    print(n, makespan(jobs, n), floor)
# 1 20 20
# 2 10 10
# 4 5 5
# 8 4 4
# 16 4 4

def throughput(threads, cores, compute, wait, horizon=2_000):
    """Toy model: each thread loops compute (needs a core) then wait (needs none).
    Unit time steps; returns tasks finished per time unit."""
    state = [("ready", 0)] * threads                 # (phase, time left in it)
    done = 0
    for _ in range(horizon):
        running = sum(1 for p, _ in state if p == "cpu")
        for i, (p, _) in enumerate(state):           # hand free cores to ready threads
            if p == "ready" and running < cores:
                state[i], running = ("cpu", compute), running + 1
        nxt = []
        for p, left in state:
            if p == "ready":
                nxt.append((p, 0))
            elif left > 1:
                nxt.append((p, left - 1))
            elif p == "cpu":
                nxt.append(("io", wait) if wait else ("ready", 0))
                done += wait == 0
            else:
                nxt.append(("ready", 0))
                done += 1
        state = nxt
    return done / horizon

cores, compute, wait = 4, 2, 6                      # 25% of each task needs a core
print("rule of thumb:", cores * (1 + wait // compute))   # rule of thumb: 16
for t in (1, 4, 8, 16, 32, 64):
    print(t, round(throughput(t, cores, compute, wait), 2))
# 1 0.12
# 4 0.5
# 8 1.0
# 16 1.99
# 32 1.99
# 64 1.99

# Little's law for the IO side: busy threads = arrival rate x time each one is held.
requests_per_s, seconds_held = 400, 0.125
print(requests_per_s * seconds_held)                # 50.0 threads busy on average

Throughput doubles with every doubling up to 16 threads, then stops: four cores that each need two time units per task finish at most two tasks per unit (the 1.99 is the warm-up). Checked on 400 seeded random batches against a brute force that tries every assignment of tasks to workers, and on 40 random simulations against both walls:

import itertools
import random
from fractions import Fraction

def best_makespan(tasks, workers):
    """Brute force: try every assignment of tasks to workers."""
    best = sum(tasks)
    for owners in itertools.product(range(workers), repeat=len(tasks)):
        load = [0] * workers
        for cost, w in zip(tasks, owners):
            load[w] += cost
        best = min(best, max(load))
    return best

rng = random.Random(30)
ok = True
for _ in range(400):
    m = rng.randint(1, 4)
    tasks = [rng.randint(1, 9) for _ in range(rng.randint(1, 7))]
    lpt, opt = makespan(tasks, m), best_makespan(tasks, m)
    floor = max(max(tasks), -(-sum(tasks) // m))
    ok &= floor <= opt <= lpt <= Fraction(4, 3) * opt  # longest-first is never far off

for _ in range(40):                                 # the simulation never beats its two walls
    cores, compute, wait = rng.randint(1, 4), rng.randint(1, 4), rng.randint(0, 8)
    t = rng.randint(1, 24)
    got = throughput(t, cores, compute, wait, horizon=400)
    wall = min(t / (compute + wait), cores / compute)
    ok &= got <= wall + 1e-9
    if t >= cores * (compute + wait) / compute:     # at or past the rule of thumb
        ok &= got >= 0.9 * cores / compute          # ... the cores are (nearly) saturated
print(ok)                                           # True

The 4/3 check is a known bound for longest-task-first scheduling: it is never more than a third worse than the best schedule.

The complexity

• Throughput is min(threads / (C + W), cores / C) tasks per unit time: linear in threads until the cores saturate, flat after.
• Batch time is at least max(longest task, total work / threads).
• Cost of oversizing: one stack per thread (commonly megabytes of reserved address space), more context switches, and more concurrent load on shared resources.

Where it goes wrong

• A pool bigger than the downstream. Sixty threads against a database pool of twenty means forty threads waiting on a lock you cannot see; size to the scarcest resource.
• One pool for everything. CPU work and slow IO in the same pool make each other's latency unpredictable; split them.
• Unbounded queues in front of the pool. The pool caps threads, not memory; bound the queue so overload pushes back.
• Guessing W / C. Measure it: the fraction of time a task holds a core under real load, not in a benchmark with a fast mock.

When it shows up in interviews

As "how many threads would you use?" in concurrency and backend rounds, as a follow-up to producer-consumer and semaphores, and inside system design whenever a service calls a slower dependency. The same reasoning sizes connection pools and worker counts in async code.

How to say it in an interview

"It depends on what the tasks wait on. For CPU-bound work I'd start at the number of cores, since more threads only take turns; in Python that means processes. For IO-bound work a waiting thread uses no core, so I'd start at cores times one plus wait over compute, or apply Little's law, arrival rate times time held, and add headroom. Then I'd cap it by the scarcest downstream resource, bound the queue in front, and tune from measured queue length and latency. And no pool size beats the longest single task."