Sizing a Thread Pool

The Idea

A pool size is a bet about where the bottleneck is. CPU-bound work saturates at roughly the core count; extra threads only add context switches. IO-bound work spends its time blocked, so many more threads fit. Either way the longest single task is a floor nothing can go below.

Real-World Example

A pizzeria with one oven. Hiring a fourth chef speeds up prep, which was the queue. Hiring a tenth does not, because the oven never got faster and the kitchen now has a traffic problem.

The Code

def makespan(tasks, workers):
    free = [0.0] * workers                 # when each worker is next idle
    for cost in sorted(tasks, reverse=True):
        first = min(range(workers), key=lambda w: free[w])
        free[first] += cost
    return max(free)

jobs = [4, 4, 4, 4, 1, 1, 1, 1]            # 20 units of work, longest is 4
for n in (1, 2, 4, 8, 16):
    print(n, makespan(jobs, n))
# 1 20.0 / 2 10.0 / 4 5.0 / 8 4.0 / 16 4.0 -- the longest task is the floor

def makespan(tasks, workers):
  free = [0.0] * workers
  for cost in sorted(tasks, reverse=True):
      free[min(range(workers), key=lambda w: free[w])] += cost
  return max(free)

print(makespan([3, 3, 3], ___))   # 3.0