Load Balancing Algorithms: Round Robin vs Least Connections

Almost every system design answer draws a "load balancer" box in front of the servers, and interviewers then ask how it chooses. "Round robin" is the usual answer, and it is fine until requests stop costing the same. The useful answer names three policies, says when each fails, and explains what health checks and sticky sessions do to the pool.

The problem it solves

One server runs out of capacity long before your traffic does, so you run several identical servers. Clients should not have to know how many there are or which ones are alive. A load balancer gives them one address, accepts every request or connection, and forwards each one to a healthy server behind it.

That creates two jobs:

• Routing: pick a server for each request, spreading the work so that no server is overloaded while others idle.
• Health: notice when a server stops answering and stop sending it traffic before users see errors.

A balancer may see only TCP connections or whole HTTP requests; the policies below apply to both.

The intuition

Round robin cycles through the pool: server 1, 2, 3, 1, 2, 3. It needs no state beyond a counter and is perfectly even in request count. That is its blind spot. If one request in ten is a long upload, the servers that happen to receive the uploads fall behind, and round robin keeps handing them their full share anyway. Weighted round robin gives bigger machines more turns, but it is just as blind to the cost of each request.

Least connections sends each request to the server with the fewest open connections right now. It needs the balancer to count what is in flight, and in return it adapts to uneven work: a server busy with slow requests has a high count and is skipped until it catches up. When request cost varies widely, this is the policy to name.

Hashing on a key, such as the client address or a user ID, sends the same key to the same server every time. That is useful when a server keeps something per key, such as a local cache. The catch is resizing: with hash(key) % n, removing one of three servers moves about two-thirds of the keys. Consistent hashing reduces that to roughly the share of the server that left.

Health checks run beside all three. The balancer probes each server, and one that fails leaves the pool until it recovers; the survivors absorb its traffic, provided they have room.

Sticky sessions pin a user to one server so state in its memory stays reachable, at the price of an uneven pool and logged-out users when it dies. Session state usually moves to a shared store instead.

Watch it run

The client resolves one public address; three servers sit behind it, and the client never learns their names. A request arrives, and the balancer has to pick a healthy server. Round robin cycles the pool: request 1 goes to server 1, request 2 to server 2, request 3 to server 3, perfectly even on paper. Request 4 wraps back to server 1. But requests are not equal. Server 1's two are long uploads. Least connections watches who is genuinely free and sends request 5 to server 2 instead. Meanwhile the balancer keeps probing every server in the pool. Server 3 stops answering its health check, so it leaves the pool, and traffic keeps flowing across the two that are left. Sticky sessions would pin a user to one server: an uneven pool, and a crash logs them out. And the balancer is now the front door everything depends on, so it needs its own redundancy.

The code

A toy model of the routing policies, not a real proxy: a pool, a health set, and a count of open connections per server. The hash uses zlib.crc32 because Python's built-in hash of a string changes between runs:

import zlib

class ToyBalancer:
    """Toy model of a balancer's routing policies, not a real proxy."""
    def __init__(self, servers, policy):
        self.servers, self.policy = list(servers), policy
        self.healthy = set(self.servers)
        self.active = {s: 0 for s in self.servers}    # open connections per server
        self.turn = 0

    def set_health(self, server, up):                  # the result of a health probe
        if up:
            self.healthy.add(server)
        else:
            self.healthy.discard(server)

    def pick(self, key=""):
        pool = [s for s in self.servers if s in self.healthy]
        if not pool:
            return None                                # nobody left to answer
        if self.policy == "round-robin":
            s = pool[self.turn % len(pool)]            # blind rotation
            self.turn += 1
        elif self.policy == "least-conn":
            s = min(pool, key=lambda x: self.active[x])   # ties go to the first listed
        else:                                          # "hash": same key, same server
            s = pool[zlib.crc32(key.encode()) % len(pool)]
        self.active[s] += 1
        return s

    def finish(self, server):
        self.active[server] -= 1

The animation's sequence. Requests 2 and 3 are short and finish; 1 and 4 are uploads still open on server 1. Round robin keeps feeding server 1 its turn, while least connections avoids it:

for policy in ("round-robin", "least-conn"):
    lb = ToyBalancer(["s1", "s2", "s3"], policy)
    first = [lb.pick() for _ in range(4)]
    lb.finish("s2")
    lb.finish("s3")                                    # the short requests finish
    print(policy, first, [lb.pick() for _ in range(3)])
# round-robin ['s1', 's2', 's3', 's1'] ['s2', 's3', 's1']
# least-conn ['s1', 's2', 's3', 's1'] ['s2', 's3', 's2']

lb = ToyBalancer(["s1", "s2", "s3"], "least-conn")
lb.set_health("s3", False)                             # s3 fails its health check
print([lb.pick() for _ in range(4)])                   # ['s1', 's2', 's1', 's2']

Five thousand requests, one in ten an upload thirty times longer than the rest. The number printed is the peak of open connections on the busiest server:

import random

def busiest(policy, seed, n=5000):
    random.seed(seed)
    lb, open_until, peak = ToyBalancer(["s1", "s2", "s3"], policy), [], 0
    for t in range(n):
        for end, s in open_until:
            if end == t:
                lb.finish(s)
        open_until = [(end, s) for end, s in open_until if end > t]
        cost = 30 if random.random() < 0.1 else 1
        open_until.append((t + cost, lb.pick()))
        peak = max(peak, max(lb.active.values()))
    return peak

print(busiest("round-robin", 20), busiest("least-conn", 20))   # 7 4

The resizing cost of modulo hashing, with 10,000 user keys and one of three servers removed:

keys = [f"user-{i}" for i in range(10000)]
lb = ToyBalancer(["s1", "s2", "s3"], "hash")
before = {k: lb.pick(k) for k in keys}
lb.set_health("s3", False)
print(round(sum(before[k] != lb.pick(k) for k in keys) / len(keys), 2))   # 0.66

The model's incremental counters and picks against a brute force that recomputes every server's open connections from the full request log, over 1,000 random runs of arrivals, completions and health flips:

random.seed(20)
ok = True
for _ in range(1000):
    policy = random.choice(["round-robin", "least-conn", "hash"])
    lb, log = ToyBalancer(["s1", "s2", "s3", "s4"], policy), []   # log: [server, open?]
    for _ in range(40):
        event = random.random()
        if event < 0.15:
            lb.set_health(random.choice(lb.servers), random.random() < 0.6)
        elif event < 0.4 and any(o for _, o in log):
            entry = random.choice([e for e in log if e[1]])
            entry[1] = False
            lb.finish(entry[0])
        else:
            counts = {s: sum(1 for x, o in log if x == s and o) for s in lb.servers}
            pool = [s for s in lb.servers if s in lb.healthy]
            key = random.choice(["a", "b", "c"])
            s = lb.pick(key)
            if not pool:
                ok &= s is None
                continue
            ok &= s in pool                                   # never a failed server
            if policy == "least-conn":
                ok &= counts[s] == min(counts[x] for x in pool)
            if policy == "hash":
                ok &= s == pool[zlib.crc32(key.encode()) % len(pool)]
            log.append([s, True])
        ok &= all(lb.active[s] == sum(1 for x, o in log if x == s and o) for s in lb.servers)
print(ok)                                                     # True

The complexity

• Round robin: constant work per request and no state beyond a counter, but blind to request cost.
• Least connections: a scan of the pool, or a heap, per request, plus a counter the balancer must keep accurate; it adapts to uneven work.
• Hashing: constant work and stable placement per key, but a pool change remaps most keys unless you use consistent hashing.
• The balancer itself: every request passes through it, so it must be redundant or it is the single point of failure.

Where it goes wrong

• Round robin on uneven work. Equal request counts are not equal load.
• Health checks that only test the port. A server can accept connections while its database is down.
• Sticky sessions as a design. They unbalance the pool and turn a crash into logged-out users. Move session state to a shared store.
• Modulo hashing on a changing pool. One server leaving moves most keys and cold-starts every per-server cache.
• One balancer. It is the front door; run it redundantly.

When it shows up in interviews

It comes up inside nearly every system design question, from a URL shortener to a notification system, usually as "how does the load balancer choose?" or "what happens when a server dies?" Key-based routing leads into consistent hashing, and protecting servers from overload leads into rate limiting.

How to say it in an interview

"Clients see one address; the balancer forwards each request to a healthy server. Round robin is fine when requests cost about the same. When cost varies, such as uploads mixed with small reads, I would use least connections, which sends each request to the server with the fewest in flight. If a server keeps per-user state, hashing on the user ID keeps them together, and I would use consistent hashing so a pool change moves few keys. Health checks remove failing servers automatically. I would avoid sticky sessions by keeping session state in a shared store, and run the balancer itself redundantly."