Skip to content
BytePatterns

Taints and Tolerations vs Node Affinity: Repel or Attract

7 min readBytePatterns

Taints and tolerations vs node affinity: a taint on the node repels Pods, affinity on the Pod attracts it, and a dedicated GPU pool needs both rules at once.

Taints and tolerations vs node affinity in one line: a taint is set on a node and repels every Pod that lacks a matching toleration, while node affinity is set on a Pod and attracts it to nodes with the right labels. A toleration only lets a Pod past a taint; it never pulls the Pod towards that node. Everything below comes from the Kubernetes documentation pages listed at the end, as of October 2026.

Side by side:

RuleSet onWhat it does
Taintthe noderepels Pods without a matching toleration: NoSchedule, PreferNoSchedule, NoExecute
Tolerationthe Podlets the Pod past one taint; does not attract it
Node affinitythe Podattracts it to labelled nodes, as a requirement or a preference
nodeSelectorthe Podthe simple form: the node must carry every listed label

The problem it solves

Some nodes have GPUs, some are spot capacity, some are about to be drained. Two different questions follow:

  • Keep the wrong Pods off a node. A web Pod on a GPU node wastes the most expensive hardware in the cluster.
  • Get the right Pods onto a node. A training job must land where the GPU is, not wherever the scheduler finds room.

The first is a property of the node, the second a property of the Pod, so Kubernetes uses two mechanisms.

The intuition

A taint is a "keep out" sign on a node. It has a key, an optional value and an effect, as in gpu=true:NoSchedule. The effect decides how hard the sign is:

  • NoSchedule: no new Pod lands here without a matching toleration. Pods already running stay.
  • PreferNoSchedule: the scheduler tries to avoid the node, but will use it if nothing else fits.
  • NoExecute: Pods without a toleration are evicted immediately, even running ones. A toleration with tolerationSeconds lets a Pod stay for that long and then evicts it; a toleration without it lets the Pod stay for good.

A toleration is a pass for one sign. It matches when the key and effect match and either the operator is Exists or the operator is Equal and the values match. The documentation is explicit that tolerations allow scheduling but do not guarantee it.

Node affinity is a shopping list on the Pod. requiredDuringSchedulingIgnoredDuringExecution is a hard filter: no matching node, no scheduling. preferredDuringSchedulingIgnoredDuringExecution is a weighted preference: matching nodes score higher, but the Pod still runs elsewhere if it must. "IgnoredDuringExecution" means a Pod keeps running if the node's labels change later. nodeSelector is the simplest form of the same idea: the node must have each listed label. If a Pod sets both, both must hold.

Why a dedicated GPU pool needs both

The documentation's dedicated-nodes recipe uses both. The taint keeps every other Pod out. But a toleration alone lets GPU Pods use the dedicated nodes as well as any other node, so a training job can still land on a general node. Adding a label to the GPU nodes and required affinity to the GPU Pods closes that gap. Taint alone: the pool stays clean, but GPU Pods can wander. Affinity alone: GPU Pods go to the pool, but a web Pod can land there too.

Watch it run

The animation starts with the scheduler placing one new Pod that requests 500m of CPU. It filters the three nodes: node-a has only 200m free and drops out, node-b and node-c stay feasible. It scores the survivors and binds the Pod to the best one. Then it rolls four Pods from v1 to v2. Taints and affinity live in that same filter-and-score step.

Scheduling & Rolling Updates

Step 1 of 13

A new Pod requests 500m of CPU. Until the scheduler binds it to a node, it is Pending.

The same interactive animation as the lesson — step through it with the controls.

The code

A GPU Pod with both rules, illustrative. The node side is one taint and one label, shown as comments:

# kubectl taint nodes gpu-1 gpu=true:NoSchedule
# kubectl label nodes gpu-1 pool=gpu
apiVersion: v1
kind: Pod
metadata:
  name: trainer
spec:
  tolerations:
    - key: gpu
      operator: Exists
      effect: NoSchedule        # lets the Pod past the taint
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: pool
                operator: In
                values: ["gpu"]  # keeps the Pod in the pool
  containers:
    - name: trainer
      image: registry.example.com/trainer:1.0

Next, a toy model of the lesson's scheduler, not kube-scheduler: the same nodes and the same filter-then-score loop, with taints and labels added. A node is filtered out for missing CPU, an untolerated NoSchedule taint or failed required affinity. The score then adds matched preferences and avoids untolerated PreferNoSchedule taints:

nodes = [
    {"name": "node-a", "free": 200,  "labels": {"pool": "general"}, "taints": []},
    {"name": "node-b", "free": 4500, "labels": {"pool": "general"}, "taints": []},
    {"name": "node-c", "free": 800,  "labels": {"pool": "general"},
     "taints": [("spot", "true", "PreferNoSchedule")]},
    {"name": "gpu-1",  "free": 4000, "labels": {"pool": "gpu"},
     "taints": [("gpu", "true", "NoSchedule")]},
]

def tolerates(pod, taint):
    """A toleration matches on key and effect, plus value for Equal."""
    key, value, effect = taint
    for t in pod.get("tolerations", []):
        if t["key"] == key and t.get("effect") in (None, effect):
            if t.get("operator") == "Exists" or t.get("value") == value:
                return True
    return False

def matches(labels, rule):
    key, op, values = rule
    return (labels.get(key) in values) if op == "In" else (labels.get(key) not in values)

def schedule(pod, nodes):
    """Toy model of the scheduler: filter, then score, then bind."""
    feasible, why = [], {}
    for n in nodes:
        hard = [t for t in n["taints"] if t[2] == "NoSchedule" and not tolerates(pod, t)]
        if n["free"] < pod["cpu"]:
            why[n["name"]] = "cpu"
        elif hard:
            why[n["name"]] = "taint " + hard[0][0]
        elif not all(matches(n["labels"], r) for r in pod.get("required", [])):
            why[n["name"]] = "affinity"
        else:
            feasible.append(n)
    if not feasible:
        return "Pending (" + ", ".join(f"{k}: {v}" for k, v in why.items()) + ")"
    def score(n):
        soft = sum(1 for t in n["taints"]
                   if t[2] == "PreferNoSchedule" and not tolerates(pod, t))
        pref = sum(w for w, r in pod.get("preferred", []) if matches(n["labels"], r))
        return (pref - 100 * soft, n["free"])   # toy weights; free CPU breaks ties
    return max(feasible, key=score)["name"]

GPU_TOL = {"key": "gpu", "operator": "Exists", "effect": "NoSchedule"}
GPU_POOL = ("pool", "In", ["gpu"])
pods = {
    "web":           {"cpu": 500},
    "trainer":       {"cpu": 2000, "tolerations": [GPU_TOL], "required": [GPU_POOL]},
    "tol-only": {"cpu": 500,  "tolerations": [GPU_TOL]},
    "affinity-only": {"cpu": 500,  "required": [GPU_POOL]},
    "batch":         {"cpu": 600,  "preferred": [(50, ("pool", "In", ["general"]))]},
}
for name, pod in pods.items():
    print(f"{name:14} -> {schedule(pod, nodes)}")
# web            -> node-b
# trainer        -> gpu-1
# tol-only       -> node-b
# affinity-only  -> Pending (node-a: cpu, node-b: affinity, node-c: affinity, gpu-1: taint gpu)
# batch          -> node-b

The toleration-only Pod went to node-b: it was allowed into the pool, but nothing pulled it there. The affinity-only Pod is Pending, because the one node it wants repels it. When node-b is full, PreferNoSchedule gives way, and a NoExecute taint shows its three outcomes:

nodes[1]["free"] = 0                              # node-b fills up
print(schedule(pods["batch"], nodes))             # node-c   PreferNoSchedule is only a preference
nodes[1]["free"] = 4500

def evictions(running, taint):
    """Toy model of a NoExecute taint added to a node with running Pods:
    no toleration = evicted now; tolerationSeconds = evicted later; else stays."""
    key, value, effect = taint
    out = {}
    for name, pod in running.items():
        tol = next((t for t in pod.get("tolerations", [])
                    if t["key"] == key and t.get("effect") in (None, effect)), None)
        if tol is None:
            out[name] = "evicted at t=0s"
        elif "tolerationSeconds" in tol:
            out[name] = f"evicted at t={tol['tolerationSeconds']}s"
        else:
            out[name] = "stays"
    return out

running = {
    "api":    {},
    "cache":  {"tolerations": [{"key": "maintenance", "effect": "NoExecute", "tolerationSeconds": 300}]},
    "logger": {"tolerations": [{"key": "maintenance", "effect": "NoExecute"}]},
}
print(evictions(running, ("maintenance", "true", "NoExecute")))
# {'api': 'evicted at t=0s', 'cache': 'evicted at t=300s', 'logger': 'stays'}

The dedicated-pool claim, checked on 3,000 random Pods with and without each rule while node-b's free CPU changes. No Pod without the toleration lands on the GPU node, no Pod with both rules runs outside the pool, and toleration-only Pods regularly do:

import random

random.seed(7)
gpu_nodes = {n["name"] for n in nodes if n["labels"]["pool"] == "gpu"}
leaks = strays = tol_only = tol_only_outside = 0
for _ in range(3000):
    nodes[1]["free"] = random.choice([0, 1500, 4500])   # node-b busy or idle
    pod = {"cpu": random.choice([100, 500, 1000, 3000])}
    if random.random() < 0.5:
        pod["tolerations"] = [GPU_TOL]
    if random.random() < 0.5:
        pod["required"] = [GPU_POOL]
    where = schedule(pod, nodes)
    placed = not where.startswith("Pending")
    if where in gpu_nodes and "tolerations" not in pod:
        leaks += 1                     # a pod without the toleration on a GPU node
    if "tolerations" in pod and "required" in pod and placed and where not in gpu_nodes:
        strays += 1                    # a pod with both rules outside the GPU pool
    if "tolerations" in pod and "required" not in pod and placed:
        tol_only += 1
        tol_only_outside += where not in gpu_nodes
print(leaks, strays)                   # 0 0
print(tol_only_outside, "of", tol_only, "toleration-only pods ran outside the pool")
# 236 of 766 toleration-only pods ran outside the pool

The complexity

The costs are operational:

  • A taint is one change on the node, but every Pod that should pass needs a toleration. For extended resources such as GPUs, the ExtendedResourceToleration admission controller adds it to Pods that request the resource.
  • Required affinity shrinks the feasible set. With too few matching nodes, Pods stay Pending until capacity appears.
  • Preferred affinity and PreferNoSchedule only change scores: they never cause Pending and never guarantee anything.

Where it goes wrong

  • Using a toleration as an attraction. It is permission, not direction. Add affinity or a nodeSelector.
  • Tainting without a label. Affinity needs something to match, and a taint is not a label.
  • Expecting NoSchedule to clear a node. Running Pods stay. Only NoExecute evicts.
  • Forgetting the built-in taints. The node controller taints unhealthy nodes, for example with node.kubernetes.io/not-ready and node.kubernetes.io/unreachable. Kubernetes adds a toleration for both with tolerationSeconds=300 unless one is set, so Pods stay bound for five minutes after the problem is detected.
  • Setting nodeName by hand. It bypasses the scheduler and ignores NoSchedule taints.

When it shows up in interviews

It comes up as "how do you reserve nodes for GPU jobs?", "why is my Pod Pending?" or "how do you drain a node safely?". Pending is usually a filter problem, and the scheduling event says which nodes failed and why. Sizing is the other half of the filter, covered in requests vs limits. How the replacement Pods of a rollout get placed is in rolling updates, and which object owns those Pods is in Pod vs ReplicaSet vs Deployment.

How to say it in an interview

"They point in opposite directions. A taint is on the node and repels Pods that lack a matching toleration: NoSchedule blocks new Pods, PreferNoSchedule is soft, and NoExecute also evicts running ones, after tolerationSeconds if set. Node affinity is on the Pod and attracts it to labelled nodes, either required or preferred, and nodeSelector is its simple form. A toleration is only permission, so for a dedicated GPU pool I'd use both: taint the GPU nodes to keep everything else out, and give GPU Pods the toleration plus required affinity to the pool's label so they stay in."

The scheduler's filter and score steps, and the rollout that follows, are in scheduling and rolling updates.

Frequently asked questions

What is the difference between taints and node affinity?

Direction. A taint is set on a node and repels every Pod without a matching toleration. Node affinity is set on a Pod and attracts it to nodes whose labels match. A taint protects a node from the wrong Pods; affinity steers a Pod to the right nodes. Neither replaces the other.

Does a toleration force a Pod onto the tainted node?

No. A toleration only lets the Pod past that taint; the scheduler can still place it on any other node that fits. To keep the Pod on the tainted nodes, label them and add required node affinity or a nodeSelector for that label.

When should I use NoExecute instead of NoSchedule?

Use NoExecute when Pods already running on the node must leave too, for example before maintenance. NoSchedule only blocks new Pods. A Pod that tolerates NoExecute with tolerationSeconds stays that long and is then evicted; without it, the Pod stays.

Sources