Kubernetes OOMKilled (Exit Code 137): Why, and How to Fix It

October 24, 20269 min readBytePatterns

Kubernetes OOMKilled with exit code 137: where the reason is recorded, how to tell a limit set too low from a memory leak, what fixes each and what does not.

Kubernetes OOMKilled (exit code 137) in one line: the container used more memory than its limit, the kernel killed it with SIGKILL, and the kubelet restarted it. The exit code is 128 plus signal 9. The reason is written into the container's last state, so kubectl describe pod is the first stop, and the fix depends on one question: does the container's memory come back down on its own, or does it only ever climb? A bounded workload with a low limit needs a bigger limit; a steady climb is a leak that no limit will cure. What a request and a limit are, and why CPU is throttled while memory is killed, is the concept article: requests vs limits in Kubernetes. This one is the incident. Facts come from the Kubernetes documentation listed at the end, as of October 2026.

What you see	What happened	What fixes it
`OOMKilled`, exit code 137, usage fell between kills	a bounded peak above the memory limit	raise the limit above the peak, or shrink the peak
`OOMKilled`, usage only ever climbed	memory that is never freed	fix the leak; a bigger limit only delays the kill
Pod `Failed`, reason `Evicted`	the node ran short, not the container	honest requests, fewer Pods per node

The problem it solves

A Pod is restarting. kubectl get pods shows RESTARTS climbing, maybe CrashLoopBackOff, and the logs end mid-sentence with no error, because a SIGKILL gives the process no chance to write one. Three things have to be settled in order: that it really was a memory kill and not a probe or a crash; whether the limit or the program is at fault; and what to change. Each has a specific place to look, and most of the time wasted on this incident is spent looking in the wrong one.

The intuition

Where the verdict is recorded. The documentation says a container that allocates more than its memory limit becomes a candidate for termination, and is terminated if it keeps going. The kubelet records that in the container status. kubectl describe pod shows it as Last State: Terminated, Reason: OOMKilled, Exit Code: 137. The number is the convention for death by signal: 128 plus the signal number, and SIGKILL is 9. A 137 without the OOMKilled reason means something else sent SIGKILL, for example a failed liveness probe after the grace period ran out; which probe restarts a Pod is in liveness vs readiness vs startup probe.

Where to look, in order.

kubectl get pod for RESTARTS and the current status. CrashLoopBackOff means the kubelet is delaying each restart; the documentation describes the back-off as starting at ten seconds, doubling, and capped at five minutes.
kubectl describe pod for the last state and the events list. The container status is the authoritative record; events add the back-off messages and probe failures around it.
kubectl get events --field-selector involvedObject.name=<pod> when the describe output is long or the Pod has already been replaced.
kubectl top pod --containers for live usage, which needs the metrics server installed. One reading is not enough: you want the shape over time, from a metrics system or from repeated readings.

The one diagnostic question. The documentation is clear that a memory limit is enforced reactively by the kernel: there is no slowing down, only a kill at the line. So the kill itself tells you nothing about the cause. The history does. If usage climbed, fell, climbed again, and the kill happened at a peak, the workload is bounded and the limit is simply below its peak. If usage rose in a straight line from every restart to the next kill, with the time between kills roughly constant, memory is being retained and never freed: a leak, an unbounded cache, or a queue that grows faster than it drains.

The decision tree.

Bounded peak: raise the limit above the peak with headroom, or reduce the peak by processing in smaller batches or streaming instead of loading everything. Keep the request close to the new limit, so the scheduler does not pack the node with Pods that cannot all peak at once.
Steady climb: fix the program. A bigger limit lengthens the interval between kills and nothing more.
Runtime that sizes itself from the host: a managed runtime that picks its heap size from the node's memory, not the container's limit, will overshoot a small limit every time. Set the heap explicitly, or use the runtime's container-aware setting, and leave room for the memory outside the heap.
A single step too large for any sane limit: split the work across Pods instead of one giant container.

What does not help. More replicas: the limit is per container, and ten containers that each need 600Mi under a 512Mi limit are ten kills. More CPU: it is a different resource with a different enforcement. Removing the limit: the container then competes for the node's memory, and the next thing to fail is the node, with the kubelet evicting Pods by its own ranking.

Not every memory death is this one. When the node itself runs short, the kubelet evicts whole Pods. Those show as Failed with an Evicted reason and are replaced by their controller, often on another node, while an OOM-killed container is restarted in place.

Watch it run

The lesson animation shows the request doing its job on the other side of the same manifest. One bar per Pod is CPU use as a percentage of the 250m request, against a 60% target. Traffic pushes the bars to 90%, the HPA computes six replicas, and the two new Pods wait Pending until node autoscaling adds a node. Nothing in that chain looks at the memory limit: a memory kill is a per-container event the scheduler and the autoscaler never see, which is why adding replicas does not fix it.

Requests, Limits & HPA

Step 1 of 12

replicas4average40%target60%

pend4

pend5

One bar per Pod: its CPU use as a percentage of its request, 250m. The HPA's target is 60%.

The same interactive animation as the lesson — step through it with the controls.

The code

What the incident looks like from the command line, with the lines that matter:

# illustrative: kubectl get pod, then kubectl describe pod report-7c9d4-x2k8m
NAME                  READY   STATUS             RESTARTS      AGE
report-7c9d4-x2k8m    0/1     CrashLoopBackOff   6 (41s ago)   62m

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 24 Oct 2026 09:37:02 +0000
      Finished:     Fri, 24 Oct 2026 09:44:11 +0000
    Restart Count:  6
    Limits:
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  256Mi
Events:
  Warning  BackOff  41s (x12 over 3m)  kubelet  Back-off restarting failed container report in pod report-7c9d4-x2k8m

A toy model of one container under a limit, not the kernel or the kubelet. Each minute the workload sets the usage; past the limit the container is killed and restarts from its baseline. Two workloads hit the same 512Mi limit: a report job that grows and then frees everything every ten minutes, and a leak that frees nothing:

def run_container(workload, limit, minutes=60):
    """Toy model of one container under a memory limit, not the kernel or the kubelet.
    Each minute the workload sets the usage; past the limit the container is
    killed (exit code 137) and restarts from its baseline."""
    usage, restarts, samples = 0, 0, []
    for minute in range(minutes):
        usage = workload(minute, usage)
        if usage > limit:
            restarts += 1
            usage = workload(0, 0)                 # the restart: back to the baseline
            samples.append((minute, "OOMKilled", 128 + 9))
        else:
            samples.append((minute, usage, None))
    return restarts, samples

def batch(minute, usage):
    """A report job: grows 60 MiB a minute, then frees everything every 10 minutes."""
    return 120 if minute % 10 == 0 or usage == 0 else usage + 60

def leak(minute, usage):
    """A leak: 30 MiB a minute that is never freed."""
    return 120 if usage == 0 else usage + 30

for name, workload in [("batch", batch), ("leak", leak)]:
    restarts, samples = run_container(workload, limit=512)
    kills = [m for m, state, _ in samples if state == "OOMKilled"]
    print(f"{name:6} limit 512Mi: {restarts} restarts, exit code 137, killed at minutes {kills[:4]} ...")
# batch  limit 512Mi: 6 restarts, exit code 137, killed at minutes [7, 17, 27, 37] ...
# leak   limit 512Mi: 4 restarts, exit code 137, killed at minutes [14, 28, 42, 56] ...

Both containers die with the same reason and the same exit code, at regular intervals. The kill alone does not separate them.

The history does. The diagnosis reads the samples the way you would read a usage graph: did memory ever fall while the container kept running? Then the effect of the two possible fixes, raising the limit twice over:

def diagnose(samples):
    """Toy model of reading `kubectl top pod` over time: did usage ever fall
    while the container kept running? Then the workload is bounded and the
    limit sits below its peak. If it only ever climbs until the kill,
    something is never freed."""
    fell = any(before[1] != "OOMKilled" and after[1] != "OOMKilled" and after[1] < before[1]
               for before, after in zip(samples, samples[1:]))
    return "bounded peak: raise the limit" if fell else "monotonic climb: fix the leak"

for name, workload in [("batch", batch), ("leak", leak)]:
    _, samples = run_container(workload, limit=512)
    print(f"{name:6} -> {diagnose(samples)}")
    for limit in (512, 768, 1024):
        restarts, _ = run_container(workload, limit=limit)
        print(f"         limit {limit:4}Mi: {restarts} restarts in an hour")
# batch  -> bounded peak: raise the limit
#          limit  512Mi: 6 restarts in an hour
#          limit  768Mi: 0 restarts in an hour
#          limit 1024Mi: 0 restarts in an hour
# leak   -> monotonic climb: fix the leak
#          limit  512Mi: 4 restarts in an hour
#          limit  768Mi: 2 restarts in an hour
#          limit 1024Mi: 1 restarts in an hour

The report job peaks at 660 MiB, so a 768Mi limit ends the kills for good. The leak is killed at every limit; doubling it only halves the restart rate. That is the whole diagnosis: a fix that works is permanent, a fix that only slows the problem down was the wrong fix.

The complexity

The costs here are in time and headroom rather than in Big-O:

Time to the verdict. The container status gives the reason in one command; the shape of usage over time needs minutes to hours of samples, which is why a metrics system pays for itself in the first incident.
Headroom against density. A limit well above the peak protects one Pod and lowers how many Pods fit a node, since the request should follow the limit up. A limit at the peak is cheaper and kills on the first unusual day.
Restart cost. Every kill is a cold start plus the back-off, up to five minutes of downtime per attempt while in CrashLoopBackOff. A Pod that restarts every fourteen minutes under a leak is unavailable far more than it looks.

Where it goes wrong

Raising the limit in a loop. 512Mi to 1Gi to 2Gi, each one lasting a little longer. That history is the leak diagnosis; read it instead of repeating it.
Trusting the logs. A SIGKILL leaves no final log line. The absence of an error is itself the clue; the reason is in the container status, not in stdout.
Reading a probe kill as a memory kill. Both end in exit code 137. Only one has Reason: OOMKilled; the other has a probe failure in the events.
Confusing eviction with an OOM kill. An evicted Pod is Failed and replaced; an OOM-killed container restarts in place. The first is a node problem, the second a container problem.
Scaling out to fix it. Replicas do not change a per-container limit.

When it shows up in interviews

As a debugging scenario: "a Pod keeps restarting with exit code 137, walk me through it." The expected path is describe for the reason, the distinction between a low limit and a leak from the usage shape, the fixes for each, and the difference from node-pressure eviction. A good follow-up is what you would set the request to afterwards, which leads back to the requests, limits and HPA lesson and to HPA vs VPA vs Cluster Autoscaler, where the VPA's recommendations come from exactly this usage history.

How to say it in an interview

"Exit code 137 is 128 plus SIGKILL. I confirm it with kubectl describe pod: the last state should say OOMKilled, otherwise the kill came from somewhere else, like a liveness probe. Then I look at memory over time, from metrics or repeated kubectl top. If usage rose and fell and the kill happened at a peak, the limit is below a bounded peak and I raise it, with the request following. If usage climbed in a straight line between every restart and the kill interval is constant, it is a leak, and a bigger limit only delays the next kill, so I fix the code or the runtime's heap setting. More replicas or more CPU do not help, because the limit is per container. If instead the Pod was Evicted, the node ran short, and that is a requests problem."

Frequently asked questions

What does exit code 137 mean in Kubernetes?

The container was killed by signal 9, SIGKILL: 128 plus 9. When the kernel killed it for exceeding its memory limit, the container's last state also shows Reason: OOMKilled. A 137 without that reason came from another sender, such as the kubelet after a failed liveness probe.

How do I fix OOMKilled in Kubernetes?

Look at memory over time. If usage rose and fell and the kill happened at a peak, raise the memory limit above that peak and keep the request close to it. If usage only climbed between restarts, fix the leak or the runtime's heap setting; a bigger limit only delays the next kill. Replicas and CPU do not help.

Is OOMKilled the same as a Pod being evicted?

No. OOMKilled is a container exceeding its own memory limit; the kubelet restarts that container in place, and the Pod stays. Eviction happens when the node runs short of memory: the kubelet marks the whole Pod Failed with reason Evicted, and its controller replaces it, often on another node.

Sources

Resource Management for Pods and Containers — Kubernetes Documentation
Assign Memory Resources to Containers and Pods — Kubernetes Documentation
Pod Lifecycle — Kubernetes Documentation
Debug Running Pods — Kubernetes Documentation
Resource metrics pipeline — Kubernetes Documentation
Node-pressure Eviction — Kubernetes Documentation