Fix Docker Exit Code 137 (OOMKilled): Why It Happens and How to Stop It

What exit code 137 actually means

Exit code 137 is 128 + 9. The 128 + part is the shell convention for "terminated by a signal," and 9 is SIGKILL. So 137 means your process was hard-killed - no chance to clean up, no graceful shutdown.

Two things commonly send that SIGKILL:

• The kernel OOM killer. Your container hit its memory cgroup limit, or the whole host ran out of RAM, and the kernel picked a process to kill to stay alive.
• A docker stop that timed out. Docker sends SIGTERM, waits (10s by default), and if the process is still alive it escalates to SIGKILL. That also produces 137.

Both look identical in docker ps -a. The rest of this is about the first case - OOMKilled - because that's the one that quietly recurs.

Confirm it was OOM, not something else

Before changing anything, confirm the cause. Docker records whether the OOM killer was involved:

docker inspect my-service --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

If that prints true 137, you're done guessing - it was OOM. If it prints false 137, the SIGKILL came from somewhere else (most often a docker stop timeout).

For the fuller picture:

docker inspect my-service --format '{{json .State}}' | python3 -m json.tool

You can also confirm from the kernel side. The OOM killer logs every kill:

sudo dmesg -T | grep -i -E 'out of memory|oom-kill|killed process'
# or, on a systemd host:
journalctl -k | grep -i -E 'out of memory|oom-kill'

You're looking for a line like Out of memory: Killed process 12345 (java).

One distinction that changes your fix: did you hit the container's own --memory limit, or did the whole host run out of RAM? If OOMKilled is true but the host has plenty of free memory, the container hit its own cgroup limit. If the host itself was starved, the kernel may kill the biggest process regardless of which container it's in - sometimes an innocent bystander. dmesg shows the cgroup and total-vm in the kill line, which tells you which case you're in.

Why it happens

A few root causes cover most of what I see:

• No --memory limit at all. The container can grow until the host is exhausted. This is the one that takes down neighbours.
• A limit that's just too low for the real working set. The app was always going to need ~400 MB and you capped it at 256 MB.
• A real leak. Memory climbs steadily under load and never comes back down. A limit only changes when it dies, not whether.
• A runtime that ignores the cgroup limit. This is the classic. An old JVM sees the host's total RAM, sizes its heap for that, and blows past the container limit. Node has a similar story - its old-space heap defaults to roughly 1.5–2 GB regardless of the container limit unless you tell it otherwise with --max-old-space-size.
• A batch job that spikes. Steady-state memory is fine, but one large request or a big file load briefly doubles it and trips the limit.

The runtime-unaware case is worth dwelling on because it surprises people: the container limit and the runtime's idea of "how much memory exists" are two different numbers, and if the runtime's number is bigger, it will happily allocate its way into an OOM kill.

How I'd diagnose it (in order)

Cheapest, least invasive first. I don't reach for a profiler until the simple checks rule things out.

• Watch live usage against the limit. docker stats shows current memory and the limit side by side:

  docker stats --no-stream my-service
  

  If MEM USAGE / LIMIT reads 254MiB / 256MiB right before it dies, you're pegged at the limit - that's your answer.

• Reproduce and watch it climb. Leave docker stats streaming (drop --no-stream) while you drive load. Steady climb that never recedes points at a leak; a sharp spike on one operation points at a batch/request problem.

• Check the configured limit. Confirm what the container was actually given:

  docker inspect my-service --format 'mem={{.HostConfig.Memory}} memswap={{.HostConfig.MemorySwap}}'
  

  0 means no limit. Otherwise it's bytes.

• Check the app-level heap settings. Look at how the runtime was told to size itself - JVM flags, NODE_OPTIONS, whatever applies. Mismatch between this and the container limit is a common culprit.

• Only now, a profiler. If usage is legitimately high and you need to know what's holding memory, attach the language's heap profiler. This is the expensive step, so I earn my way to it.

A worked example

Say I've got a JVM service running with a tight limit:

docker run -d --name my-service --memory=256m my-registry/my-service:1.4.2

It OOMs under load. docker inspect confirms it:

docker inspect my-service --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
# true 137

docker stats shows it pinned at the limit before each death. The problem is twofold: the limit is a bit low for the real working set, and the JVM isn't sizing its heap to the container.

First, give it a limit that reflects reality. I measured the steady-state working set at around 350 MB, so I'll allow headroom for the JVM's non-heap overhead (metaspace, thread stacks, off-heap buffers) on top of the heap:

docker run -d --name my-service \
  --memory=512m \
  -e JAVA_OPTS="-XX:MaxRAMPercentage=70.0" \
  my-registry/my-service:1.4.2

-XX:MaxRAMPercentage=70.0 tells the JVM to cap its heap at 70% of the container limit - leaving the other 30% for non-heap memory so the process total stays under 512 MB. On JDK 10+ container support (-XX:+UseContainerSupport) is on by default, so the JVM reads the cgroup limit rather than the host's RAM. On older JVMs you'd set an explicit -Xmx instead, but percentage-based is more robust across environments.

For a Node service the equivalent is bounding the old-space heap under the limit:

docker run -d --name my-worker \
  --memory=512m \
  -e NODE_OPTIONS="--max-old-space-size=384" \
  my-registry/my-worker:2.1.0

384 (MB) sits comfortably under the 512 MB container limit, leaving room for Node's other allocations.

The pattern in both cases: pick the container limit from measured usage plus headroom, then tell the runtime to keep its heap under that limit.

What to watch out for

• Setting the limit too low just to "cap" it. If the app has a leak, a tight limit doesn't fix the leak - it converts a slow degradation into a fast crash loop. You've made it more visible, not healthier. Fine as a deliberate blast-radius guard; not fine as a substitute for fixing the leak.
• No limit at all. One unbounded container can consume the host and get other containers OOM-killed. The victim in dmesg may be a service that did nothing wrong.
• Swap accounting. --memory and --memory-swap are different knobs. If you set --memory=512m and leave swap unset, Docker may allow up to twice the memory in swap, which masks the real usage - the container limps along swapping instead of failing cleanly. Set --memory-swap equal to --memory to disable swap for that container when you want hard, predictable behaviour.
• Measuring the wrong number. MEM USAGE in docker stats includes page cache, which can make usage look scarier than the actual anonymous (unreclaimable) memory that drives OOM decisions. Watch the trend and the kill line in dmesg rather than a single snapshot.

Making it repeatable

To stop this being a recurring surprise:

• Right-size from data. Use docker stats or your metrics stack to find the real working set under load, then set --memory to that plus honest headroom.
• Pin the limit and the runtime heap together. Set --memory and a matching runtime flag (-XX:MaxRAMPercentage, --max-old-space-size, etc.) so the two numbers can't drift apart.

# docker-compose.yml
services:
  my-service:
    image: my-registry/my-service:1.4.2
    mem_limit: 512m
    restart: on-failure
    environment:
      JAVA_OPTS: "-XX:MaxRAMPercentage=70.0"

• Add a restart policy and an alert. restart: on-failure keeps you online through a transient spike, and an alert on OOMKilled events or restart count means you hear about it before your users do. Be honest about the restart policy though: it buys time, it does not fix a leak. A container that restarts every ten minutes is telling you something you shouldn't silence.

If you'd rather not reassemble all of this under pressure the next time a container flaps, I keep the reusable Docker patterns - limits, healthchecks, restart policies - as a reference set of Docker runbook patterns so it's a lookup, not an investigation.

Wrapping up

Exit code 137 is almost always a memory conversation, not a crash. The durable fix isn't a bigger number - it's making the container limit and the runtime's own idea of "available memory" agree, sized from what the app actually uses. Get those two to match and 137 stops being a mystery.