DEV Community

Marking a container unhealthy didn't stop it from taking traffic

I had a container sitting in docker ps marked (unhealthy) for the better part of a minute while every request I sent through it kept coming back 200. That shouldn't be possible if a health check is supposed to mean anything, so I stopped and built a small stack to find out exactly what Docker's HEALTHCHECK actually controls, and what it doesn't.

The honest answer turned out to be: almost nothing, on its own. I had three things pointed at the same container - Docker's own health status, a restart: always policy, and nginx sitting in front as a reverse proxy - and none of the three reacted to the other two the way I assumed they would.

The setup

Three identical Python backends behind an nginx reverse proxy, all on plain Docker Compose, bridge networking, nothing exotic:

services:
  back1:
    image: swarmtest:v1
    restart: always
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health', timeout=1)"]
      interval: 2s
      timeout: 1s
      retries: 2
      start_period: 2s
  # back2, back3: identical, no restart policy
  proxy:
    image: nginx:1.27-alpine
    ports: ["18090:80"]

Each backend is a tiny HTTP server with four routes: / returns a version string, /health reports the health flag, /break flips the health flag without touching anything else, and /crash flips the health flag and makes / itself start returning 500.

That split matters, because it let me test two different failures separately: a health probe lying, and a health probe telling the truth.

nginx sat in front with a plain upstream block:

upstream backend {
  server back1:8080 max_fails=2 fail_timeout=5s;
  server back2:8080 max_fails=2 fail_timeout=5s;
  server back3:8080 max_fails=2 fail_timeout=5s;
}

Finding 1: an unhealthy container keeps serving exactly as before

I hit back1's /break endpoint, which only flips the health flag - the real / route still answers correctly. Docker noticed within a few seconds:

NAMES              STATUS
swarmtest-back1-1  Up About a minute (unhealthy)

Then I fired 160 real requests through nginx over the next 44 seconds while that status sat there unchanged.

  • Docker health flips to unhealthy: 3.64s after the break
  • Requests served successfully (200): 160 / 160
  • Requests that failed: 0

Nothing downstream of Docker's health state even looked at it. nginx has no concept of a Docker health check - it only knows about actual TCP connections and actual HTTP responses. A container can be "unhealthy" for as long as you like and keep taking its full share of traffic, provided the thing it's actually serving still works.

Finding 2: restart: always doesn't touch it either

I'd assumed a restart policy was the backstop here - if nginx doesn't care, surely Docker itself will restart something it's just told you is broken. It doesn't.

I added restart: always to back1, broke it for real this time with /crash (so / now also returns 500), and polled docker inspect every 5 seconds for 35 seconds:

RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z

Same RestartCount, same StartedAt, every single poll.

A restart policy reacts to the container's main process exiting with a non-zero code (or just exiting, depending on the policy). It has no wiring to the health check at all. The process inside my container never exited - it just kept answering 500 - so from the restart policy's point of view, nothing had gone wrong.

That's the real limitation, and it's worth being specific about it: in plain Docker Engine and Compose, HEALTHCHECK is a reporting mechanism, full stop. Nothing acts on it unless you orchestrate with something that does - Swarm will reschedule a service's task, and Kubernetes has its own liveness/readiness machinery - or you write the thing that does.

Finding 3: the actual cause of the 500s, and why the obvious fix line didn't fix it

Having established that nothing automatic happens, I wanted to know what did happen to real traffic when a backend genuinely started failing. I crashed back1 the same way and sent 300 requests over 61 seconds.

  • Successful (200): 200
  • Failed (500, reached the client): 100

A clean third of requests failed, matching the round-robin share, and the rate never improved across the full minute - the first failure landed at 0.41s, the last at 60.77s. max_fails=2 and fail_timeout=5s were both set on the upstream, so I expected nginx to pull back1 out of rotation after two bad responses. It never did.

The config I'd written was:

proxy_next_upstream error timeout http_502 http_503 http_504;

That line looked complete to me - it's the set of codes I see in most nginx reverse-proxy examples. But nginx's own docs are specific about this: http_500 is only counted as a failed attempt if you list it explicitly, same as 502, 503, 504 and 429.

My backend was returning plain 500, which wasn't in that list, so as far as max_fails was concerned, every one of those requests was a perfectly normal, successful round trip to back1. The failing status code quietly passed straight through to the client.

I added http_500 to the list and reloaded. Same crash, same 300 requests, same 61-second window:

  • Failed requests reaching the client before (http_500 missing): 100 / 300
  • Failed requests reaching the client after (http_500 added): 0 / 300
  • back1 present in any response before: yes, continuously
  • back1 present in any response after: no, after the first couple of attempts

With http_500 in the list, nginx retried the failing attempt against another backend inside the same client request, and back1 was excluded from the pool for the fail_timeout window after its second failure. The client never saw a single error.

This is the finding I'd act on fastest: if your application returns a bare 500 on its own errors - which is the default in most frameworks when something throws - and your proxy_next_upstream list was copied from a tutorial that only has 502 503 504, your failover silently does nothing for the exact case it's most likely to matter for.

Finding 4: the fix hides the failure from more than just the client

Masking the error at the proxy is good for whoever's using the app, but it's worth being honest about the trade-off. Any monitoring that counts response codes at the client-facing edge will now show a clean 200 for a request that actually failed once internally and was quietly retried.

If you're alerting on nginx access-log status codes rather than on upstream-level metrics, a backend can be failing every single request routed to it and your dashboards won't move.

I didn't instrument latency closely enough in this run to put a number on the added cost of that silent retry, but it's an extra full request-response cycle per failure, and that's not nothing under load.

Finding 5: what it actually costs to fix it yourself

Since nothing reacts to the health flag automatically, I wrote the small watcher a lot of teams end up writing: poll docker inspect, and after three consecutive unhealthy polls, issue a docker restart. Here's a real run, crash included:

t=2.11 health=healthy streak=0 CRASHED at relative t≈2s
t=6.22 health=unhealthy streak=1
t=7.26 health=unhealthy streak=2
t=8.28 health=unhealthy streak=3
t=8.28 ACTING: restarting swarmtest-back1-1
t=18.57 restart command returned

Two numbers stood out. Docker's own health check took about 4 seconds to report the problem at all, because interval: 2s and retries: 2 mean it needs two full cycles of bad probes before flipping status - that's a real, configurable cost, and it's one most people leave at whatever the tutorial they copied used.

My watcher added its own 1-second poll and a 3-strikes rule on top, so the whole detection side cost about 6.3 seconds. The bigger number was the restart itself: 10.3 seconds from "restart issued" to the command returning. My app doesn't handle SIGTERM, so Docker waited out its default 10-second grace period before killing it outright. The detection logic I wrote was the fast part. The actual recovery was dominated entirely by a stop timeout nobody had configured on purpose.

After the restart finished, I checked docker inspect out of habit, and RestartCount still read 0. It turns out that counter only moves when the restart policy fires it, not when something external calls docker restart, scripted or not. I'd been treating RestartCount as my go-to signal for "is this thing flapping" on real services. A watcher like the one above would quietly break that, so before I'd trust it anywhere that matters, I'd need to log the restarts myself rather than lean on Docker's own counter.

What I got wrong on the way

I spent a genuinely annoying ten minutes convinced my nginx config change wasn't taking effect, because docker exec ... cat /etc/nginx/nginx.conf kept showing the old file after I'd edited it on the host.

The bind mount hadn't actually broken - my edit had. I'd used sed -i, which on this system replaces the file by writing a new temp file and renaming it over the old path. A bind mount of a single file is attached to the inode that existed when the container started, not to the path, so the rename left my running nginx looking at an orphaned, deleted-but-still-open copy of the old config forever.

Switching back to a plain cat > file <<EOF, which truncates and rewrites the existing inode in place, fixed it immediately - once I recreated the container to re-establish the mount against the current file, since the old one was unrecoverable once detached.

Run it yourself

This is the core loop I used to measure availability through the proxy while breaking a backend on purpose:

import time, urllib.request, subprocess, json
def hit():
  try:
    r = urllib.request.urlopen("http://127.0.0.1:18090/", timeout=2)
    return True, r.status, r.read().decode()
  except Exception as e:
    return False, getattr(e, 'code', None), str(e)
subprocess.run(["docker", "exec", "<your-backend-container>", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/crash')"])
log = []
t0 = time.time()
for i in range(300):
  ok, code, body = hit()
  log.append({"t": round(time.time() - t0, 2), "ok": ok, "code": code})
  time.sleep(0.2)
json.dump(log, open("log.json", "w"))

Point it at any Compose stack with a proxy in front of a few backends, and docker inspect --format '{{.State.Health.Status}}' <name> alongside it to correlate health state with what clients actually saw.

Where I landed

I'm adding http_500 to every proxy_next_upstream line I own, today, not as a nice-to-have. I'm also going to stop treating HEALTHCHECK as a safety net for anything outside Swarm or Kubernetes - in plain Docker and Compose it's a status light, and if I want it to do something, I have to wire that up myself and budget for the stop-grace-period cost when I do. The watcher pattern works, but it's worth building with the restart latency in mind rather than assuming "detected" means "fixed" a second later.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.