A Symlink Swap Is Not a Zero-Downtime Deploy

TL;DR - I had Capistrano-style releases on a plain VM: build into releases/ , swap a symlink, restart the service. Atomic, I thought. Then I put curl in a loop and watched an actual deploy: 502 on every single one, for the length of the app's boot. The files were atomic. The process never was. And the nginx directive everyone reaches for to paper over that does absolutely nothing in the shape most people write it. The half that was already right The release layout is the well-trodden one, and there's nothing wrong with it: /srv/app/ releases/20260831-101500-a1b2c3d/ releases/20260830-093000-9f8e7d6/ current -> releases/20260831-101500-a1b2c3d Build into a fresh directory, then move the pointer. One detail worth getting right, because two of the three obvious ways to write it are broken: # WRONG - -sf follows an existing symlink-to-a-directory and # quietly creates /srv/app/current/20260831-101500-a1b2c3d ln -sf "$RELEASE" /srv/app/current # STILL WRONG - -n fixes that, but this is unlink() then symlink(). # There is a window where the path resolves to nothing. ln -sfn "$RELEASE" /srv/app/current # RIGHT - create beside, then rename(2) over the top. Atomic. ln -sfn "$RELEASE" /srv/app/current.tmp mv -Tf /srv/app/current.tmp /srv/app/current mv needs -T for the same reason ln needed -n : without it, moving onto an existing symlink-to-a-directory moves into it. Get that right and your files switch instantly. A request that started on the old release finishes against the old tree; the next one gets the new one. Genuinely atomic, genuinely nice. Which is exactly why the next line goes unexamined for months. The half nobody looks at systemctl restart app restart is stop , then start . Between those two, nothing is listening on your app's port. t+0.00 mv -Tf current files switch, atomic ✓ t+0.01 systemctl stop app port closes t+0.01 GET / → 502 t+0.4 GET / → 502 t+1.2 GET / → 502 t+2.8 app finished booting, binds port t+2.9 GET / → 200 The deploy is atomic right up to step 3. Everything red after it is the part the symlink swap never covered. Three seconds of hard 502s, on every deploy, recovering by itself. That last part is why it survives so long: nobody files a bug for something that fixed itself before they could screenshot it. It gets called "flaky." It isn't flaky. It's exactly what you told the machine to do. Two shapes, two different 502s If you serve a process on a port, it's the obvious one: location / { proxy_pass http://127.0.0.1:3000; } Process dies → port closes → connect() refused → nginx has nothing to say but 502. The second shape catches people out, and it caught me. Conventional PHP on a VM: nginx owns the docroot and only .php reaches the app. root /srv/app/current/public; location ~ .php$ { include fastcgi_params; fastcgi_pass unix:/run/app/php-fpm.sock; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; } The reasoning goes: nginx serves the files itself, so a restart can't matter much. It matters completely - if the fpm master runs under the same systemd unit as the app. Which is often exactly how you want it, because then logs, resource limits and lifecycle all work per-app instead of being shared through one system-wide pool. The cost of that choice: restarting the unit kills the master, the socket file is unlinked, and fastcgi_pass gets ENOENT until it comes back. Same 502, different mechanism, and the static assets keep serving perfectly the whole time - which makes it look like an application bug rather than a deploy bug. The directive that does nothing Everyone's first fix, mine included: location / { proxy_pass http://127.0.0.1:3000; proxy_next_upstream error timeout; # ← inert } This changes nothing. proxy_next_upstream retries the next peer in the upstream group. A proxy_pass at a literal host:port is a group of one. There is no next peer, so there is nothing to retry to, and you get the same 502 you got before - now with a config line that makes you think you handled it. The version that works looks like a typo: upstream app { server 127.0.0.1:3000 max_fails=0; server 127.0.0.1:3000 max_fails=0; # yes, the same address twice } location / { proxy_pass http://app; proxy_next_upstream error timeout invalid_header non_idempotent; proxy_next_upstream_tries 3; proxy_connect_timeout 2s; } Listing the address twice is the entire mechanism. Now there is a next peer, so a refused connect gets retried instead of reported. max_fails=0 is not decoration either. The default is max_fails=1 fail_timeout=10s , and these two "servers" are one process - so a single refused connect marks both peers down and nginx serves 502 for the next ten seconds. You'd have converted a 200 ms gap into a ten-second outage while believing you'd added resilience. And while you're in there, stop showing people the nginx default page: error_page 502 504 = @unavailable; location @unavailable { default_type text/html; add_header Retry-After 5 always; return 503 ' Starting up This service is starting. Refresh in a few seconds.'; } 503 with Retry-After , not 502. A process that is starting is not a broken upstream, and the status code is the only part of that a client, a CDN or a health checker can act on. But be clear-eyed: all of this buys you a retry, not zero downtime. It covers a sub-second gap. It does not cover a three-second boot. For that you need to stop having a gap at all. What actually fixes it: two of everything Blue-green, but with systemd doing the work. A template unit gets you both colours from one file: # /etc/systemd/system/app@.service [Unit] Description=app (%i) After=network-online.target [Service] Type=simple User=app WorkingDirectory=/srv/app/slots/%i EnvironmentFile=-/srv/app/slots/%i.env ExecStart=/usr/bin/node server.js Restart=always RestartSec=5 [Install] WantedBy=multi-user.target systemctl start app@blue and app@green are now two independent services. The layout grows a slots directory: /srv/app/ releases/ / current -> releases/ the version that is SERVING slots/blue -> releases/ one colour's release slots/green -> releases/ slots/blue.env PORT=41000 slots/green.env PORT=41001 And the deploy becomes a sequence rather than a restart: active=$(systemctl is-active --quiet app@blue && echo blue || echo green) standby=$([ "$active" = blue ] && echo green || echo blue) # 1. Build the release. Nothing serving is touched. build_into "$RELEASE" printf 'PORT=%s\n' "$STANDBY_PORT" > "/srv/app/slots/$standby.env" ln -sfn "$RELEASE" "/srv/app/slots/$standby.tmp" mv -Tf "/srv/app/slots/$standby.tmp" "/srv/app/slots/$standby" # 2. Start the standby beside the one that is serving. systemctl restart "app@$standby" # 3. Prove it answers, on its OWN port - not through the proxy, # which is still pointing at the version we are replacing. curl -fsS --retry 20 --retry-delay 1 "http://127.0.0.1:$STANDBY_PORT/up" >/dev/null # 4. The switch. A reload does not drop connections in flight. write_upstream "$STANDBY_PORT" nginx -s reload # 5. Only now is the old one expendable. systemctl disable --now "app@$active" Same deploy, two colours. Blue keeps answering until step 4, and step 4 is a reload, not a restart. Step 3 is the one people skip and shouldn't. Probe the standby's own port, never the public URL - the public URL is still answered by the version you're trying to replace, so a probe through it always passes and proves nothing. Step 4 is the moment traffic moves, and it is the only moment anything visible changes. An nginx -s reload starts new workers for the new config and lets the old workers finish what they're holding. Nobody gets cut off mid-response. The thing container people never have to think about If you've only done blue-green with containers, this is the part that doesn't transfer. Each container gets its own network namespace, so both colours can bind port 3000 and only the network alias tells them apart. The name moves; the number stays. On a plain host there is one port space. Two processes cannot both bind 3000. So the number is what moves, and the hostname stays 127.0.0.1 . Everything downstream follows from that one fact: - your proxy repoint has to rewrite the port, not the upstream host; - you need somewhere to keep "which port is this colour on" - I write it to slots/ .env and read it back at switch time; - pick your slot ports above the registered range and below ip_local_port_range (32768-60999 on most boxes), or you'll eventually collide with an ephemeral port the kernel handed out for an outbound connection. 41000-41999 is a quiet neighbourhood. And one consequence worth refusing rather than shipping: if your app serves two domains on two ports, a colour only rebinds the primary one. The second domain keeps answering from the colour you are about to kill. That's a half-switched release, which is worse than an honest restart - so detect it and refuse, loudly, with a message that says why. Two rules I'd argue for in any implementation A colour's WorkingDirectory points at its own release, never at current . This sounds like a detail and it's the whole thing. Two versions can only run at once if each has its own working directory. Point both at current and you've rebuilt the restart with extra ceremony. current moves at promote, not at deploy. It should keep meaning "the version that is serving" right up until the switch. That matters because your companion units follow it: # worker unit - deliberately on current, not on a colour [Service] WorkingDirectory=/srv/app/current ExecStart=/usr/bin/php artisan queue:work Restarting that at promote - and only at promote - is also exactly right for a queue worker. A worker holds the code it started with, and the moment the new version becomes canonical is the moment it should pick it up. Not while it's still a candidate. Gate on what you can actually reach Back to the php-fpm shape, because it's where I had to be honest with myself. A standby fpm master listens on a unix socket and nothing else. There's no port to curl. And the