When a Deploy Breaks Production: The Beginner's Rollback Playbook

It Happens to Everyone You pushed. The pipeline went green. You opened the site to admire your work - and it's a white screen. Or a 500. Or the checkout button silently does nothing. Your heart rate doubles, because this isn't localhost anymore. Real users are looking at your broken deploy right now. Here's what nobody tells beginners: every developer you admire has broken production. What separates seniors from juniors isn't avoiding it - it's having a calm, rehearsed answer to "what now?". This post is that answer: how to get back to the last working version fast, the one database trap that turns a bad evening into a disaster, and the five-point contingency plan that makes the next incident boring. Rule Zero: Restore First, Diagnose Later Your instinct will scream "find the bug and push a fix!" Resist it. Debugging under pressure produces bad code, and a rushed "hotfix" is how one outage becomes two. The professional move is boring: - Restore service - get users back onto the last version that worked. - Then diagnose - calmly, on staging, with coffee. Rolling back is not admitting failure. Rolling back is the plan. The First Two Minutes: Confirm and Read Before touching anything, spend two minutes knowing what you're dealing with: # Is it really down, or is it your browser cache? curl -I https://your-app.com # What is the app actually saying? tail -50 storage/logs/laravel.log # And the web server? sudo tail -50 /var/log/nginx/error.log Thirty seconds of reading beats thirty minutes of guessing: - 502 Bad Gateway - PHP-FPM is down or crashed. Often fixed by sudo systemctl restart php8.4-fpm alone. - 500 with a stack trace in the log - the trace names the exact file and line that broke. That tells you which commit to blame. - "Class not found" right after a deploy - composer install didn't run or finished halfway. - Works for you, broken for users - stale caches. Try php artisan config:clear && php artisan route:clear before anything dramatic. If one of those quick fixes restores service - great, you're done with the emergency. If the problem is genuinely the code you just shipped, keep reading. The Rollback Ladder How you roll back depends on how you deploy. Find your row: | How you deploy | How you roll back | Time to recover | |---|---|---| | Releases + symlink pattern | Point the symlink at the previous release | ~10 seconds | In-place git pull (Forge default) | git revert + redeploy | 1-2 minutes | | Envoyer / deploy tool with releases | Press the rollback button | Seconds | | FTP drag-and-drop | Restore your backup copy - then please adopt one of the above | Pray | Option 1: The Symlink Rollback (if you use the releases pattern) If you deploy with the releases/symlink pattern, every previous deploy is still sitting on the server. Rolling back is re-pointing one symlink: # Newest first - the SECOND line is your last good release ls -1t /var/www/app/releases # Point "current" back at it and reload PHP ln -sfn /var/www/app/releases/20260709210000 /var/www/app/current sudo systemctl reload php8.4-fpm Ten seconds, service restored, and the broken release is still on disk for you to examine. This one command is the entire reason the releases pattern exists - the full setup is in Zero-Downtime Deploys to a VPS with GitHub Actions. Option 2: git revert (in-place deploys) If your deploy is "git pull on the server" - which is also what Laravel Forge does by default - roll back by reverting the bad commit and deploying again, from your own machine: # On YOUR machine - never surgery on the server git revert HEAD # creates a NEW commit that undoes the last one git push origin main # your pipeline deploys the revert like any other change Shipped several bad commits? Revert a range: git revert --no-edit HEAD~3..HEAD . Why revert and not reset --hard + force-push? Revert moves history forward - the broken commit stays visible for the post-mortem, nobody's local clone breaks, and your CI pipeline treats the fix like any normal deploy. Force-pushing a shared branch mid-incident is how you turn one emergency into two. What NOT to Do (the panic list) - Don't SSH in and edit files with nano/vim. The next deploy silently erases your edit, and now production doesn't match git - the worst kind of bug to find later. - Don't push an untested "quick fix". You already shipped one thing that didn't work as expected tonight. - Don't restart random services hoping. Read the log first - it usually tells you exactly what's wrong. - Don't delete anything. The broken release/commit is evidence you'll want tomorrow. The Migration Trap - Read This Twice Rolling back code is easy. The database is where beginners get genuinely hurt. If your bad deploy ran migrations, rolling back the code does not undo the schema. And the tempting command… php artisan migrate:rollback # ⚠ can DELETE real user data …runs each migration's down() method. If that down() drops a table or column that has been collecting real user data since the deploy, that data is gone forever. No undo. Three rules to keep you safe: - Old code + new column = usually fine. Extra columns don't break the previous release. So in most incidents you can roll back the code, leave the schema alone, and decide calmly later. This is the default move. - Only run migrate:rollback whendown() is provably harmless - e.g. dropping a brand-new table that nothing wrote to yet. - Otherwise, fix forward: write a new migration that corrects the mistake, test it on staging, and ship it through the normal pipeline. Write migrations that make rollback safe The habit that prevents this whole class of problem (the grown-up name is expand and contract): never remove or rename a column in the same deploy that stops using it. Add the new nullable column and deploy. Migrate the data and deploy. Only deploys later - when nothing can possibly still read the old column - remove it. Every deploy in between is safely rollbackable. The Circuit Breaker: Maintenance Mode One special case: the broken code might be corrupting data - half-written orders, payments recorded twice. Frozen is better than corrupting. Laravel gives you a circuit breaker: # Users see a maintenance page that auto-refreshes every 30s. # YOU can still browse the site via https://your-app.com/peek-9f2k php artisan down --refresh=30 --secret="peek-9f2k" # ...roll back, verify... php artisan up Ten seconds of maintenance page is invisible in the monthly stats. Corrupted orders are not. The 10-Minute Incident Runbook Everything above, as the checklist to keep somewhere you can find while panicking: - Confirm it's really down: curl -I https://your-app.com - Read the last 50 lines of laravel.log and the Nginx error log - If broken code could corrupt data → php artisan down first - Roll back the code: symlink flip, or git revert + push - Leave the database alone unless you are certain the rollback is safe - fix forward tomorrow - Verify: curl -I returns 200, log stays quiet, click the critical flow once yourself - php artisan up if you went down - Tomorrow: reproduce on staging, write the test that would have caught it, fix properly, redeploy. (Red CI and can't see why? Here's the systematic way to debug it.) The Contingency Plan: Five Things Before Your Next Deploy Incidents are survived in the moment but prevented in advance. Five habits, cheapest first: - Back up the database before every deploy that migrates. One line in your deploy script, right before artisan migrate :mysqldump -u forge -p"$DB_PASSWORD" your_db | gzip > ~/backups/pre-deploy-$(date +%Y%m%d%H%M).sql.gz - Make "last good version" findable. The releases pattern gives you this for free (keep the last 5). On simpler setups, tag every deploy: git tag deploy-$(date +%Y%m%d%H%M) && git push --tags . - Health-check after every deploy. Laravel ships a /up endpoint - curl it as the last step of your pipeline so a dead deploy fails loudly in CI instead of quietly in production. - Deploy small, deploy often, never Friday evening. A deploy with 3 changed files has 3 suspects. A deploy with 3 weeks of work has 300. - Rehearse the rollback once on staging. Flip the symlink. Revert a commit. Time yourself. An untested rollback plan is a wish, not a plan. You're Allowed to Break Things A broken deploy doesn't make you a bad developer - shipping means occasionally shipping wrong. What you're building with the runbook above is the thing that actually matters: a short, boring path back to working. Once rollback takes ten calm seconds instead of a panicked hour, deploying stops being scary - and developers who aren't scared of deploying ship faster than everyone else. If you want the full foundation this playbook sits on - automated tests before every deploy, quality gates, and the zero-downtime setup that makes rollback a one-liner - start at the beginning of the series: What Is CI/CD? A Plain-English Guide for Laravel Developers. Got your own production horror story? Drop it in the comments - I read every one. Originally published at dineshstack.com - read the full version with code samples and updates there. Top comments (0)