The SSH Lockout That Made Me Close Port 22 For Good

The wrong assumption that cost the first two hours

My hosting provider's browser-based rescue console kept rejecting root's password, even right after resetting it through their panel. I assumed the reset was broken and spent ages fighting it. It wasn't broken.

On a standard Ubuntu cloud image, the account the rescue console logs in as is ubuntu, not root - a convention from how cloud images provision the default user. Root's password can be perfectly correct and still be useless there.

On top of that, my ubuntu account turned out to be locked (passwd -S showed L) since the box was created, completely unrelated to any password value. Two separate, unglamorous facts, neither documented anywhere I'd looked, together cost me two hours.

The second problem, stacked on top

Separately - and this part I never fully root-caused - direct SSH and HTTP to the server's public IP was timing out entirely, not rejecting auth, just hanging. I ruled out the firewall (0 rules at the panel level), fail2ban (ban window had long expired). My best guess: something about the source IP I was connecting from, which rotates on a home/mobile connection. It resolved itself once that IP changed.

The apps themselves never went down - I checked via an external monitor the whole time. Only my own access was blocked.

The actual recovery

When both normal SSH and the console login are dead, most hosting panels have an "Emergency Mode" or rescue mode: it boots a minimal OS and mounts your real disk read/write somewhere like /mnt. From there:

lsblk  # find the real disk, not the rescue OS's own
mount --bind /dev /mnt/sdX/dev
mount --bind /proc /mnt/sdX/proc
mount --bind /sys  /mnt/sdX/sys
chroot /mnt/sdX /bin/bash

Inside the chroot, check lock status before touching anything:

passwd -S ubuntu      # L = locked, P = usable
passwd -u ubuntu      # unlock
echo 'ubuntu:newpassword' | chpasswd   # NOT interactive `passwd` here

That last point matters more than it looks: pasting a password into an interactive passwd prompt from a rescue-mode terminal garbled it badly for me the first time. chpasswd piped from echo is the reliable way to do it non-interactively.

One more trap: a lot of panels offer a one-click "reset firewall configuration." It's not a targeted unblock - it fully disables the firewall. Don't reach for it as a first move.

The fix that actually matters

The real fix wasn't unlocking accounts - it was removing the single point of failure that caused this: SSH access tied to a rotating public IP with no fallback path.

I installed Tailscale on the server and on every device I administer it from. Now admin access goes over the tailnet's stable private IP, completely independent of whatever public IP my ISP hands me that day. I deliberately didn't enable Tailscale's own SSH server - sshd stays the single SSH auth surface, Tailscale just becomes the network path to reach it.

Once every CI pipeline that deployed to this box was migrated to connect the same way - join the tailnet as a short-lived tagged node, then SSH over the stable internal IP - I closed the public port 22 rule entirely. SSH is Tailscale-only now, full stop.

One gotcha worth flagging if you do this: a tagged device in Tailscale's ACL is not automatically covered by the default "logged-in members can reach each other" rule. A CI node can join the tailnet fine and still get connection timeouts on the actual SSH attempt, because the tag was never explicitly granted access in the ACL policy. It needs its own explicit grant.

What I'd tell past-me

• On a cloud VPS, check the actual console account before assuming it's root - it usually isn't.
• Don't let a single rotating IP be your only path to a server. Put a stable, IP-independent access layer (a mesh VPN like Tailscale, or equivalent) in place before you need it, not while you're locked out.
• Tagged/automated identities need explicit ACL grants - "it joined the network" and "it can reach anything" are two different guarantees.
• A locked-out admin panel doesn't mean a down service - verify what's actually broken (access vs. the app itself) before you start firefighting.