How I got a suspended TrueNAS pool back online without replacing a single disk

The rack had just come back online after an outage. Most things looked fine - VMs booted, the network reconverged, monitoring caught up on the missed intervals. Then I ran zpool status on one of the TrueNAS SCALE nodes and saw output I hadn't seen in production before. An 11-disk RAIDZ1 vdev. Multiple drives marked as faulted. One member listed as removed. The pool itself in SUSPENDED state.

The reflex, in that moment, is to start replacing disks. That reflex is wrong - at least as the first move. This piece is how I got the pool back online without touching a single drive, and what I learned about how ZFS reports failure states along the way.

The pool in question is on my TrueNAS SCALE machine, which runs alongside a separate CORE box on a different chassis. The recovery pattern is the same on either - this isn't SCALE-specific ZFS behaviour - but the incident happened on SCALE, so that's where the console output in this piece comes from. For privacy, I'll refer to the affected pool as MediaPool-A.

What the console showed

The output was the classic suspended-pool warning:

state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.

Then the vdev listing. The RAIDZ1 topology looked much worse than it turned out to be - most of the 11 disks were showing as faulted, and one was marked as removed. If you'd shown me that output out of context and asked me to guess what happened, I'd have said "shelf died" or "backplane failure." The vdev looked shot.

But there was a shape to the failure that mattered. Multiple disks going down at the same time is exactly what you'd expect if something they shared had gone down - an HBA path, a SAS cable, a backplane, a controller - rather than the disks themselves failing individually. Simultaneous multi-disk failure is one of those things that sounds catastrophic and usually isn't. Independent failures don't cluster in time. Shared-path failures do. That's the diagnostic reframe the piece is built around.

Before you touch a disk, ask what those disks have in common that isn't the disks.

The reflex to resist

The most important part of ZFS recovery is what you don't do in the first ten minutes. Here's the list, in order of danger:

• Don't add disks to the pool. Whatever's shown as unassigned in the TrueNAS GUI may not actually be unused. If the pool is suspended and the disks are still recognised by the OS, they're still valid pool members even if the GUI is being coy about it. Adding them to a new vdev structure is the fastest way to permanently lose the pool.
• Don't wipe any disk. Same reasoning. If a disk still has valid ZFS labels for this pool, wiping it is destructive.
• Don't replace any disk. Not until you've confirmed there's a disk that actually needs replacing. See earlier point on multi-disk-fault patterns.
• Don't export or destroy the pool. zpool destroy is exactly what it sounds like. zpool export is safer but shouldn't be done while the pool is in an unstable state.
• Don't start a scrub while the pool is suspended. A scrub is not the recovery mechanism when the pool has been suspended due to I/O access problems. The pool needs to be brought back into a stable online state first. Scrubbing before that either does nothing useful or, worse, hammers the disks while they're on an unreliable path.

That last one catches people. Scrubs are the go-to "is everything OK?" tool in ZFS-land, and the muscle memory when something looks wrong is to run one. Muscle memory is wrong here. A scrub is validation, not recovery. It belongs at the end of the recovery process, not the start.

Step one: are the disks actually there?

Before running anything against the pool, I checked whether the operating system could see the disks. This is the cheapest possible diagnostic and it usually tells you which world you're in.

smartctl --scan

The disks were visible. All of them. That was the first good sign - it meant the OS could see the hardware, which meant whatever had happened during the outage had resolved at the hardware/path level, which meant the pool was recoverable in software without touching any physical component.

Then I checked kernel logs for signs of ongoing HBA or SAS trouble:

dmesg -T | grep -iE "sas|scsi|mpt|mps|mpr|reset|timeout|failed|medium error|sense|I/O error|device offline|link"

The kernel log showed events from the outage itself - expected, since the disks had briefly gone away and come back - but nothing recent. No active storms of SAS resets. No timeouts still firing. No disks going offline. The path was stable now, whatever had happened during the outage.

That was the second good sign, and it was the one that gave me confidence to move to the recovery step. If the kernel had still been throwing SAS resets or timeouts in the current second, I would have stopped, escalated to the physical layer - cables, HBA, PSU - and only come back to ZFS after the path was clean. Running zpool clear against a still-unstable path just re-triggers the fault, and does it at higher urgency this time.

zpool clear, safely

Once the two "is the path stable?" checks came back clean, the recovery command was the one ZFS itself had already suggested:

zpool clear MediaPool-A
zpool status -v MediaPool-A

The pool returned to an ONLINE state. One disk briefly showed a small number of checksum errors on the first status check after the clear - expected, because the outage had generated some genuinely bad reads that ZFS had queued up. A second zpool clear zeroed the counters, and the follow-up status came back clean:

MediaPool-A = ONLINE
all disks = ONLINE
READ / WRITE / CKSUM = 0 errors
No known data errors

No files reported as corrupted. No permanent errors. The pool was back.

The recovery itself was two commands. The rest of the work was in the "is the path actually stable?" checks before them - cheap to run, and the difference between a clean clear and re-triggering the fault at a moment when the pool was already suspended.

What actually happened

The most likely cause was a temporary I/O path disruption during the outage. Something in the shared path - HBA, cable, expander, backplane, or the disks all briefly losing power at slightly different times as the rack came back - presented as multiple simultaneous device failures to ZFS. ZFS did exactly what it's designed to do: suspended the pool to protect the data from being written to devices it wasn't sure were healthy.

Once the path stabilised, the disks were visible again. zpool clear told ZFS "the fault condition is resolved, retry," and ZFS did. Because the disks and their data were still intact underneath, the retry succeeded, and the pool came back with no permanent state loss.

The three ZFS terms that mattered here are worth stating precisely, because their intuitive meanings are misleading:

• FAULTED does not always mean a dead disk. It means ZFS has decided this device is unreliable enough to stop trusting for the moment, based on the errors it's seen. A device can be FAULTED for reasons that have nothing to do with the disk itself.
• REMOVED does not always mean physically removed forever. It means the device that was here at some point isn't visible right now. It can be transient.
• SUSPENDED does not always mean the pool is lost. It means ZFS has stopped I/O on the pool to protect it, pending resolution of whatever fault condition triggered the suspension.

Context is what tells you which of those interpretations applies. Multiple simultaneous FAULTED devices, immediately after a rack outage, on drives you can still see with smartctl and don't have active kernel errors on, points at "transient path disruption during the outage" and away from "disks died." A single FAULTED device weeks after normal operation points the other way, and the response is completely different.

What I'd make instinctive

Two things, and one thing I already had right.

Look at the shared layer first. When multiple disks fault at once, the useful question is not "which disks failed?" but "what do they share that could have failed?" HBA, SAS path, backplane, PSU, controller firmware. Independent failures don't cluster in time; shared-path failures do. Every panic-response-to-multi-disk-fault I've ever done has, eventually, come back to something at the shared layer. Skipping straight to the shared layer as the first diagnostic saves you the panic.

zpool clear is safe only after the path is confirmed stable. It is not a magic fix. If the underlying path is still unstable, zpool clear re-triggers the fault immediately, and the second suspension arrives faster than the first because ZFS has now seen the pattern twice. Confirming path stability first - smartctl --scan, then dmesg | grep for active errors - takes about thirty seconds and is what makes zpool clear a recovery tool instead of a way to make things worse.

The thing I had right: not scrubbing while the pool was suspended. That's an instinct worth keeping. Scrub validates a stable pool. It doesn't recover an unstable one.

Closing

This was a clean recovery because the underlying condition turned out to be transient. The pool came back with no touch to any physical disk, no wipe of any data, no rebuild of any vdev. The recovery cost was two commands and about ten minutes.

Not every ZFS recovery is that clean. The same outage that suspended MediaPool-A also took down a second pool on the same box in a different, harder way - one where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to prove the disks were still there before running any of the commands above. That's the next piece.

If you're recovering from something like this now: check the disks are visible, check the kernel logs are quiet, then run zpool clear. In that order. Everything else you might be tempted to do can wait.