Disk full but no big files to be found: tracking down deleted, still-open files with lsof
"The disk is full, I already deleted the big log file - but not a single byte was freed." A classic. The file is gone from the directory, but its content still takes up space.
Why deleted files still use space
On Linux, rm only removes the directory entry, the name. The file system releases the data only when no name points to the file and no process has it open any more.
If a service keeps a file open - typically a web server, a database or an application with its log file - the space stays allocated. Often the process even keeps writing to it. The file just can't be reached through a path any more.
That explains the disagreement between two tools: df asks the file system how much space is used and includes the file. du walks the directories and no longer finds it.
Finding deleted but still-open files
First confirm the gap:
df -h /var
du -xsh /var
If the numbers are far apart, lsof shows the suspects. The +L1 option lists open files with a link count below one, i.e. already deleted:
sudo lsof -nP +L1
The interesting columns are COMMAND, PID, FD, SIZE/OFF and the path marked (deleted).
To limit the output to one file system, append the mount point:
sudo lsof -nP +L1 /var
If lsof isn't installed, /proc helps:
sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null
Freeing the space
Reload or restart the service. Usually the cleanest way. Many services reopen their log files on reload, for example:
sudo systemctl reload nginx
sudo systemctl restart servicename
A reload only helps if the service reopens its logs while doing it. Otherwise a restart is needed - on production systems, with care and at a suitable time.
Truncate the file through /proc. If a restart isn't possible right now, the deleted file can be truncated to zero bytes through its file descriptor. PID and FD number come from the lsof output (for FD 4w, the number is 4):
sudo sh -c ': > /proc/1234/fd/4'
This is a stopgap: the content is gone afterwards, so be sure it really is a log file you can do without.
Why it happens in the first place
- Manually deleting a live log file with
rm. Better: empty it instead of deleting it, for example withtruncate -s 0 file.log. - Log rotation without reopening.
logrotaterenames the file or removes old rotations, but the service keeps writing to the old one. Fix: reload the service in thepostrotateblock, or usecopytruncatefor services that can't reopen their logs. - Hung or forgotten processes, such as an old worker that was never stopped.
Not every gap between df and du has this cause. Other suspects: files hidden under a mount point mounted on top of them later, and on ext4 the blocks reserved for root. And if df -h shows free space but nothing can be written, check df -i - the inodes may be exhausted.
How to check
- Compared
df -handdu -xshfor the same file system? - Ran
lsof -nP +L1and found processes with(deleted)files? - Reloaded or restarted the service, then checked
df -hagain? - Reviewed the logrotate configuration for the service (
postrotateorcopytruncate)? - Ruled out inodes with
df -iand hidden mount points? - Set up monitoring for disk usage?
A full disk is often just the symptom. The cause is frequently a log nobody rotates - a good example of why the second look pays off.
Originally published at daniele-michelin.com. More guides, free checks (SPF, TLS, headers) and an error-message lexicon: daniele-michelin.com/en/.
Comments
No comments yet. Start the discussion.