Linux Performance Troubleshooting: The 60-Second Check and Beyond

When a production server starts misbehaving - latency spikes, dropped requests, OOM kills - you don't have time to read man pages. You need a systematic approach that rules out the most common culprits in under a minute, then drills into the root cause with targeted tools. This is exactly how I debug Linux performance issues on production systems, and it works whether you're on a bare-metal box, a VM, or a container host. This guide covers the full troubleshooting workflow. Here's what we'll walk through: the 60-second checklist (uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, ss, sar), CPU hot-spot analysis with perf and flamegraphs, memory pressure including OOM and swap, disk I/O latency with iostat and blktrace, network troubleshooting with ss, tc, and tcpdump, and finally a decision tree to choose the right tool for any symptom.

1. The 60-Second Checklist

When you first SSH into a struggling box, run these commands in order. Each takes one look at a different subsystem. By the time you're done, you'll know which resource is the problem.

$ uptime                                     # load averages - is the system under pressure?
$ dmesg -T | tail -20                        # kernel messages - OOM kills, TCP drops, I/O errors
$ vmstat -SM 1 3                             # system-wide CPU, memory, and I/O overview
$ mpstat -P ALL 1 1                          # per-CPU balance - one core pegged or evenly spread?
$ pidstat 1 3                                # per-process CPU usage - who's burning cycles?
$ iostat -xz 1 1                             # disk I/O - await, svctm, %util
$ free -m                                    # memory usage - total, available, cached, swap
$ ss -tupwn                                  # socket connections - listening, established, TIME_WAIT
$ sar -n DEV 1 1                             # network throughput - packets/sec, errors, drops

I have an alias called perf60 that runs all of these. The point isn't to memorize every flag - it's to have a checklist you always run before diagnosing deeper. Skipping this step is how people spend an hour debugging a memory leak that the OOM killer already logged in dmesg.

Pro tip: Save these commands as a shell script or alias and scp it to every new server you provision. Your future on-call self will thank you.

2. CPU - Finding the Hot Spots

If uptime shows load averages significantly higher than the number of CPUs, and mpstat shows high %user or %system across cores, you have a CPU-bound issue. The next step is figuring out what is burning those cycles.

Per-Process with pidstat

$ pidstat -p ALL -u 1 3
Linux 6.8.0 (web-02) 07/20/2026 _x86_64_ (8 CPU)
02:14:33      UID       PID    %usr   %system  %guest   %CPU    CPU  Command
02:14:34     1001      1234   85.2      2.1     0.0    87.3      3  nginx
02:14:34        0      4567    0.3     12.4     0.0    12.7      7  kworker/u16:1

Here, nginx worker 1234 is using 87% of a single core. That's suspicious - either it's stuck in an infinite loop or it's genuinely serving heavy traffic. The high %system on the kworker suggests something happening at the kernel level - check disk or network interrupts next.

Profiling with perf and Flamegraphs

When you need to know which function is hot, nothing beats perf. It uses hardware performance counters and stack sampling with negligible overhead - safe for production:

# Record CPU samples for 30 seconds
$ perf record -F 99 -a -g -- sleep 30

# Generate the report
$ perf report --stdio --sort comm,dso,symbol

The -F 99 flag samples at 99 Hz (below 100 to avoid aliasing with common timer frequencies). The -g flag captures call graphs. Pipe the output into Brendan Gregg's flamegraph scripts for a visual:

$ perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > cpu-flame.svg

A flamegraph immediately shows you the hot code paths. Wide bars at the top are functions spending the most CPU time. If you see __do_fault or page_fault near the top, your application is page-faulting heavily - not a CPU problem at all, but a memory one.

3. Memory - OOM, Swap, and Page Cache

Memory issues show up in three flavors: OOM kills (the kernel murders a process), swap thrashing (the system is paging like crazy), and page cache pressure (the filesystem cache is starving applications).

Check OOM Kills First

$ dmesg -T | grep -i "oom|killed"
[Jul 20 01:23:45] mysqld invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE)
[Jul 20 01:23:45] mysqld: cpuset=/ mems_allowed=0
[Jul 20 01:23:45] Tasks state (memory reserved: 0K, swap_total: 0K)
[Jul 20 01:23:45] [ 1234] mysqld 1001 12345678 23456789123450 0 mysqld
[Jul 20 01:23:45] Out of memory: Killed process 1234 (mysqld) total-vm:12345678kB

The OOM killer logs every candidate process and the final victim. If you see your application here, the fix is not "add more RAM" (well, sometimes it is) - it's first checking for a memory leak.

Read the Numbers Correctly

The free -m output can be misleading. The available field is what matters:

$ free -m
              total  used  free  shared  buff/cache  available
Mem:          15950  8940  1020     420        5990       6010
Swap:          2048   120  1928

Linux uses free RAM for disk cache aggressively. The available field (6010 MB) estimates how much memory is truly free for new allocations - it includes reclaimable cache. If available is below 10% of total, you're in danger territory.

vmstat -SM 1 shows swap activity. If si (swap in) and so (swap out) are consistently non-zero, the system is paging - and performance will be terrible:

$ vmstat -SM 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1   120   1020    420   5990   50   45   1200   800  2000 1500  8  6 20 20  0

The si/so of 50/45 MB/s means 95 MB/s of swap traffic. That's disk-bound paging. Every page fault now waits for disk I/O. The wa column (20%) confirms it - the CPU is waiting on I/O.

Finding Memory Leaks

Use pidstat -r 1 to watch per-process RSS growth over time:

$ pidstat -r 1 10
02:20:00      UID       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
02:20:01     1001      1234   12345.0       0.0    2.5G   1.8G   11.5  java
02:20:02     1001      1234   12350.0       0.0    2.5G   1.8G   11.5  java
02:20:03     1001      1234   14420.0       0.0    2.7G   2.0G   12.6  java   ← growing

If RSS keeps climbing without stabilizing, you have a leak. For Java, run jmap -histo:live to dump heap. For other apps, /proc/PID/smaps gives the full picture of memory mappings.

4. Disk I/O - Finding the Slow Storage

High wa in vmstat or iowait in mpstat points to disk contention. The iostat -xz 1 command reveals which device and which operation is slow:

$ iostat -xz 1
Device   r/s     w/s    rkB/s     wkB/s  await  svctm  %util
sda     500       0    81920         0   24.0   0.40   20.0
sdb       0    1200        0     30720  180.0   1.20   14.4

Key fields to interpret:

• await (ms) - average time for an I/O request to complete. Modern SSDs should be under 2 ms. Above 10 ms means something is wrong.
• svctm (ms) - the actual service time for a single I/O. If await >> svctm, the requests are queuing (too much concurrency for the device).
• %util - percentage of time the device was busy. Important caveat: modern NVMe SSDs can parallelize I/O deeply, so %util can hit 100% even though the device isn't saturated. Use iostat -x 1 -m and watch aqu-sz (average queue size) instead.

Here sdb has an await of 180 ms - terrible. It's a write-heavy workload on what is likely a slow HDD or a network filesystem. The queue is deep: svctm is 1.2 ms while await is 180 ms, meaning requests are spending 99% of their time waiting in line.

Trace Specific I/O with blktrace

When you need to know which process is thrashing the disk:

# Trace sdb for 5 seconds
$ blktrace -d /dev/sdb -o - | blkparse -i - -o /tmp/blk-trace.txt
$ grep -E " D| A" /tmp/blk-trace.txt | awk '{print $12}' | sort | uniq -c | sort -rn

This shows you which process names are doing the most I/O. Combine with iotop -oP for a live view.

5. Network - Connections, Drops, and Latency

Network issues masquerade as application slowness all the time. Before blaming the code, rule out the wire.

Connection State with ss

$ ss -s
Total: 1234 (kernel 1567)
TCP:   567 (estab 234, closed 300, orphaned 3, synrecv 0, timewait 156/0), ports 45
Transport Total     IP        IPv6
*         1567      -         -
RAW       0         0         0
UDP       12        10        2
TCP       567       523       44
INET      579       533       46
FRAG      0         0         0

156 TIME_WAIT connections isn't a problem by itself. But if ss -t state time-wait shows thousands, you're exhausting ephemeral ports - typically caused by HTTP/1.0 short connections or a misconfigured proxy pool.

# Check listen backlog drops
$ ss -ltp | grep -E "^(LISTEN|State)"
$ netstat -s | grep -i "listen.*overflowed|LISTEN.*overflows"
1234 times the listen queue of a socket overflowed

A listen backlog overflow means the application isn't accepting connections fast enough. Increase net.core.somaxconn and the application's backlog parameter. But this is a band-aid - the real fix is making your app accept connections faster (thread pool sizing, event loop tuning).

Packet Drops and Retransmits

$ netstat -s | egrep "segments retransmited|packets pruned|TCPLoss"
8923 segments retransmited
156 packets pruned from receive queue
78 TCP sockets finished time wait in fast timer

Retransmits above 0.1% of total segments mean packet loss on the network. Use tcpdump -i eth0 host $UPSTREAM_HOST to capture and look for duplicate ACKs. It's often a saturated upstream switch port or a duplex mismatch.

Bandwidth and Queue Drops

$ sar -n DEV 1 1 | grep eth0
02:30:01 eth0  65000.00  8000.00  78901.00  6789.00  0.00  0.00  0.00

$ tc -s qdisc show dev eth0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 123456789 bytes 98765 pkt (dropped 456, overlimits 0)
 backlog 0b 0p requeues 0

The dropped 456 in tc output means the transmit queue is overflowing. Your application is sending data faster than the NIC can transmit it. Either increase the txqueuelen (ip link set eth0 txqueuelen 10000) or reduce the application burst rate.

6. Decision Tree - Which Tool for Which Symptom

Here's a quick reference to jump directly to the right tool based on what you observe:

Putting It All Together

Effective Linux performance troubleshooting is a skill you build by doing - but always start with the 60-second checklist. It forces you to check CPU, memory, disk, and network before jumping to conclusions. I've lost count of how many "application bugs" turned out to be a server running out of memory or a saturated network link.

The tools here (perf, vmstat, iostat, ss, blktrace, tcpdump) are available on every Linux distribution and have zero dependencies. Learn to read their output fluently, and you'll be the person in the war room who can say "it's the disk queue depth, not the application code" with confidence.

One final rule: instrument before you need it. Set up sar data collection (sysstat package), configure kernel audit logging for OOM events, and deploy a metrics agent that keeps historical data. When the next incident hits - and it will - you'll have the baseline to know whether things are actually worse than normal.