I'm a Dev Who Barely Knows the Kernel. Here's How I'm Learning How to Track a Packet with pwru

The Problem: Packets Don’t Have a Call Stack You Can Just console.log

As a web dev, when something’s broken, I add a console.log, or I set a breakpoint, or worst case I print my way to enlightenment. The request comes in, flows through my middleware, my handlers, my ORM, and I can trace every step because it’s all just… functions calling functions, in my process, in my language.

A packet inside the Linux kernel does not work like that. It gets handed from function to function across the network stack (ip_rcv, tcp_v4_rcv, netfilter hooks, qdiscs, the works), it can get cloned, NAT’d, encapsulated in a tunnel, handed to a virtual ethernet pair, sent through an eBPF program of its own, and dropped at literally any of a few thousand points, often silently. There’s no stack trace. There’s no packet.log().

The tool you reach for as a network person is tcpdump, which is great for seeing a packet arrive and a packet leave, but useless for seeing what happened to it in between, inside the kernel, especially when it never arrives at all. That’s the gap pwru (packet, where are you?) fills. It’s basically an eBPF-powered console.log for the kernel’s packet path.

Attaching a Debugger to (Checks Notes) Every Function in the Kernel

Here’s the first thing that made my jaw drop reading this codebase. Instead of the pwru authors hand-picking a list of “interesting” kernel functions to trace (which is what I assumed, naively), they let BTF (BPF Type Format, basically the kernel’s own type metadata) do the work for them.

In internal/pwru/utils.go, GetFuncs walks the entire kernel type database, looks at every single function’s signature, and asks one question: “does any of your first five parameters point to a struct sk_buff?” If yes, that function touches a packet, and it gets added to the hit list.

if ptr, ok := p.Type.(*btf.Pointer); ok {
    if strct, ok := ptr.Target.(*btf.Struct); ok {
        if strct.Name == "sk_buff" && i <= 5 {
            funcs[name] = i
            continue
        }
    }
}

That’s it. That’s the whole discovery mechanism. On a typical kernel this turns up a few thousand functions, and pwru then kprobes (or kprobe-multi attaches, in batches, concurrently) all of them, at once. Every one of those probes runs the same handler, checks your filter, and if it matches, ships an event to userspace.

Coming from web dev, this is the equivalent of somebody saying “instead of adding logging to the 12 functions we think matter, let’s just instrument literally every function in the entire codebase that touches a Request object” and then actually shipping it in a way that doesn’t melt the server. It’s completely unhinged in the best way, and it works because eBPF probes are cheap enough (and the filtering happens in-kernel, before the event ever reaches userspace) that “just trace everything” is a viable strategy.

Here’s roughly what that discovery-and-attach pipeline looks like:

(A diagram is implied; the raw text only had the sentence above. No additional content was provided, so we leave it as is.)

Wait, You Can Reuse tcpdump Filters Here?

The second thing that made me feel at home: you don’t write eBPF filter logic by hand, you write a normal pcap-filter expression. The same syntax from tcpdump. The stuff every DevOps person already has muscle memory for, like tcp and port 443 or host 10.0.0.5.

pwru --output-tuple 'host 1.1.1.1 and tcp'

How does that actually work under the hood? This is the part where I had to reread the code three times. pwru takes your pcap expression, compiles it to classic BPF (cBPF, the old-school packet filter bytecode, same lineage as tcpdump -d) using cloudflare/cbpfc to translate cBPF into modern eBPF instructions, and then splices those raw instructions directly into the loaded eBPF program, at a specific marker function it planted for exactly this purpose:

for idx, inst := range program.Instructions {
    if inst.Symbol() == "filter_pcap_ebpf" + suffix {
        injectIdx = idx
        break
    }
}

In bpf/kprobe_pwru.c there’s a deliberately empty, __noinline placeholder function called filter_pcap_ebpf_l3 / _l2 whose only job is to exist as an injection point:

static __noinline bool filter_pcap_ebpf_l3(void *_skb, void *__skb,
                                            void *___skb, void *data,
                                            void *data_end)
{
    return data != data_end && _skb == __skb && __skb == ___skb;
}

pwru finds that function in the compiled instruction stream and literally performs bytecode surgery, cutting it out and stitching in your compiled filter instead, before the program is even loaded into the kernel. It’s like if your reverse proxy let you write location blocks in nginx syntax, but under the hood it was actually recompiling and hot-patching the C binary per request. Deranged. Also very cool.

The Hard Part: Knowing It’s “the Same Packet”

Okay, here’s the thing that actually made me appreciate why this tool needed to exist and isn’t just “kprobe everything and print.” A packet does not keep a fixed identity as it flows through the kernel. NAT rewrites its IP/port, so your 5-tuple filter (host 1.1.1.1) can stop matching halfway through the journey, right after the thing you were trying to debug happens.

It can get cloned (skb_clone) or copied (skb_copy) for things like local delivery plus forwarding. It can cross a veth pair into another network namespace, get handed to an XDP program, or get bridged, where the original sk_buff might legitimately get freed and a new one used to represent conceptually “the same” packet.

So pwru maintains a small state machine to keep following a packet even after your original filter would technically stop matching. Once a packet matches your filter once, its pointer gets remembered in a skb_addresses BPF hash map (bpf/kprobe_pwru.c), and every subsequent kprobe hit checks that map first, before even bothering with your filter again:

if (cfg->track_skb &&
    bpf_map_lookup_elem(&skb_addresses, &skb_addr)) {
    tracked_by = _stackid ? TRACKED_BY_STACKID : TRACKED_BY_SKB;
    goto cont;
}

When the skb gets cloned, an fexit probe on skb_clone / skb_copy propagates the “yes, we’re tracking this one” flag from the old pointer to the new one. When the kernel finally frees it (kfree_skbmem), pwru cleans up the tracking entry so the map doesn’t grow forever.

For the gnarlier case, like bridging, where the pointer identity is genuinely lost, there’s a --filter-track-skb-by-stackid mode that fingerprints by call stack instead of pointer, so it can pick the “same” packet back up under a new address. This is basically the eBPF equivalent of trying to correlate a request across microservices without a trace ID: you’re stitching identity back together from context clues because nobody handed you a clean handle to hold onto.

Turning “Packet Vanished” into an Actual Reason

The best part for someone coming from iptables-land: when a packet dies, pwru doesn’t just say “the trace stopped here, good luck.” It hooks kfree_skb_reason and decodes the kernel’s skb_drop_reason enum, so you get an actual human string like SKB_DROP_REASON_NETFILTER_DROP instead of silence.

That’s the difference between “curl hung and I have no idea why” and “curl hung, and here is the literal function and the literal drop reason, with a timestamp.” The README’s example is exactly the workflow I’d have wanted a year ago: run a curl, install an iptables DROP rule, run pwru, and watch it print the exact function where the packet gets killed, instead of you bisecting iptables -L rules by hand at 2am.

Why This Clicked for a DevOps Brain

If you’ve ever debugged distributed systems, this whole tool is basically:

• Structured, filtered logging (pcap-filter syntax you already know), applied to
• Every relevant call site automatically (via BTF introspection instead of a hand-maintained list), with
• Identity tracking across mutation and forking (NAT, clone, tunnel, bridge) so your trace doesn’t just stop the moment the packet changes shape, and
• A resolved root cause at the end (the drop reason) instead of a shrug emoji.

That’s a debugging methodology, not just a tool. It’s the “attach a debugger to the whole system, but keep it filtered so it doesn’t drown you, and correlate identity across boundaries” pattern, and it happens to be implemented with kprobes and BTF instead of trace IDs and Jaeger.

Here’s the shape of what happens end to end, from the kprobe firing to a line on your screen:

(Again, the raw text only had the sentence above; no diagram or list was provided, so we keep it as is.)

Wrapping Up

I still don’t “know the kernel.” But I don’t need to anymore, at least not for this. pwru turned “the kernel is a black box that eats packets” into “the kernel is a big, well-typed program I can attach cheap breakpoints to, filter with syntax I already know, and follow even when it tries to shapeshift on me.”

If you’re a web/devops person who’s only ever fought the network stack from the outside with iptables and tcpdump, go read bpf/kprobe_pwru.c and internal/pwru/utils.go. It reads less like kernel wizardry and more like a really well-designed observability tool that happens to live one layer lower than you’re used to.

Packet, where are you? Increasingly, I actually know how to ask.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs - without telling you. You often find out in production. git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free. Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star it on GitHub: HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit | 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी

git-lrc Free, Micro AI Code Reviews That Run on Commit

GenAI today is a race car without brakes. It accelerates fast - you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior - without telling you. You often find out in production. git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen.

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit… View on GitHub