Ingress Architecture & Low-Level Performance: Zero-Copy Pipelines, Adaptive Backpressure, and Shared-Memory Buffering
High-volume webhook ingress is an asymmetric systems problem. A typical north-south API gateway routes a synchronous request to a local service and discards its state. A webhook relay has a different job.
It must accept untrusted, bursty HTTP payloads from external producers such as Stripe, GitHub, Shopify, or Twilio, and verify their cryptographic signatures. It must also record the event somewhere safe before returning a 2xx, then dispatch the payload asynchronously to downstream consumers.
The senders’ own documentation shows why the acknowledgment path matters. GitHub expects a 2xx within 10 seconds and recommends queueing payloads for background processing. Shopify allows 1 second to connect and 5 seconds in total. Stripe’s exact timeout isn’t publicly documented, and its guidance is to verify the signature and return a 2xx before running slow logic.
A relay that stalls on its own downstream dependencies pushes that latency straight back onto the sender. At high request rates, per-request overheads that are invisible at 1,000 RPS start to dominate. These include redundant kernel-to-user memory copies, per-request heap allocation, syscall transitions, and naive 503 responses that invite retry traffic from senders that retry.
This guide covers three low-level patterns for high-throughput ingress on modern Linux (kernel 6.1 or newer; examples assume a current LTS such as 6.6 or 6.12):
- Zero-copy ingestion with Go
splice(2)andio_uring: reducing user-space copies and syscalls between the socket and storage. - Adaptive backpressure with Little’s Law and a token bucket: shaping admission during downstream degradation without provoking avoidable retries.
- RAM-constrained buffering with
tmpfsand shared memory: a memory-mapped ring buffer that survives process crashes, with explicit caveats about what it does not survive.
Throughput figures in this article are arithmetic examples, not benchmarks. Measure on your own hardware before relying on any of them.
Zero-Copy Webhook Ingress with Go splice(2) and io_uring
The Anatomy of User-Space Copying Overhead
In a conventional Go net/http handler, persisting a request body to a write-ahead log (WAL) or forwarding it to a broker involves several steps:
- NIC to kernel socket buffer: the NIC DMAs incoming packets into kernel memory, where the TCP stack assembles them into socket buffers (
sk_buff). This step involves no CPU copy. - Kernel to user space (
read(2)): the Go runtime, woken byepoll, callsread, which copies the payload into a[]byteon the Go heap. - User-space processing: the application parses headers, validates the signature, and allocates buffers.
- User space to kernel (
write(2)): the data is copied again, either into the page cache for a file or into another socket buffer for a downstream connection.
At 100,000 webhooks per second with 16 KB payloads, each CPU copy moves 1.6 GB/s, so the two copies in steps 2 and 4 move about 3.2 GB/s of memory traffic. The per-request []byte allocations also add garbage-collection work. Go’s collector is concurrent and its pauses are usually short, but allocation rate still costs CPU and can widen tail latency under load.
Using splice(2) from Go
The Linux splice(2) call moves data between a file descriptor and a pipe without copying it through user space. Go’s standard library already uses it: io.Copy between two *net.TCPConn values takes the splice path automatically.
A webhook gateway, however, rarely proxies blindly. It must read headers such as X-Hub-Signature-256 or Stripe-Signature, find the body length, and separate the body from the HTTP framing before routing it.
Two caveats apply before the code:
- What “zero-copy” means here. Splicing from a socket into a pipe moves page references. Splicing from a pipe into a regular file or
tmpfsstill copies the data once, inside the kernel, into the page cache. You save the kernel-to-user copy and the syscall round trips, not every copy. Socket-to-socket splicing can avoid the CPU copy entirely. - Protocol constraints. This approach fits HTTP/1.1 with a known
Content-Length. Chunked encoding and HTTP/2 framing require user-space parsing, and you must handle any body bytes that the HTTP parser has already buffered. In practice this means hijacking the connection (http.Hijacker) or writing a custom HTTP/1.1 front end.
The function below transfers an exact byte count from a TCP connection to a destination file descriptor through an anonymous pipe. SPLICE_F_MOVE is deliberately omitted, because it has been a no-op since Linux 2.6.21.
package ingress
import (
"fmt"
"net"
"golang.org/x/sys/unix"
)
// SplicePayload transfers exactly byteCount bytes from an incoming TCP connection
// to a destination file descriptor through a kernel pipe, avoiding user-space copies.
// dstFd should be a regular file or other descriptor that never returns EAGAIN
// on splice; for non-blocking sockets, drive the write side through rawConn.Write.
func SplicePayload(srcConn *net.TCPConn, dstFd int, byteCount int64) (int64, error) {
rawConn, err := srcConn.SyscallConn()
if err != nil {
return 0, fmt.Errorf("failed to acquire raw conn: %w", err)
}
var pipeFds [2]int
if err := unix.Pipe2(pipeFds[:], unix.O_CLOEXEC|unix.O_NONBLOCK); err != nil {
return 0, fmt.Errorf("pipe2 failed: %w", err)
}
defer unix.Close(pipeFds[0])
defer unix.Close(pipeFds[1])
// Raise pipe capacity to 1 MiB (the default unprivileged ceiling,
// /proc/sys/fs/pipe-max-size) to reduce the number of splice calls per payload.
_, _ = unix.FcntlInt(uintptr(pipeFds[1]), unix.F_SETPIPE_SZ, 1<<20)
var total int64
var spliceErr error
// Running inside rawConn.Read lets Go's netpoller manage readiness.
err = rawConn.Read(func(srcFd uintptr) bool {
for total < byteCount {
remaining := byteCount - total
// Socket -> pipe: moves page references, no copy into user space.
n, err := unix.Splice(int(srcFd), nil, pipeFds[1], nil, int(remaining), unix.SPLICE_F_NONBLOCK)
if err != nil {
if err == unix.EAGAIN {
return false // yield to the netpoller until readable
}
spliceErr = err
return true
}
if n == 0 {
spliceErr = fmt.Errorf("unexpected EOF after %d/%d bytes", total, byteCount)
return true
}
// Pipe -> destination fd.
var drained int64
for drained < n {
w, wErr := unix.Splice(pipeFds[0], nil, dstFd, nil, int(n-drained), 0)
if wErr != nil {
if wErr == unix.EINTR {
continue
}
spliceErr = wErr
return true
}
drained += w
}
total += drained
}
return true
})
if err != nil {
return total, err
}
return total, spliceErr
}
// DuplicatePipePages uses tee(2) to duplicate pipe contents into a second pipe
// without consuming the source, so one copy of the page references can be hashed
// while the other continues on to storage. Both descriptors must be pipes.
func DuplicatePipePages(srcPipeFd, dstPipeFd, length int) (int64, error) {
return unix.Tee(srcPipeFd, dstPipeFd, length, unix.SPLICE_F_NONBLOCK)
}
Verifying Signatures Without Pulling Bytes into User Space
Webhook ingress must validate an HMAC (GitHub’s X-Hub-Signature-256, Shopify’s X-Shopify-Hmac-Sha256, Stripe’s Stripe-Signature) over the exact raw body before acknowledging. If the body goes straight from socket to storage, how do you compute the digest? There are two approaches.
- Kernel crypto API (
AF_ALG) plustee(2). Linux exposes hashing throughAF_ALGsockets, includinghmac(sha256).tee(2)duplicates pipe page references into a second pipe without copying data. One pipe can be spliced into theAF_ALGoperation socket to compute the digest, while the other is spliced to storage. Some senders sign more than the body (Stripe signs a timestamp prefix followed by the payload), so feed the prefix into the hash socket first. Because the bytes reach storage before the digest is known, treat the stored frame as uncommitted until verification succeeds. - Verify from a memory-mapped buffer in user space. If verification needs user-space logic, such as multi-tenant key lookup or RFC 9421 HTTP Message Signatures, splice the body into a
tmpfs-backed file (Section 3) that the verifier has already mapped withmmap(2). The verifier hashes the mapped bytes directly, with noread(2)copy. This still reads the memory, but it avoids an extra copy and allocation.
Reducing Syscall Overhead with io_uring
Even without copies, high-concurrency ingress pays for syscall transitions, and mitigations for speculative-execution vulnerabilities can make each transition more expensive on some CPUs. io_uring lets you queue many operations through shared-memory rings and amortize or avoid those transitions.
Calling io_uring directly from Go is awkward. Go’s M:N scheduler moves goroutines between OS threads, which conflicts with ring-per-thread designs and with IORING_SETUP_SINGLE_ISSUER. A common approach is to put the network and storage data plane in a small Rust (or C) component and keep Go as the control plane. The two share state through memory-mapped rings or a narrow C ABI.
These io_uring features matter most for ingress. Kernel versions are the minimum for each:
- Multishot accept (5.19): one submission entry keeps accepting connections and posts a completion for each. This covers TCP accept only. TLS handshakes are a separate matter.
- Provided buffer rings (
IORING_REGISTER_PBUF_RING, 5.19): the application registers a ring of buffers and the kernel picks one for each receive (IOSQE_BUFFER_SELECT), returning the buffer ID in the completion. This supersedes the olderIORING_OP_PROVIDE_BUFFERSmechanism. - Multishot receive (6.0): one receive request keeps producing completions as data arrives, each consuming a buffer from the ring.
- Submission queue polling (
IORING_SETUP_SQPOLL): a kernel thread polls the submission queue so the application can submit work without a syscall while the thread is awake. The polling thread consumes a CPU core, and it sleeps after an idle timeout, after which submission needs a wakeup call. IORING_SETUP_SINGLE_ISSUERandIORING_SETUP_DEFER_TASKRUN(6.0 and 6.1): these reduce overhead for a ring driven by a single thread.
// Rust io_uring ingress ring: multishot receive with provided buffers.
// The buffer ring itself must be registered beforehand (IORING_REGISTER_PBUF_RING;
// registration code not shown) under buffer group ID `bgid`.
use io_uring::{opcode, types, IoUring};
use std::os::unix::io::RawFd;
pub struct IngressRing {
ring: IoUring,
bgid: u16, // buffer group ID for kernel-managed buffer selection
}
impl IngressRing {
pub fn new(entries: u32, bgid: u16) -> std::io::Result<Self> {
let ring = IoUring::builder()
.setup_sqpoll(1000) // SQ thread idles out after 1000 ms without submissions.
.build(entries)?;
Ok(Self { ring, bgid })
}
/// Arms a multishot receive on an accepted webhook sender socket.
/// The kernel picks buffers from the registered ring for each completion.
pub unsafe fn arm_multishot_recv(&mut self, client_fd: RawFd, conn_token: u64) -> std::io::Result<()> {
let recv_e = opcode::RecvMulti::new(types::Fd(client_fd), self.bgid)
.build()
.user_data(conn_token);
self.ring.submission().push(&recv_e).map_err(|_| std::io::Error::new(std::io::ErrorKind::Other, "SQ full"))?;
// With SQPOLL, submit() only needs a wakeup syscall if the SQ thread is asleep.
self.ring.submit()?;
Ok(())
}
}
SINGLE_ISSUER is omitted from this snippet because its rules about which thread may submit are easy to violate in a hybrid Go/Rust layout. Add it only once ownership of the ring is unambiguous.
Truly Zero-Copy Receive: io_uring zcrx
Provided buffers keep data out of the Go heap, but the kernel still copies each packet payload into your buffer. Linux 6.15 added io_uring zero-copy receive (zcrx), which removes that kernel-to-user copy so payloads land directly in application memory.
It has demanding prerequisites: the NIC must support header/data split, and you must configure flow steering and RSS so that only the intended flows reach a hardware queue bound to the io_uring instance. The NIC is not configured by the kernel API, so this setup is done out of band. It also requires SINGLE_ISSUER, DEFER_TASKRUN, and 32-byte completion entries. Treat zcrx as a specialized option for dedicated ingress hosts, not a default.
Architecture Comparison
Metric Go net/http (epoll) Go RawConn + splice(2) Rust io_uring (provided buffers, optional SQPOLL) CPU copies, socket to storage file 2 (kernel → user, user → page cache) 1 (pipe pages → page cache, inside the kernel) 2 with a plain receive plus write; the receive copy disappears only with zcrx hardware support Syscalls per webhook Several (epoll_wait, read, write, per chunk) A few splice calls per chunk Amortized toward zero with batching and SQPOLL Go heap pressure Per-request []byte allocations Low (metadata only) None for payloads; data plane lives outside the Go heap TLS crypto/tls in user space Needs kTLS to splice plaintext Needs kTLS or user-space TLS in the data plane Minimum kernel Any 2.6.17 for splice; kTLS TX 4.13, RX 4.17 6.0 for multishot receive, 5.19 for buffer rings; 6.15 for zcrx Production note on TLS. Webhook traffic is HTTPS, and user-space TLS (crypto/tls or OpenSSL) decrypts in user space, which defeats splice.
Kernel TLS (kTLS) moves record encryption and decryption into the kernel after a user-space handshake. Once active, splice and io_uring see plaintext. Decryption is still CPU work, so it is not “free,” and Go’s crypto/tls does not natively support kTLS, so you would need a TLS library that does (OpenSSL 3.x can). Many deployments terminate TLS at a load balancer or proxy in front of the ingress, which sidesteps the issue.
Adaptive Backpressure with a Token Bucket and Little’s Law
Why Naive Throttling Can Trigger Retry Storms
When a downstr
Comments
No comments yet. Start the discussion.