The Anatomy of an IP Ban: TCP Fingerprints, Passive OS Fingerprinting, and MTU Signatures

The Anatomy of an IP Ban: TCP Fingerprints, Passive OS Fingerprinting, and MTU Signatures Core Problem Statement & Invalidation of Traditional Approaches The security posture of web-scale infrastructure historically relied on IP-based rate limiting and reputation blocklists operating at Layer 3 (L3) and Layer 4 (L4) of the OSI model. When request volumes exceeded threshold allocations, edge firewalls appended source IPv4 addresses to dynamic access control lists (ACLs) or null-routed prefixes via Border Gateway Protocol (BGP) Flowspec (RFC 5575). However, modern automated scraping fleets, distributed residential proxy networks, and IPv6 address block allocation models have rendered static IP blocking fundamentally obsolete. A single adversary operating a cluster orchestrator can distribute outbound connections across millions of distinct, residential IP addresses allocated via Carrier-Grade NAT (CGNAT) or mobile LTE/5G subnets. [Attacker Node] ---> [Residential Proxy Pool (100k+ IPs)] ---> [Target WAF / Edge Firewall] │ │ Identical Kernel Engine Inspects L3/L4 Wire Signals (e.g., Linux 5.15 / Headless) (Initial Window, Options) In this regime, traditional blocklists exhibit two catastrophic failure modes: - High Rate of False Positives: Banning a CGNAT gateway IP ($/32$ IPv4) disallows thousands of legitimate residential mobile subscribers sharing the same egress NAT interface. - Asymmetric Economic Deficit: Acquiring $10^5$ rotating IP addresses via proxy providers costs orders of magnitude less than the compute capacity expended by target engineering teams attempting to perform stateful tracking with api rate limiting internals token bucket vs leaky bucket vs sliding window counter systems. To bypass Layer 7 (L7) Web Application Firewalls (WAFs), scraping automation frameworks employ headless browsers (Chromium, Firefox) modified to forge user agents, screen dimensions, WebGL vendor strings, and TLS client hellos. While adversaries routinely synthesize L7 headers and align application layer configurations, they frequently execute these client binaries inside virtualized Linux cloud environments (e.g., AWS EC2, GCP, Docker containers). This decoupling between the claimed operating system at Layer 7 (e.g., User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ) and the underlying kernel network stack executing the TCP/IP three-way handshake generates a protocol-level mismatch. +-------------------------------------------------------------------------------+ | UNRESOLVED PROTOCOL MISMATCH | +-------------------------------------------------------------------------------+ | Layer 7 (HTTP): User-Agent: "Macintosh; Intel Mac OS X 10_15_7" | | Accept-Language: "en-US,en;q=0.9" | +-------------------------------------------------------------------------------+ | Layer 4 (TCP): Window Size: 29200 | MSS: 1460 | WScale: 7 | | Options Sequence: [MSS, SACK_PERM, TS, NOP, WSCALE] | | Kernel Stack: Native Linux 5.x/6.x (NOT macOS Kernel) | +-------------------------------------------------------------------------------+ | RESULT: PASSIVE OS DETECTION TRIGGERED -> ANOMALOUS DENSITY BLOCK | +-------------------------------------------------------------------------------+ Edge inspection systems perform Passive OS Fingerprinting (p0f) during the initial TCP SYN packet exchange. Because TCP/IP stack behavior is implemented deep within the OS kernel, spoofing network-layer parameters requires raw socket privileges (CAP_NET_RAW ), custom eBPF packet generation engines, or non-trivial guest kernel modifications. Analyzing the low-level wire parameters of incoming SYN frames-such as TCP Option Ordering, Initial Window Size, Time To Live (TTL) hop distances, and Maximum Segment Size (MSS) constraints-allows edge firewalls to classify incoming requests long before L7 HTTP payload decryption or TLS negotiation completes. Theoretical Mechanics & Protocol State Machine Passive OS Fingerprinting works by capturing static and dynamic fields embedded within the IP and TCP headers during the initial TCP SYN transmission. While RFC 793 and subsequent RFCs define the protocol semantics for TCP, implementation details-such as option sequencing, default buffer allocations, and window scaling coefficients-are strictly left to kernel implementers. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Destination Port | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Acknowledgment Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data | Reserved |N|C|E|U|A|P|R|S|F| Initial Window | | Offset| |N|W|C|R|G|C|K|S|Y|N| Size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Checksum | Urgent Pointer | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Kind = 2 | Length = 4 | Maximum Segment Size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Kind = 4 | Length = 2 | Kind = 8 | Length = 10 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TS Value (TSval) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TS Echo Reply (TSecr) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Kind = 1 | Kind = 3 | Length = 3 | Shift WSCALE | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Protocol Wire Parameters for OS Disambiguation - Initial Time To Live (TTL): The IP header TTL field prevents packet looping. Kernels assign distinct initial TTL values: - Linux kernel 2.x - 6.x: $\text{Initial TTL} = 64$ - Microsoft Windows (NT, 10, 11, Server): $\text{Initial TTL} = 128$ - Apple XNU (macOS, iOS): $\text{Initial TTL} = 64$ - FreeBSD / OpenBSD: $\text{Initial TTL} = 64 \text{ or } 255$ - Solaris / SmartOS: $\text{Initial TTL} = 255$ Because routers decrement TTL by $1$ at each hop, the observed TTL ($\text{TTL}{\text{obs}}$) reveals the distance vector $\Delta{\text{hops}} = \text{TTL}{\text{init}} - \text{TTL}{\text{obs}}$. The original initial TTL is derived by rounding $\text{TTL}_{\text{obs}}$ up to the nearest power boundary in the set ${32, 64, 128, 255}$. - TCP Initial Window Size ($W_0$): The initial receive window advertised in the SYN packet reflects default buffer allocation policies:- Linux 5.x/6.x: Often sets $W_0 = 29200$ or $64240$ (frequently calculated as an exact multiple of MSS, such as $10 \times \text{MSS}$ or $44 \times \text{MSS}$). - Windows 10/11: Frequently uses $W_0 = 64512$ or dynamic scaling base values ($8 \times \text{MSS}$). - macOS / iOS: Standard default $W_0 = 65535$. - TCP Options Ordering and Composition: RFC 793 allows TCP Options to appear in any arbitrary order within the option bytes padding boundary (up to 40 bytes). Operating system kernels standardize this order during compilation, creating unique option signatures:- Linux Kernel (Standard Modern Layout): $$\text{Options} = [\text{MSS}, \text{SACK_PERMITTED}, \text{TIMESTAMP}, \text{NOP}, \text{WINDOW_SCALE}]$$ - Microsoft Windows 10/11 Layout: $$\text{Options} = [\text{MSS}, \text{NOP}, \text{WINDOW_SCALE}, \text{NOP}, \text{NOP}, \text{SACK_PERMITTED}]$$ - Apple macOS XNU Layout: $$\text{Options} = [\text{MSS}, \text{NOP}, \text{WINDOW_SCALE}, \text{NOP}, \text{NOP}, \text{TIMESTAMP}, \text{SACK_PERMITTED}, \text{END_OF_OPTION_LIST}]$$ - Maximum Segment Size (MSS) vs. MTU Signatures: The MSS option indicates the largest contiguous payload block (in bytes) a node can process. The observed MSS directly maps to the link layer Maximum Transmission Unit (MTU): $$\text{MSS} = \text{MTU} - \text{Header}{\text{IP}} (20\text{B}) - \text{Header}{\text{TCP}} (20\text{B}) = \text{MTU} - 40$$- Standard Ethernet MTU ($1500\text{ bytes}$) $\implies \text{MSS} = 1460\text{ bytes}$. - PPPoE Interfaces ($1492\text{ bytes}$) $\implies \text{MSS} = 1452\text{ bytes}$. - Google Cloud / AWS Overlay VPC Networks ($1460\text{ bytes}$) $\implies \text{MSS} = 1420\text{ bytes}$. - WireGuard VPN Tunnels ($1420\text{ bytes}$) $\implies \text{MSS} = 1380\text{ bytes}$. When a client claims to be a macOS browser operating on a residential Comcast connection ($\text{MSS} = 1460$), but transmits a SYN packet with $\text{MSS} = 1380$, an option order matching Linux, and an initial window of $29200$, the network edge flags the flow as a virtualized Linux scraping instance routed through an encrypted tunnel. PASSIVE OS FINGERPRINT EVALUATION STATE MACHINE +---------------------------------------------------------+ | INGRESS FRAME | | (eBPF / XDP Hook) | +---------------------------------------------------------+ │ ▼ +---------------------------------------------------------+ | Parse L3 IP / L4 TCP Headers | | Extract: TTL, Initial Window Size, IP Flags (DF bit) | +---------------------------------------------------------+ │ ▼ +---------------------------------------------------------+ | Parse TCP Options Vector | | Extract: Sequence [MSS, SACK, TS, WSCALE, NOP, EOL] | | Compute: Option Kind Hash & Window Scale Exponent | +---------------------------------------------------------+ │ ▼ +---------------------------------------------------------+ | Infer Distance Vector | | TTL_init = MinPowerOf2Above(TTL_obs) | | Hops = TTL_init - TTL_obs | +---------------------------------------------------------+ │ ▼ +---------------------------------------------------------+ | Query p0f Database Trie Structure | | Lookup Signature Key: ver:TTL:olayout:wsize:quirks | +---------------------------------------------------------+ │ │ Match Found No Match / Anomaly │ │ ▼ ▼ +-----------------------+ +-----------------------+ | Assigned Kernel OS | | Flag Raw Network OS | | Identity: e.g., | | Signature as Unknown | | Linux 5.x / 6.x | | or Synthetic Stack | +-----------------------+ +-----------------------+ │ │ +------------------+------------------+ │ ▼ +-----