Massive prompt of disappointment

📝 Markdown Rendered

retoor · Level 1844 · 1h ago

This is a well-prepared prompt for creating an HTTP proxy communication from scratch, zero copy. It should be the fastest thing in the world, especially since I had a document with all caveats, also regarding performance.

The output code is Swift, very clean, very clear. I need to spend some more effort on it. Vibe coding made me very... lazy. But I think that my preparations at least have taken some time for.

theplan.md - molohttp Proxy Data-Plane Rewrite on Raw SwiftNIO

Single-shot execution plan to replace only the proxy data plane (listeners → host routing → HTTP forwarding → WebSocket forwarding) with a hand-written, streaming, zero-copy, multithreaded-plus-async SwiftNIO implementation. Hummingbird is retained only as the control-plane responder for /molohttp/* (admin, dashboard, stats, docs, health); the ACME subsystem is untouched. Every RFC edge case and production pitfall in the brief is tackled by a numbered rule in Parts B-F.

This document is written to be executed in one uninterrupted pass producing a complete, working product - not a prototype. Read Part A for the shape and order; Parts B-F are the normative spec each source file is coded against. The functional behavior contract of the current system is in funcs.md (companion document); every "current behavior" reference below resolves there.

A0. Verified ground truth

Toolchain: Swift 6.0.3 (/opt/swift-6.0.3-RELEASE-ubuntu24.04).
Pins (from Package.resolved, verified in .build/checkouts/): swift-nio 2.97.1 (NIOHTTP1 on vendored llhttp 9.3.0, all lenient flags off), swift-nio-ssl 2.36.1 (BoringSSL, server session cache + 2 tickets on by default), swift-nio-extras 1.26.0 (HTTPResponseCompressor, ServerQuiescingHelper), async-http-client 1.33.1, hummingbird 1.12.2 / core 1.6.1 / websocket 1.2.0.
NIOWebSocket, NIOAsyncChannel, typed upgraders, HTTPRequestDecoder(leftOverBytesStrategy: .forwardBytes) all present at 2.97.1.
The Hummingbird hand-off seam is public (HBHTTPResponder, HBHTTPRequest/Response, HBRequestBody, HBByteBufferStreamer, constructResponder()); only the internal HBHTTPServerHandler assembly loop must be reimplemented (~120 lines; reference readable in the checkout). HTTPSForwardingResponder already does exactly this and is reused as the seam responder.
Production: systemd molohttp.service on molodetz.nl, 4 vCPU / 8 GB, 92 domains, ports 80/443, TimeoutStopSec=10, LimitNOFILE=1048576.

A1. Architecture in one paragraph

One MultiThreadedEventLoopGroup(System.coreCount) (the data-plane ELG) owns both raw ServerBootstrap listeners (:80 plain, :443 TLS reusing SNICertProvider.makeTLSConfiguration verbatim) and every upstream connection. Each accepted client channel runs a small pipeline (HTTPRequestDecoder(.forwardBytes) + HTTPResponseEncoder + HTTPServerPipelineHandler + HTTPResponseCompressor + IdleStateHandler + one duplex FrontendHTTPHandler). FrontendHTTPHandler classifies each request head once: internal /molohttp/*, /.well-known/acme-challenge/*, and /favicon.ico are handed to the retained Hummingbird responder (constructResponder() snapshot, buffered bodies); everything else is host-routed via the unchanged LiveHostIndex, gated by the existing per-site auth + rate limit, then streamed to an upstream connection dialed on the client channel's own event loop - so client and upstream legs share one loop and body relay is lock-free channelRead → write with writability-driven backpressure (≈128 KiB in-flight ceiling per connection, replacing today's full-body buffering). Proxied WebSocket upgrades are relayed as a raw 101 tunnel (byte splice after the handshake, native close-code/backpressure semantics) instead of terminate-and-redial. The three internal WS feeds (stats/benchmark/admin-events) upgrade on the same pipeline via NIOWebSocketServerUpgrader → HBWebSocket, keeping the broadcasters unmodified. Shared state stays exactly where the current code proved it works: NIOLock snapshots for routing/TLS, actors for config/cert coordination, NIOThreadPool for file/bcrypt/archive I/O, plus per-upstream ManagedAtomic circuit-breaker state.

A2. Scope boundaries (do not cross)

Rewritten (new Sources/MoloHTTP/DataPlane/ module): listeners, TLS front, request classification, host routing façade, streaming HTTP forward path, upstream pool + selector + breaker, WebSocket tunnel, the control-plane hand-off bridge.

Retained verbatim: HostIndex.swift/LiveHostIndex, SNICertProvider.swift + SNICertCache, WebSocketProxyRegistry.swift, CompressionSupport.swift, all Config/, Admin/, Stats/, Middleware/ (as the control-plane middleware chain behind the seam), Models/, Utilities/, FileServerHandler.swift (invoked from the new HostRouter for .fileServer sites), and the entire ACME/ subsystem.

Re-wrapped only: AcmeChallengeHandler (imports Hummingbird; the ~15-line token-sanitize + off-loop file-read + application/octet-stream response reimplemented against the new request/response types, still host-agnostic on plain HTTP - see funcs.md Part 6 §7.3).

Retired: MoloHTTPServer.swift listener wiring (the actor stays as lifecycle owner, now starting DataPlaneServer + the port-less control-plane app), TLS/HTTPSServer.swift, TLS/HTTPSForwardingResponder.swift semantics fold into ControlPlaneBridge (reuse the class), Stats/StatsWebSocketSetup.swift (→ InternalWSUpgrade), Handlers/WebSocketProxyHandler.swift, and ReverseProxyHandler's AsyncHTTPClient machinery (breaker/selector semantics migrate into UpstreamHealth/UpstreamSelector).

A3. Deliberate behavior changes (all others preserved byte-for-byte)

These supersede specific funcs.md quirks and MUST be recorded in the changelog. Everything not listed here is preserved exactly (host precedence, XFF append semantics, 502-on-all-cooling, 5xx-relayed-but-breaker-counted, breaker 5/30 s, 401 without WWW-Authenticate, access-log line format, compression predicate, config schema, all admin/stats/ACME behavior).

504 for upstream connect/head timeouts (was 502) - Part E ER-3.
Whole-response 120 s deadline removed, replaced by idle-based read timeouts (long downloads/SSE now work) - Part E TO-4.
Errors and client aborts are access-logged (499 disposition) - Part E OB-3/OB-4.
Upstream-failure warn logs sampled (≤1/5 s per upstream + (suppressed N)) - Part E OB-7.
WS idle ping/close every 30 s (config websocket_ping_interval_seconds, 0 = legacy) - Part E TO-8.
Connection-nominated hop-by-hop headers stripped (not just the fixed list) - Part D R3.1 / Part E SEC-1.
Proxied WS close codes/extensions propagate end-to-end; WS denial becomes a plain 403/426 instead of upgrade-then-1008 - Part D R7.1.
Real efficiency dashboard counters (were hardcoded 0) - Part E OB-6.
Streaming replaces full buffering both directions (request max_request_body_bytes still enforced by counting streamed bytes) - Part C / Part D R8.3.
content-encoding no longer stripped from identity upstream responses (they stream unmodified) - Part D R3.5.

Config additions (all optional, defaults preserve current behavior): retry_connect_failures (false), websocket_ping_interval_seconds (0), max_connections_per_ip (0), listen[].reuse_port (false), forward_via_header (false), and an HTTPS listen port made configurable so the canary can test TLS. GlobalConfig decode already tolerates unknown/absent optional keys.

A4. Execution phases (single pass, in order)

Each phase ends at a green make verify (zero warnings) + make test. Do not proceed past a red gate. The build stays compilable throughout because the new module is added alongside the old path and the switch to it is the last wiring step.

Phase 0 - Scaffolding and dependencies

Add swift-atomics as a direct dependency (already transitive) for ManagedAtomic breaker state; add the DataPlane target directory to Package.swift (same target, new folder - no new SPM product).
Add config fields from A3 to GlobalConfig/SiteConfig models with compatibility defaults; extend ConfigValidator only where a new field needs validation (e.g. max_connections_per_ip >= 0). Existing test SiteModelCodableTests/ConfigPipelineTests must stay green (add round-trip assertions for the new fields).
Gate: make verify && make test.

Phase 1 - Piping primitives (pure NIO, unit-testable in isolation)

GlueHandler.swift (Part C C1.3 state machine, retain-cycle break in channelInactive/handlerRemoved), LoopUpstreamPool.swift + UpstreamPool.swift + UpstreamConnection.swift (Part B B5.2, Part C C3), UpstreamSelector.swift + UpstreamHealth.swift (breaker atomics, Part E CB-1..5).
Tests: EmbeddedChannel/NIOAsyncTestingChannel unit tests for glue backpressure, pool checkout/checkin/idle-eviction/stale-redial, selector policies, breaker open/close with an injected clock. (Cases 16, 18, 27 from Part F plus pool-specific cases.)
Gate: make verify && make test.

Phase 2 - Streaming HTTP forward path

ProxyRequestBuilder.swift (head transform: hop-by-hop strip incl. Connection-nominated, XFF/XRI/XFP/XFH, header_up, host rewrite, header-value CRLF validation - Part D R3, Part E SEC-3), ProxyResponseRelay.swift (response head transform, framing rules Part D R2, breaker recording, 502/504 mapping, streamed byte counting), and the proxy branch of FrontendHTTPHandler.swift (request-part state machine, body cap counting, 100-continue local termination R5, HEAD/1xx/204/304 no-body traps R2.9-R2.12, automaticallySetFramingHeaders = false strategy).
HostRouter.swift ports RouterBuilder.DispatchContext: stripHostPort → LiveHostIndex.lookup → auth (bcrypt off-loop via read-suspend, Part E SEC-11) → rate limit (actor await, SEC-12) → selection → connect. .fileServer sites call the retained FileServerHandler through the buffered response path.
Tests: Part F cases 1-19, 22-28 against the in-process TestUpstream; the 100 MB constant-memory test (case 10) is the streaming proof.
Gate: make verify && make test.

Phase 3 - Listeners, TLS, control-plane seam

Listener.swift (ServerBootstrap: backlog 4096, SO_REUSEADDR, TCP_NODELAY, allowRemoteHalfClosure, watermarks, optional per-loop SO_REUSEPORT behind the flag), TLSFrontend.swift (NIOSSLServerHandler from SNICertProvider.makeTLSConfiguration, ALPN ["http/1.1"] only per Part D R10.2, lazy-443-on-first-cert), ControlPlaneBridge.swift (reimplemented HB part-assembly loop + HTTPSForwardingResponder-style respond, 413 cap, x-forwarded-proto per listener), the classification branch of FrontendHTTPHandler + FrontendClassifier.swift, and DataPlaneServer.swift (owns the ELG, both listeners, quiescing).
Re-wrap AcmeChallengeHandler for the new types; register the ACME challenge + favicon + internal /molohttp/* routes to the bridge with correct precedence (Part F risk #5/#6).
Gate: make verify && make test + a manual boot against a local config serving one file-server site and one proxy site over both HTTP and HTTPS.

Phase 4 - WebSocket tunnel + internal WS feeds

WSTunnel.swift (raw 101 relay orchestration, leftover-bytes flush, pipeline-strip choreography - Part B B3.3, Part D R7.1) and InternalWSUpgrade.swift (NIOWebSocketServerUpgrade for the three internal paths → HBWebSocket → existing broadcasters, replacing StatsWebSocketSetup).
Tests: Part F cases 20-21; CT-15/16.
Gate: make verify && make test.

Phase 5 - Wiring, shutdown, observability

MoloHTTPServer actor rewired: construct DataPlaneServer + the port-less control-plane HBApplication (.shared(dataPlaneELG)), keep the exact startup order from funcs.md Part 6 §1.2 (certs loaded → SNI cache populated → HTTPS only if hasCerts → renewal.start → reachability polling), wire onSitesChanged to rebuild LiveHostIndex + WebSocketProxyRegistry + the immutable breaker map, SIGHUP reload, and graceful shutdown via ServerQuiescingHelper (Part E GS-1..4, including the acmeClient.shutdown() fix).
Observability: StatsMonitor.record from the data plane with streamed byte counts (Part E OB-5), real efficiency counters (OB-6), access log at completion incl. errors/499 (OB-2/3/4), sampled upstream warnings (OB-7).
Gate: make verify && make test.

Phase 6 - Protocol/chaos harness, load, docs

scripts/proxytest/proxytest.py (Part F F1.2 suites), scripts/smoke_domains.py, scripts/soak.sh; Makefile proxytest + soak targets.
Run the protocol suite + chaos cases (Part E CT-1..22), the wrk tiers, and a 30-60 min soak; meet the Part F F1.3 thresholds.
Update README.md, CLAUDE.md (Architecture, Project Structure, Changelog with the A3 behavior changes), AGENTS.md; bump MoloHTTPPaths.version to 0.3.0.
Gate: full green + thresholds met.

Phase 7 - Rollout

Part F F4: build + molohttp.prev backup → canary on 8080/8443 against real config → smoke_domains.py + proxytest + wrk on the canary → diff vs prod → systemd cutover → post-cutover checklist (admin UI, live WS, SIGHUP, ACME path) → rollback ready.

A5. Decision ledger (resolved; no open questions block execution)

#	Decision	Rationale (full detail in the cited part)
1	Raw `ServerBootstrap` + NIOHTTP1 pipeline; Hummingbird only behind the control-plane seam	mirrors verified HB-core pipeline semantics with zero framework in the forwarding path (Part B B2-B3)
2	Hand-rolled per-loop upstream pool; AsyncHTTPClient rejected for the data plane (loop-affinity not guaranteed, per-chunk async cost, owns headers a transparent proxy must control, can't relay WS), kept for ACME/benchmark	Part B B5.1
3	Client + upstream legs on one event loop; lock-free `channelRead → write` piping with writability backpressure	Part C C1 - the performance backbone
4	Proxied WS = raw 101 tunnel; internal WS = terminate via `NIOWebSocketServerUpgrader`	zero-copy, native semantics; unchanged broadcasters (Part B B3.3)
5	Breaker = per-upstream `ManagedAtomic` published in an immutable snapshot	preserves exact 5/30 s semantics lock-free (Part E CB)
6	Timeouts move to idle-based (drop the whole-response deadline); connect/head timeout → 504	correctness for streams; the only status-code change (Part E ER-3/TO-4)
7	`automaticallySetFramingHeaders = false` on both encoders, explicit framing	avoids the HEAD/POST chunked-injection traps (Part D R2.4/R2.10)
8	ALPN `["http/1.1"]` only; HTTP/1.1-only both legs this phase	Part D R10 - h2 is a later, isolated phase
9	SO_REUSEPORT, per-IP cap, retries, Via, WS ping: config flags, defaults preserve current behavior	opt-in improvements, byte-compatible by default (Part A3)
10	Streaming replaces buffering; request body cap enforced by counting streamed bytes	removes the 10 MB request buffer + unbounded response buffer (Part C, Part D R8.3)

Three items were flagged during research as design-affecting and are resolved here: (a) HTTPS port made configurable (so the canary tests TLS) - done in Phase 0 config additions; (b) file-server ETag stays process-seed for now (out of scope - file server is retained verbatim; a deterministic-hash change is a separate follow-up, noted not done); (c) compression pipeline placement - the HTTPResponseCompressor sits between ProxyResponseRelay's write and the HTTPResponseEncoder, receiving the relayed .head before body chunks stream, and its existing predicate (skip if content-encoding present) prevents double-compression of precompressed upstreams (Part F risk #3, verified as the pipeline order in Phase 3).

A6. Definition of done

make verify zero warnings, make test (existing 50 + ~28 new) green, make proxytest all suites green.
100 MB proxied transfer completes with < 64 MB RSS delta (streaming proven).
Load thresholds met (Part F F1.3): health ≥ 13k RPS, proxied ≥ baseline, memory ≤ 180 M stable, zero crashes; 30-60 min soak flat RSS + FD plateau.
All 92 production domains return identical status + compatible headers between the canary and current prod (Part F F2 checklist).
Admin UI, live dashboard WS, SIGHUP reload, ACME challenge path, and cert renewal verified post-cutover.
molohttp.prev retained until the first post-cutover soak passes.

The normative specification follows: Part B (architecture), Part C (performance/zero-copy), Part D (HTTP protocol correctness), Part E (reliability/security/observability), Part F (testing/rollout). Each source file listed in Part B B6 is coded against the rules in Parts C-E and verified by the tests in Part F.

Part B - Data-Plane Architecture (raw SwiftNIO)

All facts verified against Package.resolved and .build/checkouts/.

B1. Pinned versions and available APIs

Package	Pinned	Relevant modules
swift-nio	2.97.1	NIOCore, NIOPosix, NIOHTTP1, NIOWebSocket, NIOEmbedded, NIOConcurrencyHelpers
swift-nio-ssl	2.36.1	NIOSSL (`sslContextCallback` already used by `SNICertProvider`)
swift-nio-extras	1.26.0	NIOExtras (`ServerQuiescingHelper`), NIOHTTPCompression (`HTTPResponseCompressor`)
swift-nio-http2	1.42.0	transitive only (Hummingbird); not needed by data plane
async-http-client	1.33.1	AsyncHTTPClient (streaming async API present)
hummingbird	1.12.2	Hummingbird
hummingbird-core	1.6.1	HummingbirdCore, HummingbirdTLS
hummingbird-websocket	1.2.0	HummingbirdWSCore, HummingbirdWSClient
compress-nio	1.4.2	transitive

Toolchain: Swift 6.0.3.

Verified NIO API availability at 2.97.1:

NIOAsyncChannel, NIOTypedHTTPServerProtocolUpgrader, configureUpgradableHTTPServerPipeline - present (deliberately NOT used on the forwarding hot path; may be used in tests).
HTTPRequestDecoder(leftOverBytesStrategy:) with .forwardBytes | .fireError | .dropBytes - .forwardBytes needed for tunnel upgrades.
NIOWebSocket module ships inside swift-nio (NIOWebSocketServerUpgrader, frame decoder/encoder, aggregator). No websocket-kit anywhere. HummingbirdWSCore's addWebSocketUpgrade is a thin wrapper over NIOWebSocketServerUpgrader.
SO_REUSEPORT: no NIOCore convenience static, but NIOBSDSocket.Option(rawValue: SO_REUSEPORT) works on Linux via ChannelOptions.socketOption.
AsyncHTTPClient 1.33.1 has full streaming (HTTPClientResponse.Body: AsyncSequence of ByteBuffer, backpressured) - so "AHC can't stream" is not a valid rejection; the rejection rationale is different (B5.1).

B2. The Hummingbird hand-off seam (verified against 1.12.2 / core 1.6.1)

B2.1 `constructResponder()`

HBApplication.constructResponder() returns router.buildRouter() - an HBResponder with per-route responders already wrapped in the full middleware chain (SecurityHeaders → CORS → MoloHTTPMiddleware → Stats → AccessLog → trie router) plus a middleware-wrapped NotFoundResponder fallback. One call snapshots everything. The existing HTTPSForwardingResponder already exploits this.

B2.2 Seam types (all public)

public protocol HBHTTPResponder {
    func respond(to request: HBHTTPRequest, context: ChannelHandlerContext,
                 onComplete: @escaping (Result<HBHTTPResponse, Error>) -> Void)
    var logger: Logger { get }
}
public struct HBHTTPRequest: Sendable  { var head: HTTPRequestHead; var body: HBRequestBody }
public struct HBHTTPResponse: Sendable { var head: HTTPResponseHead; var body: HBResponseBody }
public enum HBRequestBody: Sendable  { case byteBuffer(ByteBuffer?); case stream(HBByteBufferStreamer) }
public enum HBResponseBody: Sendable { case byteBuffer(ByteBuffer); case stream(HBResponseBodyStreamer); case empty }

Critical constraint: HummingbirdCore's internal HBHTTPServerHandler (assembles HTTPServerRequestPart → HBHTTPRequest, writes HBHTTPResponse back) is not public - the data plane must reimplement its ~120-line assembly/write-back loop (reference readable at hummingbird-core/.../HTTPServerHandler.swift). Everything else at the seam is public.

B2.3 Hand-off classification

FrontendHTTPHandler classifies each request head once on .head:

MoloHTTPPaths.isInternal(path) (/molohttp or /molohttp/…) → control plane
path.hasPrefix("/.well-known/acme-challenge/") → control plane (ACME untouched)
path == "/favicon.ico" → control plane
everything else → data plane (host routing → proxy/file)

Control-plane bodies are fed buffered (.byteBuffer); no internal handler streams. Bodies over max_request_body_bytes on internal paths → 413 before hand-off. The HBApplication no longer binds a public port - it becomes a responder factory + lifecycle container (bind on 127.0.0.1:<admin_port> or none; the data plane owns 80/443). The three internal WS endpoints move to the data-plane upgrade path (B4.3).

B3. Data-plane channel pipelines

B3.1 HTTP :80

ServerBootstrap(group: dataPlaneELG)
    .serverChannelOption(ChannelOptions.backlog, value: 4096)
    .serverChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
    .childChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
    .childChannelOption(ChannelOptions.tcpOption(.tcp_nodelay), value: 1)
    .childChannelOption(ChannelOptions.allowRemoteHalfClosure, value: true)
    .childChannelInitializer { channel in
        channel.pipeline.addHandlers([
            ByteToMessageHandler(HTTPRequestDecoder(leftOverBytesStrategy: .forwardBytes)),
            HTTPResponseEncoder(),
            HTTPServerPipelineHandler(),
            HTTPResponseCompressor(initialByteBufferCapacity: 32768,
                                   responseCompressionPredicate: CompressionSupport.predicate(for: cfg)),
            IdleStateHandler(readTimeout: idle, writeTimeout: idle),
            FrontendHTTPHandler(router: hostRouter, pool: upstreamPool,
                                controlResponder: controlBridge, secure: false)
        ])
    }

.forwardBytes mandatory: on tunnel upgrade the decoder is stripped and any bytes the client sent after its handshake re-fire as raw ByteBuffer.
HTTPServerPipelineHandler preserves enableHttpPipelining: true; removed with the codec on tunnel upgrade.
IdleStateHandler → FrontendHTTPHandler closes on the event; disarmed once the connection becomes a WS tunnel.
Server: molohttp, Date (per-loop cached RFC1123), connection: keep-alive/close written by FrontendHTTPHandler on every response.
Request body cap: count streamed request bytes; > max_request_body_bytes → 413 + close. The proxy path can now stream request bodies upstream instead of buffering 10 MB.

B3.2 HTTPS :443

Identical pipeline with NIOSSLServerHandler(context:) prepended at .first. SNICertProvider.makeTLSConfiguration(cache:) reused verbatim (its sslContextCallback does per-handshake SNI lookup against the NIOLock-guarded SNICertCache; runtime installs keep working). The HTTPS FrontendHTTPHandler is secure: true, which only adds replaceOrAdd("x-forwarded-proto", "https") on the inbound head before dispatch. Improvement worth taking: allow 443 to bind lazily on first cert install (listeners are now cheap raw bootstraps).

B3.3 WebSocket upgrade - decision: raw tunnel after relayed 101

For proxied upgrades, replace the current terminate-and-redial bridge with a raw byte tunnel:

	Terminate-and-bridge (current)	Raw tunnel after 101 (chosen)
Copies	decode → reassemble → re-encode both directions	zero re-framing; ByteBuffer in, ByteBuffer out
Frame size	1 MiB cap, reassembled	unlimited; fragmentation passes through
Close codes	collapsed to 1001	propagate natively end-to-end
Subprotocols/extensions	only `sec-websocket-protocol` forwarded; permessage-deflate broken	negotiated end-to-end verbatim
Backpressure	none (unbounded pending buffer)	native TCP backpressure via glue
Failure before 101	client already upgraded; only WS close 1011	upstream's non-101 relayed as a normal HTTP response

Mechanics (manual 101 - NIOHTTP1's upgrade handler terminates rather than relays, so it cannot be used):

FrontendHTTPHandler sees .head with connection: upgrade + upgrade: websocket; gate on WebSocketProxyRegistry.shouldProxy(host:). Internal WS paths take the terminate path (B4.3).
Checkout/dial an upstream channel on the same loop with HTTPRequestEncoder + ByteToMessageHandler(HTTPResponseDecoder(.forwardBytes)) + WSTunnelClientHandler.
Forward the original request head verbatim (plus x-forwarded-*; drop the 10-header whitelist).
Upstream .head: status ≠ 101 → relay as ordinary HTTP response, keep-alive; status == 101 → write 101 to the client, then on both channels remove HTTPServerPipelineHandler (frontend), the decoder (its .forwardBytes re-fires buffered bytes), the encoder; replace the HTTP handlers with a GlueHandler pair.
Remove/lengthen IdleStateHandler on tunnel establishment.

Migration notes (observable): close codes now propagate; all handshake headers reach upstream; the 127.0.0.1→localhost rewrite hack disappears (dial the socket address directly); keep first-upstream initially to match observed behavior; denial changes from "upgrade then close 1008" to a plain HTTP 403/426 (more correct - document it).

B3.4 SO_REUSEPORT

Single ServerBootstrap on an MTELG = one accept loop, round-robin child distribution across loops. Not the bottleneck at 13k RPS. Implement Listener so it can bind N sockets (one ServerBootstrap per loop with SO_REUSEPORT) behind config flag listen[].reuse_port: false default (~30 lines). Ship single-bootstrap.

B4. Threading model

B4.1 Topology

One MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount) - the data-plane ELG. Both listeners and all upstream connections live here. Not the .singleton.
Control plane: HBApplication constructed with eventLoopGroupProvider: .shared(dataPlaneELG) - control-plane responders run on the same loops (cheap, avoids a hop at the seam; respond completing on the calling loop keeps onComplete loop-local). One-line change to a separate ELG if admin isolation is ever needed.
NIOThreadPool stays, max(8, activeProcessorCount * 2), injected into the control plane: file I/O, ACME reads, bcrypt subprocess, stats archive writes.

B4.2 Same-loop channel pairing (core rule)

Every upstream dial/checkout on the frontend channel's own event loop: ClientBootstrap(group: context.eventLoop). Client and upstream then share one loop → piping is plain synchronous handler code: no locks, no hop(to:), no execute. The per-loop pool (B5) guarantees checkouts never cross loops.

B4.3 Concurrency model per layer

Layer	Model	Why
Byte forwarding (HTTP body relay, WS glue)	ChannelHandler + write/read, no futures per chunk	zero-copy, zero per-part allocation
Request head processing, upstream checkout, response head relay	ChannelHandler + `EventLoopFuture`	connect/checkout are the only async points
Control-plane hand-off	callback seam (`HBHTTPResponder.onComplete`)	the Hummingbird 1.x contract
Internal WS endpoints (stats/benchmark/admin-events)	`NIOWebSocketServerUpgrader` on the frontend pipeline for the three internal paths, then `HBWebSocket(channel:type:.server, maxFrameSize: 1 << 20, logger:)` + `WebSocketHandler` (replicating `HummingbirdWSCore.addWebSocketUpgrade`)	`StatsBroadcaster`/`BenchmarkWebSocket`/`AdminEventsBroadcaster` keep working unmodified
Config reload, cert install, pool sweeps, prewarm, breaker cooldown	actors / async (`ConfigManager`, `CertManager` unchanged; pool maintenance as scheduled loop tasks or a small actor publishing immutable snapshots)	coordination, not hot path
Routing/TLS shared state	`NIOLock` + build-outside/swap-under-lock (`LiveHostIndex`, `SNICertCache`, `WebSocketProxyRegistry`) - unchanged	sub-microsecond reads
Shutdown	`ServerQuiescingHelper` (NIOExtras) per listener + existing `GracefulShutdown`	matches HB semantics

B5. Upstream connection pooling

B5.1 AsyncHTTPClient for the data plane - rejected

AHC 1.33.1 can stream both ways and is a client library (doesn't violate "no framework in the forwarding path" by the letter), but it is still wrong here:

Loop affinity is a preference, not a guarantee - its pool may serve a connection on another loop; every such request pays cross-loop hops and defeats the same-loop design.
Per-chunk abstraction cost - bodies cross an async-sequence producer/consumer boundary (allocation + executor hop per chunk) instead of channelRead → write.
Transparency fights AHC - it owns Host, Content-Length/Transfer-Encoding, connection headers, 100-continue; a proxy wants byte-accurate control of exactly these.
No upgrade relay - cannot relay a client's raw WS handshake; the tunnel path needs raw channels anyway, so running two upstream stacks is worse than one.
Existing pain: first-created-client-wins timeout config per host:port, https-default-port-80 pool-key bug.

Keep AHC where it belongs: AcmeClient (untouched) and BenchmarkRunner - control plane only.

B5.2 Hand-rolled per-loop pool

UpstreamPool  (immutable [ObjectIdentifier(EventLoop): LoopUpstreamPool], built once)
 └─ LoopUpstreamPool          - only touched from its own loop → zero locks
     ├─ idle:   [PoolKey: Deque<PooledConnection>]
     ├─ counts: [PoolKey: Int]
     └─ checkout(key) -> EventLoopFuture<PooledConnection>
PoolKey: (host, port, tls, tlsInsecure)   // default 443 for https - fixes the https→:80 key bug
PooledConnection: Channel + UpstreamHTTPHandler + lease state + idleTimer: Scheduled<Void>
UpstreamHealth  (single NIOLock-guarded map - cross-loop, rare writes)

Checkout (on the frontend's loop): pop idle until isActive; else if counts[key] < perLoopCap dial via ClientBootstrap(group: thisLoop); else queue the waiter (bounded; overflow → 502/503 fast-fail). perLoopCap = max(4, 256 / loopCount) (64/loop on 4-vCPU prod → 256 aggregate).
Check-in: on response .end with reusable framing (no connection: close either side, no tunnel, response fully relayed) → return to idle deque, arm a close timer scheduleTask(in: .seconds(max(keepalive_timeout_seconds, 30))), cancel on next checkout. Non-reusable → close.
TLS upstream: tlsInsecure inserts NIOSSLClientHandler with certificateVerification = .none; else full verification with serverHostname from the upstream URL (activating the ignored tls_server_name is a natural v0.3 add - flag, don't silently change).
Timeouts: connect via ClientBootstrap.connectTimeout (per-site, no longer frozen at first client creation); read deadline via a per-request Scheduled task (see Part D for idle-based replacement).
Prewarm: 2 connections per unique upstream, round-robined across loops, parked idle (replaces the HEAD / hack; keep HEAD as an option).
Circuit breaker / selection: UpstreamHealth keeps exact semantics (5 consecutive → 30 s cooldown, 5xx trips breaker but response relayed verbatim, lazy re-admit); UpstreamSelector reimplements round_robin / random / ip_hash (djb2) / least_connections - RR counters and active-connection counts can be per-loop (eventually-consistent, zero contention); ip_hash stays deterministic.
Streaming both ways replaces buffering; hop-by-hop header stripping per existing sets; keep stripping accept-encoding upstream (compression stays a frontend concern).

B6. File structure

Sources/MoloHTTP/DataPlane/
├── DataPlaneServer.swift        # owns dataPlaneELG, both Listeners, quiescing, start/stop; owns :80/:443
├── Listener.swift               # ServerBootstrap assembly (backlog, SO_REUSEADDR, TCP_NODELAY, optional SO_REUSEPORT, pipeline install)
├── TLSFrontend.swift            # NIOSSLContext from SNICertProvider (reused), NIOSSLServerHandler insertion, lazy-443 policy
├── FrontendHTTPHandler.swift    # duplex handler: part assembly, keep-alive, Server/Date headers, idle events, dispatch decision
├── FrontendClassifier.swift     # internal / ACME / favicon / proxied-WS / host-dispatch decision (pure over head)
├── HostRouter.swift             # LiveHostIndex lookup + per-site auth/rate-limit gate (ports RouterBuilder.DispatchContext)
├── ControlPlaneBridge.swift     # HBHTTPRequest assembly (413 cap), HTTPSForwardingResponder-style respond, HBHTTPResponse write-back
├── ProxyRequestBuilder.swift    # upstream head transform: hop-by-hop strip, x-forwarded-*, header_up, host rewrite (exact ordering)
├── ProxyResponseRelay.swift     # response head transform + streaming body relay, breaker recording, 502/504 mapping
├── UpstreamPool.swift           # immutable per-loop pool map, checkout façade, prewarm, shutdown
├── LoopUpstreamPool.swift       # per-loop idle deques, caps, waiter queue, idle timers (lock-free, loop-confined)
├── UpstreamConnection.swift     # pooled client channel wrapper + client pipeline
├── UpstreamSelector.swift       # round_robin / random / ip_hash / least_connections over healthy subset
├── UpstreamHealth.swift         # circuit breaker: 5 failures → 30 s cooldown, lazy re-admit
├── WSTunnel.swift               # 101 relay orchestration: dial, head relay, pipeline strip choreography
├── GlueHandler.swift            # paired duplex byte piping + backpressure + half-closure/close propagation
└── InternalWSUpgrade.swift      # NIOWebSocketServerUpgrader for the three /molohttp WS paths (replaces StatsWebSocketSetup)

Retired: Server/MoloHTTPServer.swift listener wiring (actor stays as lifecycle owner, now starting DataPlaneServer + control-plane app), TLS/HTTPSServer.swift, Stats/StatsWebSocketSetup.swift, Handlers/WebSocketProxyHandler.swift, ReverseProxyHandler's AHC machinery (semantics migrate into UpstreamHealth/UpstreamSelector). Kept verbatim: HostIndex.swift, SNICertProvider.swift, WebSocketProxyRegistry.swift, CompressionSupport.swift, HTTPSForwardingResponder.swift (as the seam responder), all Config/ACME/Admin/Stats/Middleware, FileServerHandler (invoked from HostRouter for .fileServer sites).

B7. Decision summary

#	Decision	Rationale
1	Raw `ServerBootstrap` pipelines with `HTTPRequestDecoder(.forwardBytes)` + encoder + `HTTPServerPipelineHandler` + `HTTPResponseCompressor` + `IdleStateHandler` + duplex `FrontendHTTPHandler`	mirrors verified HB-core pipeline with zero framework; `.forwardBytes` enables lossless WS tunnel
2	Control-plane seam = reimplemented ~120-line part-assembly loop feeding `HBHTTPResponder.respond` (`HTTPSForwardingResponder` reused)	`HBHTTPServerHandler` is internal; the seam types it needs are public
3	Proxied WS = raw byte tunnel after relayed 101; internal WS = terminate via `NIOWebSocketServerUpgrader` → `HBWebSocket`	zero-copy, native semantics for the proxy; unchanged stats/admin broadcasters
4	One MTELG(cores) shared by listeners and upstream channels; upstream dialed on the frontend's loop; control plane `.shared`; NIOThreadPool retained	lock-free piping; no per-chunk async overhead
5	Hand-rolled per-loop pool; AHC rejected for the data plane, kept for ACME/benchmark	same-loop guarantee is the perf backbone; fixes the https-port and frozen-timeout bugs
6	SO_REUSEPORT behind a config flag, off by default	accept isn't the current bottleneck; cheap insurance
7	Streaming replaces buffering both directions (request cap still enforced; behavioral quirks preserved unless explicitly changed in Part D)	removes the 10 MB request buffer and unbounded response buffer

Part C - Performance and Resource Management Design

All API names verified against the pinned versions in Package.resolved: swift-nio 2.97.1, swift-nio-ssl 2.36.1 (BoringSSL vendored), swift-nio-extras 1.26.0, checked in .build/checkouts/.

C1. Zero-copy piping between client and upstream channels

C1.1 What "zero-copy" means here

NIO has no splice(2) support (C7.5); the achievable zero-copy is userspace pass-through without byte copies: the ByteBuffer produced by the kernel read on leg A is written to leg B as the same storage, moved through channelRead → write with only reference-count traffic. ByteBuffer is COW; as long as the pipe never mutates the buffer and never holds two live references while one mutates, no memcpy occurs between socket-in and socket-out (TLS legs excepted, C2.2).

C1.2 Topology: both legs on ONE event loop

Hard rule: the upstream channel is created on the client channel's own event loop. EventLoop conforms to EventLoopGroup:

ClientBootstrap(group: clientChannel.eventLoop)   // NOT the ELG - the specific loop
    .connectTimeout(.milliseconds(connectMs))
    .channelOption(ChannelOptions.socketOption(.tcp_nodelay), value: 1)
    .channelOption(ChannelOptions.autoRead, value: false)
    .connect(host: host, port: port)

Consequences: every channelRead, write, channelWritabilityChanged, timer, and close on either leg executes on the same thread. The GlueHandler pair needs no locks, no eventLoop.execute hops, no atomics. This is the single biggest structural win over the current AsyncHTTPClient path.

C1.3 The glue pair and its state machine

One ChannelDuplexHandler class, instantiated twice and cross-linked (the proven GlueHandler pattern from NIO's connect-proxy example, ~80 lines, implemented locally):

final class GlueHandler: ChannelDuplexHandler {
    typealias InboundIn = NIOAny; typealias OutboundIn = NIOAny; typealias OutboundOut = NIOAny
    private var partner: GlueHandler?
    private var context: ChannelHandlerContext?
    private var pendingRead = false
    static func matchedPair() -> (GlueHandler, GlueHandler)
}

Per-leg event handling, exactly:

Event on leg A	Action
`handlerAdded(ctx)`	store `context`
`channelRead(data)`	`partner.context.write(data, promise: nil)` - same `NIOAny`, same `ByteBuffer` storage
`channelReadComplete`	`partner.context.flush()` - flush once per read burst, never per buffer
`read()` (outbound, from B's pipeline)	if `partner.channel.isWritable` → `context.read()`; else set `pendingRead = true` and do not propagate
`channelWritabilityChanged`	if `isWritable` and `partner.pendingRead` → `partner.pendingRead = false; partner.context.read()`
`channelInactive`	`partner.close(mode: .all)` if partner writes drained, else close after last write promise; then drop both `partner` refs (C2.3)
`errorCaught`	close self and partner
user event `ChannelEvent.inboundClosed` (half-closure)	`partner.close(mode: .output)` to propagate FIN without killing the reverse direction

C1.4 Backpressure protocol (precise)

autoRead = false on both legs. Nothing is read unless the pipe explicitly asks.
Reading driven by the downstream consumer: the response head having been written triggers the first context.read() on the upstream leg; the client leg is read only while the request body is being forwarded.
After each channelRead(A) → write(B), leg A's next read() is gated on B's writability: write with promise: nil, flow control via Channel.isWritable + channelWritabilityChanged against PendingWritesManager watermarks. Verified defaults at 2.97.1 (PendingWritesManager.swift:336): low 32 KiB, high 64 KiB. B over 64 KiB buffered → isWritable false → A stops issuing read() → A's recv buffer fills → TCP window closes → the origin peer is throttled end-to-end. B drains below 32 KiB → channelWritabilityChanged → stalled read() replayed.
Watermark tuning: keep 32/64 KiB but set explicitly so the numbers are pinned: ChannelOptions.writeBufferWaterMark, value: .init(low: 32*1024, high: 64*1024). Worst case buffered at 500 connections: 500 × 64 KiB = 31 MiB - acceptable; 512 KiB marks would be 250 MiB - not.
Maximum in-flight bytes per proxied connection: recv allocation (≤64 KiB) + high watermark (64 KiB) + kernel socket buffers - ≈128 KiB userspace ceiling per connection, versus today's unbounded full-response buffering.

C1.5 HTTP framing on the legs

Client leg keeps NIOHTTP1 configureHTTPServerPipeline framing; upstream leg gets HTTPRequestEncoder + HTTPResponseDecoder(leftOverBytesStrategy: .forwardBytes). The glue pair pipes typed HTTPServerRequestPart/HTTPClientResponsePart - .body(ByteBuffer) carries the untouched buffer (NIOHTTP1 decoders slice the incoming buffer; slices share storage). Header rewriting (X-Forwarded-*, hop-by-hop stripping) happens exactly once on .head, never on .body.

C2. Buffer strategy

C2.1 Receive allocation

AdaptiveRecvByteBufferAllocator - verified defaults at 2.97.1: minimum: 64, initial: 2048, maximum: 65536. For proxy legs override to (minimum: 1024, initial: 16384, maximum: 65536): starting at 16 KiB avoids the ramp-up on every fresh upstream connection; the 64 KiB cap matches the write high watermark.
maxMessagesPerRead: verified default 4. Keep 4 - with autoRead off it bounds the burst per explicit read() to 4 × 64 KiB = 256 KiB worst case. NIO 2.97.1 pools recv buffers per channel (recvBufferPool, capacity = maxMessagesPerRead) and reuses a buffer when the pipe releases it before the next read - reuse works automatically because write(B) + kernel send usually releases the buffer within the same loop tick, and only if no handler squirrels the buffer away.

C2.2 When copying is forced

TLS legs: NIOSSLHandler must encrypt - one copy per direction per TLS leg, unavoidable.
Header rewrite on .head: new HTTPRequestHead/HTTPHeaders - negligible, once per request.
Compression toward the client (HTTPResponseCompressor): copies by definition. Keep the current predicate gating; streamed pass-through of already-compressed upstream bodies skips it entirely.
Never call buffer.getBytes/Data(buffer:)/String(buffer:) on body parts. Byte counting uses buffer.readableBytes only.

C2.3 ByteBuffer reference-counting pitfalls

Retain cycle: the GlueHandler pair holds partner strongly in both directions - MUST be broken by nilling partner (and context) in channelInactive/handlerRemoved on both sides, or every proxied connection leaks two handlers, two channels, and buffers. This is the number-one leak in glue-pattern proxies.
Holding buffers too long: any handler that stores a ByteBuffer keeps the whole 16-64 KiB backing store alive and defeats recv-pool reuse (reuse only at refcount 1). Stats record integers, never buffers.
Accidental COW copies: never mutate body buffers; any transformation produces a new buffer.

C3. Connection management

C3.1 Upstream pool - per-event-loop, lock-free

One pool instance per event loop per upstream key (scheme://host:port[+tls-insecure], keeping the current key quirks). A pooled connection lives on exactly one loop and is only checked out by requests on that loop - the pool is a plain class touched from a single thread: no NIOLock, no actor.

Sizing on 4 vCPU (4 loops): 64 idle-capable connections per loop per upstream key (4 × 64 = 256, preserving today's aggregate); overflow beyond 64 creates an unpooled connection closed after use (soft-limit semantics). Pre-warm: 2 connections total per upstream, created round-robin across loops at startup by UpstreamPrewarmer.

Checkout protocol: pop an idle connection, verify channel.isActive synchronously on the loop before handing it out, else dial fresh.

C3.2 Timeouts - NIO primitives

Timeout	Value	Primitive
Upstream connect	`connection_timeout_ms` > 0 else 5000 ms	`ClientBootstrap.connectTimeout(_:)`
Pool idle	`max(keepalive_timeout_seconds, 30)` s	`IdleStateHandler(readTimeout:)` added while parked, removed at checkout (`RemovableChannelHandler`); on `IdleStateEvent.read` → close + evict
Per-request deadline/read	`read_timeout_ms` > 0 else 120 000 ms	`eventLoop.scheduleTask(deadline: NIODeadline.now() + ...)` armed at dispatch, `Scheduled.cancel()` on response `.end` (see Part D for idle-based replacement of the whole-response deadline)
Client keepalive idle	`max(keepalive, 30)` s read+write	`IdleStateHandler` on the client leg

NIODeadline.now() (monotonic) everywhere in the data plane; no wall-clock Date in the hot path.

C3.3 Zombie prevention - every error path closes BOTH legs

Single invariant, one place: the per-exchange ProxyExchange state object owns both channel references and has exactly one idempotent tearDown(error:) (guarded by a Bool, single-threaded) that always: cancels the deadline task, closes the client leg, closes/evicts the upstream leg, decrements activeConnections, calls recordFailure/recordSuccess, and nils the glue partner links. Leak scenarios covered:

Client drops mid-response - upstream leg closed, not returned to the pool (mid-body stream is poisoned for HTTP/1.1 reuse).
Upstream stalls (connected, no bytes) - caught only by the deadline/idle task (no IdleStateHandler on the leg during an active exchange); fires → both legs closed.
TLS handshake failure on upstream - failure path still releases the pool slot, runs recordFailure, and the client leg receives its 502 rather than hang.
Pool checkout raced with close - isActive check at checkout plus first-write failure → treated as a dial failure with one transparent redial (request head not yet answered), not a client-visible 502.
Half-closure misuse - with allowRemoteHalfClosure = true on the upstream leg, a client FIN propagates as close(mode: .output); full teardown still runs when the response completes or the deadline fires.

Debug-only zombie audit: a 60 s scheduled task per loop logging exchanges older than 2 × deadline.

C4. DNS resolution

Verified at swift-nio 2.97.1 (GetaddrinfoResolver.swift): ClientBootstrap.connect(host:port:) goes through HappyEyeballsConnector + GetaddrinfoResolver, which offloads the blocking getaddrinfo to a DispatchQueue - it does not block the event loop at this version.

Recommendation regardless: fast-path IP literals with SocketAddress(ipAddress:port:) and connect(to:), skipping Happy Eyeballs entirely (removes the DispatchQueue hop for the 99% case - upstreams here are IP literals/localhost). For hostname upstreams: resolve at config load/reload off-loop, cache the SocketAddress with a 60 s TTL refreshed by a background scheduled task; on connect failure, force a re-resolve before the breaker counts a second failure. Preserve the existing quirks verbatim: the WS path's ://127.0.0.1 → ://localhost and ://[::1] → ://localhost substitutions, and Host/SNI derivation from the URL string rather than the resolved address.

C5. TLS

C5.1 Server side - SNI and session resumption

Verified in vendored BoringSSL (2.36.1): server SSL_CTX defaults are session_cache_mode = SSL_SESS_CACHE_SERVER and num_tickets = 2 - TLS 1.2 session-ID cache and TLS 1.3 tickets are already on by default; NIOSSL 2.36.1 exposes no tuning API and none is needed.

Critical verified detail: sslContextCallback + NIOSSLContextConfigurationOverride does not rebuild an NIOSSLContext per handshake. SSLConnection.applyOverride (verified at SSLConnection.swift:506) applies the override via SSL_use_certificate/private-key calls on the individual connection - refcount bumps on the already-parsed NIOSSLCertificate/NIOSSLPrivateKey objects. Per-handshake cost: one locked dictionary lookup + wildcard walk + a handful of BoringSSL setter calls. Verdict: keep the single-parent-context + override design. Do not pre-build one NIOSSLContext per domain: the override path is already cheap, and a single parent context means one ticket-key set and one session cache shared across all SNI domains - per-domain contexts would fragment ticket keys and rotate them on every config reload. Two small upgrades: replace the NIOLock in SNICertCache.lookup with an atomically-swapped immutable snapshot (lock-free reads on the handshake path); keep certificates pre-parsed exactly as today. The genuinely expensive part is building the parent NIOSSLContext - once at boot and once per cert-set change, never per connection.

C5.2 Client side - TLS to upstreams

Cache one NIOSSLContext per upstream key (including +tls-insecure), built lazily at first dial, stored in the per-loop pool structure (context objects are thread-safe to share). BoringSSL client-side session caching plus pooling makes upstream handshakes rare after warmup. transport.tls_server_name remains ignored unless consciously activated (flag as a behavior decision, not a silent change).

C5.3 Handshake on-loop

NIOSSL performs the handshake on the event loop. RSA-2048 signing ≈ 0.5-1 ms per full handshake; ECDSA P-256 ≈ 0.1 ms. Acceptable on 4 loops at this deployment's rates (mostly resumed). Do not build an off-loop signing shim; NIOSSLCustomPrivateKey exists at 2.36.1 as the escape hatch if this ever measures as a bottleneck.

C6. Allocation pressure in the hot path

No Date() per request. Measure with NIODeadline.now() at .head-in and .end-out; the stats ring-buffer second index comes from a per-loop cached epoch second refreshed by a 1 s scheduled task - zero Foundation calls per request.
No String interpolation in per-request logging. Guard debug logs with a level check; error-path metadata only. Access logging keeps its buffered/thread-pool design.
Precompute header fragments. Static parts of the forwarded-header rewrite and per-site constant header_up results are built once per site config generation as ready HTTPHeaders templates; per request only variadic values (client IP, host) are appended. The hop-by-hop strip set is a static let of lowercased names.
Stats churn: pass the host String already extracted for routing; call record exactly once per exchange at teardown with integer arguments only.
No NIOAny unwrap-rewrap churn: forward opaquely where possible; unwrap once where typed parts are needed (.head rewrite).

C7. OS and socket tuning

Accept model: keep ServerBootstrap's single listening socket with round-robin child distribution across the 4 loops. SO_REUSEPORT multi-accept becomes measurable only at ~10k+ new connections/s - note as a one-line change if accept ever shows in perf.
TCP_NODELAY on both legs - set explicitly on every upstream ClientBootstrap (AsyncHTTPClient did this for us; raw NIO does not).
Backlog 4096 retained; verify sysctl net.core.somaxconn=4096 (Ubuntu 24.04 default is 4096 - verify, do not assume).
SO_KEEPALIVE on pooled upstream connections so silently dead upstreams are reaped by the kernel; the pool idle timeout (≤300 s) reaps first, so no TCP_KEEPIDLE sysctl surgery.
splice(2): NIO 2.97.1 has none (verified). ByteBuffer piping is the achievable zero-copy. Do not attempt a raw-fd splice side-channel - it would bypass TLS and NIO's state machine.
Do not touch tcp_tw_reuse/ip_local_port_range unless ephemeral-port exhaustion appears (unlikely: pooled keep-alive to a handful of localhost ports). LimitNOFILE=1048576 already covers FD headroom.

C8. Instrumentation without cost

Streamed byte counting: the glue handler increments two plain Int fields on the exchange object as it pipes (single-threaded - no atomics, zero allocation).
One StatsMonitor.record call per exchange, at teardown, with the existing signature record(host:method:status:durationMs:bytesSent:bytesRecv:) - durationMs from NIODeadline delta, bytesSent = bytesToClient, bytesRecv = bytesToUpstream. The lock+ring-buffer side stays untouched; only the caller and clock change.
Real efficiency counters: StatsMonitor currently hardcodes zero_copy_ratio: 0, splice_transfers: 0, bytes_via_splice: 0. The new plane reports passThroughExchanges (glue-path bodies, no re-buffering/compression), bufferedExchanges (compressed/transformed/internal), bytesPassedThrough; zero_copy_ratio = passThroughExchanges / totalExchanges; keep the JSON keys for dashboard compatibility, document the new semantics.
Per-loop counters, merged on tick: if the single NIOLock in StatsMonitor ever contends at >20k RPS, shard the hot counters per event loop and let the 1 s tick() sum them. Not required at the current 13k RPS baseline; pre-approved next step.

C9. Consolidated numbers table

Parameter	Value	Mechanism
Event loops	4 (`System.coreCount`)	`MultiThreadedEventLoopGroup`
Upstream leg placement	client channel's loop	`ClientBootstrap(group: clientChannel.eventLoop)`
autoRead	off, both legs	`ChannelOptions.autoRead = false`
Write watermarks	low 32 KiB / high 64 KiB (explicit)	`ChannelOptions.writeBufferWaterMark`
Recv allocator	Adaptive 1 KiB / 16 KiB / 64 KiB	`AdaptiveRecvByteBufferAllocator(minimum:initial:maximum:)`
maxMessagesPerRead	4 (default, pinned)	`ChannelOptions.maxMessagesPerRead`
Userspace ceiling per connection	≈128 KiB	watermarks + recv cap
Pool size	64/loop/upstream key (256 aggregate)	per-loop lock-free pool
Pool idle timeout	`max(keepalive,30)` s; 300 s default	`IdleStateHandler(readTimeout:)` while parked
Connect timeout	5000 ms default	`ClientBootstrap.connectTimeout`
Request read timeout	120 000 ms default	`eventLoop.scheduleTask` + `Scheduled.cancel()`
TLS server sessions	cache + 2 tickets, BoringSSL defaults	nothing to configure at NIOSSL 2.36.1
SNI	single parent context + per-connection override (kept)	`sslContextCallback`, pre-parsed cert material
Backlog / somaxconn	4096 / 4096	bootstrap option + sysctl verify
TCP_NODELAY / SO_KEEPALIVE	on / on (upstream legs)	socket options on ClientBootstrap

Part D - HTTP/1.1 Protocol Correctness Specification

Pinned toolchain (verified in .build/checkouts/swift-nio): swift-nio 2.97.1, NIOHTTP1 backed by vendored llhttp 9.3.0. HTTPDecoder.swift calls no c_nio_llhttp_set_lenient_* function - every llhttp lenient flag is off (full strict mode). All "parser rejects/accepts" claims verified against this exact checkout.

The rewrite must preserve the observable behavior in Part 2 except where a rule here explicitly supersedes a buffering-era quirk. Bodies become streamed end-to-end; the head is always re-serialized, never relayed as raw bytes.

D0. Baseline architecture rules

R0.1 (MUST) Operate on HTTPServerRequestPart / HTTPClientResponsePart message parts (.head / .body(ByteBuffer) / .end(HTTPHeaders?)). Client-facing and upstream-facing heads are always rebuilt from parsed structures and re-serialized by the NIO encoders. Raw head bytes MUST never be copied between legs. This single rule neutralizes every parse-discrepancy smuggling class: whatever ambiguity the front parser tolerated cannot survive re-serialization.
R0.2 (MUST) Each upstream connection uses the paired NIO client codec (HTTPRequestEncoder + HTTPResponseDecoder on the same channel). The response decoder queues request methods and forces skipBody for HEAD/CONNECT and 1xx/204/304 - this pairing keeps the upstream connection framing-synchronized. Reusing a decoder without feeding it the request method is forbidden.
R0.3 (MUST) Include HTTPServerProtocolErrorHandler (or equivalent) in the server pipeline: on any HTTPParserError write 400 Bad Request + Connection: close + Content-Length: 0, then close the channel. Uniform fate of every malformed message the parser rejects.

D1. Request smuggling defense

llhttp 9.3.0 strict (as pinned) already rejects: TE+CL together (either order), duplicate Content-Length, non-digit CL, Transfer-Encoding where chunked is not the final coding, invalid chunk sizes/extensions, missing CRLF after chunk data, obs-fold, whitespace/non-token chars in header names, space before colon, bare LF/CR line endings, trailing data after Connection: close, unknown HTTP version, whitespace between start-line and first header. All of these become a 400 + close via R0.3.

Our rules for what the parser does NOT reject:

R1.1 (MUST) Multiple Host header lines pass llhttp. Respond 400 + close when head.headers["host"].count != 1 on HTTP/1.1 (RFC 9112 §3.2).
R1.2 (MUST) Never enable any LENIENT_* decoder flag - a security regression, forbidden forever.
R1.3 (MUST) Never forward the client's framing headers (Transfer-Encoding, Content-Length, Connection, Keep-Alive); the upstream leg's framing is regenerated per D2. The decoder has already de-chunked the body - chunk sizes/extensions/line endings never reach the upstream.
R1.4 (MUST) Chunk extensions dropped by design (RFC 9112 §7.1.1 permits ignoring them).
R1.5 (MUST) On any HTTPParserError close the client connection after the 400. Resynchronizing after a framing error is forbidden.
R1.6 (MUST) Header values assembled by molohttp itself (X-Forwarded-*, header_up config values, host rewrite) MUST be validated against \r, \n, \0 before insertion so config cannot inject CRLF upstream (llhttp guarantees parsed inbound values are already clean).

D2. Framing on both legs

Request leg (client → upstream)

R2.1 (MUST) Client had valid Content-Length N → forward Content-Length: N, stream body; the front decoder guarantees exactly N bytes. Do not convert to chunked.
R2.2 (MUST) Client was chunked → forward chunked (no CL; the encoder auto-inserts transfer-encoding: chunked, serializes each .body as a chunk, .end(trailers) as last-chunk + trailer section). Re-chunk, never raw-chunk-passthrough.
R2.3 (MUST) Never emit both CL and TE upstream.
R2.4 (MUST) Bodiless requests: forward with neither framing header. Because HTTPRequestEncoder decides hasBody from the method, a bodiless POST head would get chunked injected. Strategy (global): set automaticallySetFramingHeaders = false on both encoders and manage framing explicitly per these rules - every decision auditable.

Response leg (upstream → client)

R2.5 (MUST) Upstream Content-Length + identity body → relay CL verbatim, stream body (streaming means CL passthrough is now correct, unlike the buffering era which stripped it).
R2.6 (MUST) Upstream chunked → re-chunk to an HTTP/1.1 client, relay trailers (D6).
R2.7 (MUST) Upstream close-delimited (no CL/TE, ends at EOF) → to HTTP/1.1 client re-encode as chunked and terminate on upstream EOF; discard the upstream connection. To HTTP/1.0 client stream identity + close with Connection: close.
R2.8 (MUST) HTTP/1.0 clients never receive chunked; unknown length → R2.7 close-delimited path.

Bodiless responses - desync-critical

R2.9 (MUST) Any 1xx, 204, 304, every response to HEAD (and 2xx to CONNECT - moot, CONNECT rejected): write .head + .end only, never .body.
R2.10 (MUST - encoder trap) HTTPResponseEncoder frames from status.mayHaveResponseBody only - it does not know the request method. A HEAD 200 with neither CL nor TE would get chunked injected + a 0\r\n\r\n body → desync. For HEAD: preserve upstream Content-Length verbatim and suppress automatic framing (R2.4), write zero body bytes.
R2.11 (MAY) For 204/304 the manual-framing strategy MAY preserve CL while writing no body.
R2.12 (MUST) Never decode upstream responses without method pairing (R0.2).

D3. Hop-by-hop headers

R3.1 (MUST) Strip in both directions: Connection, every header nominated in the Connection value (split on comma, trim OWS, case-insensitive - llhttp does not do this for us), Keep-Alive, Proxy-Connection, Proxy-Authenticate, Proxy-Authorization, TE, Trailer, Transfer-Encoding, Upgrade. Upgrade survives only on the gated WebSocket path (D7).
R3.2 (MUST, compat) Also strip Accept-Encoding on the request leg (upstreams answer identity; compression toward the client is the server layer's job). Note: the current strip set contains the non-standard trailers; the correct header is Trailer - the rewrite uses the correct name (a deliberate, safe correction).
R3.3 (MUST, compat) Inject on the request leg, in contract order:
- x-forwarded-for: append as an additional header line (add), value = client IP or "unknown". Multiple same-name field lines are semantically identical to a comma list (RFC 9110 §5.3); keep append semantics. Optional comma-merge only if measured upstreams tolerate it; byte-compatible append wins by default.
- x-forwarded-proto: replaceOrAdd, incoming first value else "http".
- x-forwarded-host: replaceOrAdd, incoming Host first value else "".
- x-real-ip: add, client IP or "unknown".
- then header_up add/set/remove, then the Host decision (upstream_host_rewrite: {upstream_host} → upstream URL host; literal → literal; nil → client Host) - exactly per contract.
R3.4 (SHOULD) Via: 1.1 molohttp behind a config flag, default off for byte-compatibility (documented RFC deviation).
R3.5 (MUST) Response leg strips the same R3.1 set (Connection-nominated included). CL/TE governed by D2. content-encoding is no longer stripped (identity bodies stream unmodified).

D4. Connection lifecycle

R4.1 (MUST) Keep-alive default per the decoder: HTTP/1.1 persistent unless Connection: close; HTTP/1.0 non-persistent unless Connection: keep-alive. Honor per-message.
R4.2 (MUST) Client Connection: close → complete the response, set Connection: close, close the client channel; the upstream connection returns to the pool if its response completed cleanly. Client close signals never leak upstream.
R4.3 (MUST) Upstream signals close (header / close-delimited / EOF) → close the upstream connection, never pool it. The client stays alive if its relayed response was completely framed. If upstream EOF truncates a CL-framed response mid-body, abort the client connection (no clean terminator) so the client sees truncation, not a forged-complete response.
R4.4 (MUST) Pipelining: include HTTPServerPipelineHandler - it serializes (buffers subsequent pipelined requests, suppresses reads until the current response completes). One exchange per connection at a time; concurrent handling forbidden.
R4.5 (MUST) Half-close: enable allowRemoteHalfClosure. Client FIN after a complete request → finish, write response, close. Client FIN mid-request-body → abort, cancel upstream, no response.
R4.6 (MUST - resource-leak firewall) Client disconnect with an in-flight upstream exchange → immediately close the upstream channel. A partially-consumed streaming exchange can never be pooled. Tie to the client channel's closeFuture.
R4.7 (MUST) Pool admission: re-enter only when (a) response .end received, (b) keep-alive per R4.1 on the upstream leg, (c) no trailing bytes (decoder error = connection poison).
R4.8 (MUST, compat) Preserve: per-site breaker (5 failures / 30 s), all four selection policies, one attempt per request (streamed request bodies make replay impossible once body bytes flow - this is the documented reason), upstream 5xx relayed verbatim but counted as breaker failure. (Timeout status code: see Part E - introduce 504 with a compatibility note, superseding the contract's 502-for-timeouts quirk.)

D5. Expect: 100-continue

R5.1 (MUST) Terminate the expectation locally. On Expect: 100-continue (case-insensitive): strip Expect from the forwarded head; write HTTPResponseHead(status: .continue) to the client when ready to pull body bytes; then stream the body upstream normally. NIOHTTP1 provides no automatic 100-continue in the raw pipeline; the encoder handles interleaved informational heads.
R5.2 (MUST NOT) Do not half-forward the expectation (either variant deadlocks). Full relay is explicitly deferred; local termination loses only the early-reject optimization, never correctness.
R5.3 (MUST) Expect with any other value → 417 Expectation Failed, no upstream contact.
R5.4 (MUST) Never send 100 to HTTP/1.0; skip the 100 if body bytes already arrived.
R5.5 (MUST) If rejected before body read (413 precheck, no upstream, auth), send the final status instead of 100 and close per R8.4.

D6. Trailers

R6.1 (MUST) Relay trailers (verified plumbing: decode into .end(HTTPHeaders?), encode after last-chunk only when chunked). Upstream response trailers → client .end(trailers) (client leg chunked per R2.6); client request trailers → upstream .end(trailers) (R2.2).
R6.2 (MUST) Filter the R3.1 hop-by-hop set out of relayed trailers; drop trailers entirely when the receiving leg is not chunked (RFC-conformant - trailers are droppable metadata).
R6.3 (MUST NOT) Never merge trailers into the header block.

D7. Upgrade and CONNECT

R7.1 (MUST - supersedes terminate-and-redial) WebSocket: raw 101 relay. Gate exactly as today (WebSocketProxyRegistry.shouldProxy: reverse-proxy site with websocket == true, first upstream). On a gated request with Connection naming upgrade and Upgrade: websocket: forward the head upstream preserving Upgrade/Connection: upgrade/Sec-WebSocket-* verbatim (exempt from R3.1 on this path only) plus the forwarded header set. Upstream 101 → relay verbatim (Sec-WebSocket-Accept must survive), remove HTTP encoders/decoders/pipeline handlers from both channels, splice raw duplex with backpressure. Upstream non-101 → relay as a normal HTTP response, stay in HTTP mode. NIO support verified: decoder pauses at upgrade, buffered post-upgrade bytes recovered via leftOverBytesStrategy: .forwardBytes - both channels' leftover bytes MUST be flushed into the splice before free-flow. Fixes contract quirk 6 (close-code loss, 1 MiB cap, unbounded pre-attach buffering, no backpressure) wholesale.
R7.2 (MUST) Non-WebSocket Upgrade (h2c, TLS upgrade, anything) and WebSocket on non-gated hosts: strip Upgrade + Connection nomination (R3.1), forward as a plain request; never 101. Replaces the cosmetic UpgradeDenied() log-spam.
R7.3 (MUST) CONNECT → 501 Not Implemented + Connection: close + close; never routed, never forwarded.

D8. Malformed and malicious input

R8.1 (MUST) Head size: the pinned decoder caps each element at 80 KiB but has no aggregate head limit and no header-count limit. Add a byte-counting handler: default 64 KiB total head and 128 header fields (configurable), exceeded → 431 + close.
R8.2 (MUST) Slowloris: read-idle timeout via IdleStateHandler covering the header phase - head not complete within max(keepalive_timeout_seconds, 30) s → 408 + close. Same machinery bounds stalled bodies. (Part E tightens the header-phase window to a dedicated 10 s.)
R8.3 (MUST) Request body cap: keep max_request_body_bytes (default 10 MB) by counting streamed bytes in the body-relay handler. Declared Content-Length over the limit → 413 immediately, before upstream contact.
R8.4 (MUST) 413 mid-stream: counter trips after forwarding began - no upstream head relayed yet → 413 + Connection: close, close both; upstream head already relayed → abort both without a synthetic status. Never keep-alive after a mid-stream 413.
R8.5 (MUST) Slow-client response path: propagate backpressure (stop read() on the upstream while the client is unwritable, resume on writable; symmetric on the request leg). Unbounded inter-leg buffering forbidden. Write-idle timeout closes stalled clients; on trigger R4.6 tears down the upstream.
R8.6 (MUST) Upstream response body size remains unbounded by policy; streaming means no proxy-memory risk, R8.5 caps in-flight buffering, and the read deadline is the backstop.

D9. Request-target and Host rules

R9.1 (MUST) Absolute-form request-target (GET http://host/path HTTP/1.1): parse out authority + path-query; route by authority (ignoring Host per RFC 9112 §3.2.2); forward origin-form (path-query only) upstream. OPTIONS * → route by Host, forward * verbatim for OPTIONS only, else 400. Authority-form outside CONNECT → 400.
R9.2 (MUST) Missing Host on HTTP/1.1 → 400. HTTP/1.0 without Host → default/fallback site if configured, else 400. Multiple Hosts → 400. Host with port: strip for lookup (bracketed IPv6 preserved), forward original where Host passthrough applies.
R9.3 (MUST) Percent-encoding/path bytes: never decode, re-encode, or normalize when building the upstream target. Preserve upstream.url + rawPath (+ ? + rawQuery), no prefix stripping, double-slash tolerance. File-server securePath operates on its own decoded copy, never feeding back into forwarded bytes.
R9.4 (MUST) Fragments: forward verbatim, never strip/interpret.
R9.5 (MUST) Forward the method verbatim (unknown-but-tokenized surface as .RAW(value:)) except CONNECT (R7.3).

D10. HTTP/2 / HTTP/3 scope

R10.1 (MUST) This phase is HTTP/1.1-only on both legs (HTTP/1.0 accepted client-side; upstream requests always 1.1).
R10.2 (MUST) The TLS listener's ALPN MUST advertise only http/1.1 (TLSConfiguration.applicationProtocols = ["http/1.1"]). Advertising h2 while speaking NIOHTTP1 → h2 preface into an h1 parser → 400s/hangs.
R10.3 (MUST NOT) No h2c upgrade (R7.2 strips it), no h2 prior knowledge, no Alt-Svc, no QUIC/HTTP3.
R10.4 (note) A later h2 phase needs an ALPN-switched pipeline, swift-nio-http2 payload↔head translation, :authority↔Host mapping, per-stream flow control into the D8.5 backpressure model, and RFC 8441 extended-CONNECT for WS-over-h2 - none of which changes the upstream h1 leg.

D11. Parser/codec facts index (pinned swift-nio 2.97.1)

Fact	Consequence
All llhttp lenient flags off	strict rejections in D1 apply
TE+CL, dup CL, bad chunks, obs-fold, bad tokens, bare LF/CR rejected	400 + close, no proxy normalization needed
Request TE without final `chunked` → error	RFC 9112 §6.1 satisfied by parser
Response close-delimited detection	drives R2.7
Per-field 80 KiB cap, no aggregate cap	R8.1 aggregate limit is on us
HEAD/CONNECT/1xx/204/304 skip-body + request-method pairing	R0.2, R2.12
Encoder frames by status only; would chunk HEAD responses	R2.10 trap
Encoder auto-chunks 1.1 body without CL; strips CL/TE on `.no`	R2.2, R2.11
Trailers decode into `.end(HTTPHeaders?)`, encode after last-chunk	D6
`HTTPServerPipelineHandler`: serialized pipelining, read suppression, half-close ordering	R4.4, R4.5
Upgrade pause + resume + `.forwardBytes`	R7.1 splice
`HTTPServerProtocolErrorHandler`: auto 400 + close	R0.3
100-continue only in `NIOHTTPObjectAggregator` (unused)	D5 is proxy code

Part E - Error Handling, Reliability, Security, Observability

Rule IDs: ER errors, TO timeouts, CB circuit breaker, RT retries, SEC security, GS shutdown/reload, OB observability, CT chaos tests. MUST = required for v1 parity/correctness; SHOULD = strongly recommended, may ship behind config.

E1. Partial-failure matrix

Terminology: client leg = accepted downstream channel; upstream leg = pooled/dialed upstream channel; head-committed = the response status line has been written to the client (after this point no status can change - only truncation + connection close remain).

ER-1 (MUST) Every failure path MUST free both channels, return or destroy the pooled upstream connection as specified, decrement activeConnections exactly once, and complete the per-request state machine exactly once (idempotent completion guard - NIO can deliver errorCaught and channelInactive for the same event).
ER-2 (MUST) A pooled upstream connection MUST be returned to the pool only if the upstream response was read to completion and the connection is keep-alive. On any error, cancellation, or early client abort, the upstream connection MUST be closed and destroyed, never returned (a half-consumed HTTP/1.1 stream is unusable).
ER-3 (MUST - behavior change) Connect timeout and response-head timeout → 504 Gateway Timeout; connect refused/reset/DNS → 502 Bad Gateway. Compatibility note: v0.2.0 returns 502 for timeouts; 504 is semantically correct and what nginx/Caddy emit. Document in the changelog. No legacy flag.
ER-4 (MUST) Upstream 5xx relayed verbatim (never converted to 502) but recorded as a breaker failure.
ER-5 (MUST) Once head-committed, no error may synthesize an error response. The client leg is closed without a Content-Length-satisfying body (truncation); for chunked responses the terminal 0\r\n\r\n MUST NOT be sent, so the client detects truncation.
ER-6 (MUST) Error bodies for data-plane-generated statuses (400/401/404/413/431/502/503/504) use the existing RFC 9457 rendering.

#	Failure	Peer leg action	Status to client	Logged	Breaker	Cleanup
F1	Client RST/close mid-request	Abort/cancel upstream dial or close upstream	none possible	access log `499`, debug	nothing	both closed; buffers released; activeConnections−−
F2	Client RST/write error mid-response	Close upstream (destroy)	none (head committed)	`499` + bytes-so-far	nothing	cancel upstream reads, close both
F3	Upstream connect refused / DNS / unreachable	client leg intact	502 "Upstream connection failed"	warn (sampled)	failure	activeConnections−−; RT-1 may retry first
F4	Upstream connect timeout (TO-1)	cancel connect	504 "Upstream connect timeout"	warn (sampled)	failure	connect future cancelled; activeConnections−−
F5	Upstream RST/EOF before response head complete	destroy upstream	502 "Upstream connection failed"	warn (sampled)	failure	destroy upstream; discard client body remainder by closing after response
F6	Upstream response-head timeout (TO-2)	destroy upstream	504 "Upstream timeout"	warn (sampled)	failure	as F5
F7	Upstream RST/EOF mid-response-body	client: stop writes, close without chunked terminator (ER-5)	none (head committed)	error `upstream_body_truncated` + access log actual bytes	failure	destroy upstream, close client
F8	Upstream between-chunks read timeout (TO-3) mid-body	as F7	none (head committed)	error `upstream_read_timeout`	failure	as F7
F9	Upstream response head parse error (garbage/oversized per SEC-6)	destroy upstream	502 "Invalid upstream response"	warn (sampled), length only - never raw payload	failure	as F5
F10	TLS handshake failure, client leg	n/a	none (pre-HTTP)	debug only (counter metric)	nothing	channel closed by NIOSSL
F11	TLS handshake failure, upstream leg	client leg intact	502 "Upstream TLS failure"	warn (sampled) + SSL error	failure	destroy upstream; activeConnections−−
F12	Pool checkout returns dead connection	destroy checked-out conn	none yet - MUST retry checkout/dial once transparently (pool artifact, not RT-1). Second failure → F3/F5	debug `stale_pooled_connection`	not a failure on first stale hit	destroyed stale; fresh dial proceeds
F13	Write timeout to slow client (TO-6)	destroy upstream	none (usually head committed); pre-head just close	`499`, debug `client_write_stall`	nothing	close both; release pending write buffer
F14	All upstreams cooling / empty list	n/a	502 "No upstream available" (keep 502 in v1 for compat; SHOULD become 503 later)	warn (sampled per site)	nothing	no resources acquired
F15	Request body exceeds `max_request_body_bytes` mid-stream	destroy upstream	413 if head not committed, then `Connection: close`	access log 413	nothing	close upstream (poisoned), close client
F16	WS upstream dial failure (after client upgrade accepted)	n/a	WS close 1011	warn (sampled)	failure (SHOULD - recording protects HTTP traffic to the same upstream)	client WS closed; no bridge
F17	WS bridge: either side closes/RSTs	close peer with 1001 (codes not propagated in v1)	n/a	debug	nothing	both WS channels closed, pending dropped

ER-7 (MUST) Backpressure replaces buffering as the primary defense: when the upstream channel is not writable, suspend client-leg reads; symmetrically on the reverse direction. F13 fires only when a stall persists beyond TO-6, not on transient unwritability.

E2. Timeouts inventory

ID	Timeout	Default	Applies to	Rule
TO-1	Upstream connect (TCP+TLS)	5000 ms (`connection_timeout_ms`)	dial start → channel active	MUST
TO-2	Upstream response-head	`read_timeout_ms` (120 000 ms)	last request byte → response head parsed	MUST - new distinct timer; default equals current deadline so head-stage behavior is compat-identical
TO-3	Upstream between-chunks read idle	`read_timeout_ms`, reset per chunk	mid-body	MUST
TO-4	Whole-response deadline	removed	-	MUST NOT exist. Behavior change: v0.2.0's single 120 s wall-clock deadline caps every download/SSE stream at 120 s; TO-2 + TO-3 (idle-based) replace it so a response may run for hours while bytes flow. Changelog item.
TO-5	Client header read (slowloris)	10 s from first byte until head complete; then `max(keepalive_timeout_seconds, 30)` s idle between requests	client leg	MUST - on expiry simply close (408 responses to attackers waste writes)
TO-6	Client write stall	30 s zero forward progress	client leg	MUST → F13
TO-7	Client body inter-chunk idle	`max(keepalive_timeout_seconds, 30)` s	client leg mid-body	MUST
TO-8	WS idle	server-initiated ping every 30 s, close 1001 after 2 missed pongs; bridge activity resets	WS bridge	SHOULD, on by default; config `websocket_ping_interval_seconds` (0 = disabled = legacy). Verified: molohttp never calls `initiateAutoPing` today, so WS tunnels currently live until TCP death.
TO-9	Shutdown drain	8 s	all in-flight	MUST - bounded by systemd `TimeoutStopSec=10`

TO-10 (MUST) All timers run on the channel's own event loop (Scheduled<Void>, cancelled on completion) - no global timer wheel, no locks.

E3. Circuit breaker port

Preserve exactly: per-upstream-URL state; recordFailure on transport error or relayed 5xx; 5 consecutive failures → cooldownUntil = now + 30 s; recordSuccess (status < 500) resets the failure count but not an active cooldown; cooldown cleared lazily at selection time; while cooling, excluded from selection; all-cooling → F14.

CB-1 (MUST) Per-upstream final class BreakerState: Sendable with two ManagedAtomic (swift-atomics - already available transitively via swift-nio; add as a direct dep): consecutiveFailures: ManagedAtomic<Int>, cooldownUntilEpochSec: ManagedAtomic<Int64>, plus activeConnections: ManagedAtomic<Int> and a lastError in a tiny NIOLockedValueBox<String> (cold path).
CB-2 (MUST) The [upstreamURL: BreakerState] map is immutable, built at config load/reload, published via the same snapshot mechanism as the host index. Per-request access is a dictionary read on an immutable map - zero synchronization. States for upstreams surviving a reload SHOULD carry over by URL key so a reload does not reset open breakers.
CB-3 (MUST) Lock-free ops: recordFailure → wrappingIncrementThenLoad(.relaxed), if >= 5 store now+30 cooldown and reset failures to 0; recordSuccess → store 0; isAvailable(now) → load cooldown, if <= now and > 0 CAS to 0 (lazy reset, exactly-once winner) → available.
CB-4 (decision) Rejected: per-event-loop counters (changes "5 consecutive" semantics - unacceptable drift); single NIOLockedValueBox per upstream (correct but two relaxed atomics are cheaper on the hot path). The benign race (two loops both hitting failure #5) is harmless - same cooldown value. Chosen: CB-1 atomics.
CB-5 (MUST) LB parity: least_connections reads the activeConnections atomic; round-robin counter is one ManagedAtomic<Int> per site (keyed as today); ip_hash/random stateless.

E4. Retries

RT-1 (SHOULD, config retry_connect_failures, default false) A single transparent retry against the next healthy upstream iff all of: (a) the failure is F3/F4/F11/F12-second-failure (zero request bytes reached any upstream application); (b) the method is idempotent (GET/HEAD/PUT/DELETE/OPTIONS/TRACE) - keep idempotent-only in v1 for simplicity; (c) at most 1 retry; (d) the failed upstream's breaker failure recorded before the retry selects.
RT-2 (MUST) No retry after any request byte written to an upstream socket, and never on F5-F9.
Recommendation: include in v1, off by default. The streaming design makes "zero bytes sent" a trivially checkable state-machine phase (retry legal only before sendingRequestHead); ~50 lines in the connect phase; off by default preserves exact v0.2.0 behavior.

E5. Security hardening

SEC-1 (MUST) Hop-by-hop stripping per D3.1 (including Connection-nominated headers - the current fixed-list-only behavior misses attacker-nominated hop-by-hop headers).
SEC-2 (MUST) Smuggling: TE+CL, TE other than a single final chunked, multiple/conflicting CL → 400 + Connection: close. Run the decoder with lenient modes disabled (never downgrade). Because we always re-frame toward the upstream (never forward the client's TE), the classic TE.CL/CL.TE desync cannot cross the proxy even if parsing were lenient - re-framing is the structural defense, strict parsing is defense-in-depth.
SEC-3 (MUST) Header injection via CR/LF/NUL: inbound values are clean (llhttp). Outbound is the real risk - HTTPRequestEncoder does not validate header values. Validate every header name/value the data plane synthesizes or takes from config (header_up add/set, host rewrite, XFF/XRI/XFP/XFH) against \r, \n, \0; violations dropped + warn-logged at config load (config source) or replaced with "invalid" (runtime source).
SEC-4 (MUST) Request-line length cap 8 192 bytes → 414 + close.
SEC-5 (MUST) Client head limits: total header bytes ≤ 16 384 → 431 + close; header count ≤ 128 → 431.
SEC-6 (MUST) Same limits on the upstream response head → F9 (502).
SEC-7 (MUST) Slowloris: TO-5 (10 s head completion) + SEC-8 + backlog 4096. Zero-complete-request connection hitting TO-5 → closed, no response, slow_client_closed metric.
SEC-8 (SHOULD, config max_connections_per_ip, default 0 = unlimited) Per-IP concurrent-connection cap at accept time (per-loop shard or a single NIOLockedValueBox<[IPAddr: Int]> touched once per accept/close). Needed because the request-based rate limiter cannot see a flood of connections that never complete a request. Over cap → immediate close, no response, sampled warn.
SEC-9 (MUST) URL handling: forward the raw request-target byte-for-byte (D9.3); never percent-decode/normalize before proxying. Reject targets with raw control bytes or spaces (llhttp does) with 400. File-server path resolution decodes exactly once then applies securePath (with the sibling-prefix fix).
SEC-10 (MUST) Ordering per request, before any upstream dial or checkout: host lookup → auth → rate limit → breaker/LB selection → connect. An unauthenticated or rate-limited request never consumes an upstream connection.
SEC-11 (MUST) Bcrypt verification (CryptoUtilities.verifyPassword - forks python3, ~50-200 ms, memoized in the 1024-entry NIOLock cache) MUST NOT run on an event loop. Data-plane flow: on reaching auth, (a) set autoRead = false and withhold read() so no body bytes accumulate; (b) call the existing async verifyPassword (cache hits return fast, misses hop to DispatchQueue.global() as today); (c) on resume hop back to the loop (eventLoop.execute), re-enable reads on success or write 401 and close (body in flight). The SHA256:SALT fast path (non-bcrypt) is pure CPU and MAY run inline.
SEC-12 (MUST) RateLimitBucket (actor) awaited the same way: reads suspended during the hop, resumed on the loop. StatsMonitor.recordRateLimit(allowed:) called for every rate-limited-site request. SHOULD (v2 follow-up, not v1): per-loop buckets to remove the actor hop - semantics change (per-loop limits) must be flagged.
SEC-13 (MUST) 401 keeps the current shape: no WWW-Authenticate header, RFC 9457 body, connection kept alive unless a request body is mid-stream (then Connection: close).

E6. Graceful shutdown and config reload

GS-1 (MUST) SIGTERM/SIGINT (exactly-once) → in order: (1) close both listening sockets (via ServerQuiescingHelper); (2) idle keep-alive client connections closed immediately, in-flight requests get Connection: close on their response and close at completion; (3) WS bridges closed 1001 goingAway both legs immediately; (4) wait for in-flight to reach zero or TO-9 (8 s); (5) force-close survivors; (6) flush AccessLogger, stop stats sampler, shut down upstream pools, stop ACME renewal, call acmeClient.shutdown() (closing the v0.2.0 gap).
GS-2 (MUST) The 8 s drain leaves 2 s inside TimeoutStopSec=10 for pool shutdown and log flush; exit 0 even if stragglers were force-closed.
GS-3 (MUST) SIGHUP reload (repeatable, serialized by ConfigManager): each request captures an immutable snapshot (Site config, breaker-state map reference, host index) at dispatch (LiveHostIndex design). A reload swaps the published snapshot; zero effect on in-flight requests including their timeout values and header_up rules. WS bridges under the old config keep running. Breaker state carry-over per CB-2. Listen addresses/limits/compression remain startup-bound.
GS-4 (MUST) Pools for upstreams removed by a reload drain lazily: in-flight checkouts complete, idle connections closed, pool destroyed when the last lease returns or after idle timeout.

E7. Observability

OB-1 (MUST) Exact current access-log line format, byte-compatible: <remoteAddr> - - [<dd/MMM/yyyy:HH:mm:ss Z>] "<METHOD> <uri> HTTP/1.1" <status> - <%.4f>s\n. Port AccessLogger unchanged (buffered, 32 KiB threshold, dedicated 1-thread pool, per-second cached timestamp, flush on shutdown).
OB-2 (MUST) Log at response completion (last byte flushed or connection closed), not head-commit, so duration covers streaming time. Compat note: for long streams duration grows - expected.
OB-3 (MUST - behavior change, recommended) Error paths get access-logged. Today AccessLogMiddleware uses .map so any thrown error produces no log line - an operational blind spot. The new plane logs every request that reached a complete head, with the status actually sent (or 499 disposition). Changelog item (log volume and status distribution change).
OB-4 (MUST) Client-abort disposition uses status 499 (nginx convention) in the access log and stats; never sent on the wire.
OB-5 (MUST) Per-request, call StatsMonitor.shared.record(host:method:status:durationMs:bytesSent:bytesRecv:) with streamed byte counts (actual bytes written to/read from the client), replacing the buffered readableBytes approximation. Skip internal paths as isInternalPath does today; keep connectionOpened/Closed pairing including on error paths.
OB-6 (MUST) Feed real efficiency counters (today hardcoded 0): passThroughExchanges, bufferedExchanges, bytesPassedThrough; zero_copy_ratio = passThroughExchanges / totalExchanges. Keep the JSON keys for dashboard compat; document the semantics. Additive new counters: upstream_connect_errors, upstream_timeouts, breaker_open_total, client_aborts, slow_client_closed, retries_attempted, tls_handshake_failures.
OB-7 (MUST) Repeated upstream failures sampled: per upstream URL, warn the 1st failure of a burst, then ≤ 1 warn per 5 s with a (suppressed N) count; breaker open/close transitions always log unsampled. Current code warn-logs every failure (500 lines/s under a dead upstream at 500 RPS). Implementation: last-logged epoch + suppressed counter in BreakerState.
OB-8 (SHOULD) Per-request debug tracing keyed by a generated request id on debug level only; not added to any wire header in v1.

E8. Chaos / abuse test list

Harness: slowhttptest, nc/socat, a programmable raw-TCP mock upstream, wrk overlays.

#	Test	Expected
CT-1	Slowloris headers (`slowhttptest -c 1000 -H -i 10`)	each conn closed at TO-5 (10 s), no response; `slow_client_closed` increments; healthy traffic unaffected; FDs return to baseline
CT-2	Slow POST body (`slowhttptest -B`)	closed at TO-7; upstream (if dialed) destroyed per ER-2
CT-3	Client never reads response	backpressure suspends upstream reads (ER-7); after 30 s zero progress → F13, both closed, memory bounded, `499`
CT-4	Client RST mid-request-body	F1: upstream destroyed, breaker untouched, `499`, no leaked pool lease
CT-5	Mid-transfer client kill under load	F2 per conn; other conns unaffected; zero crash, `client_aborts` matches
CT-6	Upstream connect refused	F3: 502, breaker +1; 5 requests → open, 6th gets F14 502 without a SYN (verify via tcpdump); after 30 s re-admit
CT-7	Upstream accepts, never sends head	F6: 504 after TO-2 (set `read_timeout_ms=2000` in test), breaker +1, conn destroyed
CT-8	Upstream garbage head	F9: 502, warn sampled, no raw payload in logs
CT-9	Upstream RST mid-body	F7: client sees truncation (no chunked terminator / short body), `upstream_body_truncated`, breaker +1
CT-10	Upstream stalls mid-body	F8: same client outcome as CT-9; 504 never sent (head committed)
CT-11	Huge request header (20 KB single / 200 headers)	431 + close (SEC-5), memory bounded
CT-12	Huge request line (16 KB URI)	414 + close (SEC-4)
CT-13	TE+CL smuggling corpus	all rejected 400 + close (SEC-2); logging mock upstream receives zero smuggled second requests
CT-14	CR/LF in config `header_up` value	rejected/warned at config load; upstream never receives the injected header (SEC-3)
CT-15	100k idle WS	with TO-8 on: closed 1001 after ~90 s; FD/memory return to baseline. With ping interval 0: legacy held-open behavior documented
CT-16	WS upstream dies mid-bridge	client receives 1001; bridge freed (no leaked channels)
CT-17	Per-IP connection flood (`max_connections_per_ip=100`)	conns 101+ closed at accept (SEC-8); other IPs unaffected
CT-18	Stale pool connection	F12: transparent single re-dial, request succeeds, `stale_pooled_connection` debug, breaker clean
CT-19	SIGTERM under load	GS-1: in-flight complete with `Connection: close`, idles/WS dropped, exit within 10 s exit code 0, access log fully flushed
CT-20	SIGHUP storm under load (10×/s)	zero request failures from reload (GS-3); RSS stable; breaker state survives (CB-2)
CT-21	Log-flood protection (dead upstream + 1000 RPS/60 s)	warn lines for that upstream ≤ ~14 with accurate `(suppressed N)` (OB-7); access log records every request (OB-3)
CT-22	bcrypt auth under load (200 concurrent first-time creds)	event loops never blocked (health P99 flat during the burst - the real regression test); subprocess concurrency bounded; repeats hit the memo cache

E9. Summary of deliberate behavior changes vs v0.2.0 (changelog)

504 for upstream connect/head timeouts (was 502) - ER-3.
Whole-response 120 s deadline removed, replaced by idle-based read timeouts - long downloads/SSE now work - TO-4.
Errors and client aborts access-logged (499 disposition) - OB-3/OB-4.
Upstream-failure warn logs sampled - OB-7.
WS idle ping/close (configurable, was: never reaped) - TO-8.
Connection-nominated hop-by-hop headers stripped - SEC-1.
Unchanged for compat: 502 on all-upstreams-cooling (F14), 5xx relay + breaker semantics, breaker thresholds (5/30 s), 401 without WWW-Authenticate, no retries by default, access-log line format.

Part F - Testing, Verification, Config/Deploy, Rollout

F1. Test strategy (layered)

F1.1 Layer A - Swift unit/integration tests (extend `Tests/MoloHTTPTests/`)

New files: DataPlaneTests.swift, ProxyStreamingTests.swift, WebSocketProxyTests.swift. Pattern: each test boots the new NIO data plane on an ephemeral port (bind :0, read the bound port) and a local NIO test upstream (TestUpstream fixture) on another ephemeral port. The fixture can: echo request head+body as JSON, serve a fixed byte count in chunks, emit chunked+trailers, count accepted TCP connections, delay responses, return arbitrary status, act as a WS echo server. In-process, no docker, runs in make test.

Test cases (keep the existing 50 green; add these):

#	Test	Assertion
1	MissingHost400	no/empty Host → 400 "Missing Host header"
2	UnknownHost404	unconfigured host → 404 "No site configured for host: X"
3	ExactHostBeatsWildcard	`a.example.com` + `*.example.com`; exact hit for `a`, wildcard for `b`
4	WildcardSingleLabelOnly	`*.example.com` matches `x.example.com`, NOT apex, NOT `a.b.example.com`; config-order ties; `:port` stripped
5	XForwardedHeaders	echo upstream sees XFF appended as an additional line, XFP replaceOrAdd, XFH = incoming Host, XRI added, `connection: keep-alive`
6	HostRewriteModes	nil → client Host; `{upstream_host}` → upstream host; literal → literal; applied after header_up
7	HopByHopRequestStripping	`te`/`trailers`/`keep-alive`/`proxy-authorization`/`upgrade`/`accept-encoding` never reach upstream; upstream responds identity
8	HopByHopResponseStripping	upstream `keep-alive`/`proxy-authenticate`/`te`/`upgrade` never seen by client; framing owned by the new layer
9	HeaderUpRules	add → set → remove in order, before host rewrite
10	100MBStreamingConstantMemory	100 MB in 64 KB chunks; correct hash; `VmHWM` delta < 64 MB; instrument buffered-but-unwritten high-water < 4 MB
11	ChunkedWithTrailersRelay	chunked + trailers relayed (or pins the decided drop behavior)
12	KeepAliveConnectionReuse	50 sequential requests → upstream `acceptedConnections == 1`
13	ConnectionClosePropagation	client `Connection: close` → client socket closed, upstream pooled; upstream `Connection: close` → next request opens a new upstream, no desync
14	502OnConnectionRefused / 504OnTimeout	connect refused → 502 "Upstream connection failed"; head/connect timeout → 504 (behavior change ER-3)
15	Upstream5xxRelayedVerbatim	503 relayed as 503 but breaker-counted; 3xx relayed, never followed
16	CircuitBreakerOpenClose	5 failures → immediate 502 "No upstream available" without touching upstream; after injected-clock cooldown, re-admit + failures reset
17	RateLimitConfigStatusAndBody	`rps:1, burst:1, status:429, body:"slow down"` → 2nd request 429 + body; key_by ip/header/global
18	BasicAuth401	wrong pw / bad base64 / missing header → 401, no `WWW-Authenticate`; correct passes; first-matching-rule path scoping
19	APIKeyAuth	`x-api-key` missing → 401; matching → 200
20	WebSocketEchoThroughTunnel	`websocket: true`; upgrade → first-upstream dial; text+binary echo; raw tunnel relays all handshake headers verbatim
21	WebSocketClosePropagation	(raw-tunnel v1) upstream close code propagates end-to-end; non-WS host → 403/426; upstream dial fail → 1011
22	SlowlorisIdleTimeout	partial head, stall → closed at TO-5 (run with low override to keep the suite fast)
23	TEplusCLRejected	raw socket with both TE and CL → 400 + close, never forwarded; CL mismatch and obs-fold rejected
24	Expect100Continue	`Expect: 100-continue` + body → 100 Continue then final; body reaches upstream intact
25	HEADNoBodyAnd304NoBody	HEAD → headers only, zero body bytes; upstream 304 relayed with no body/chunked framing
26	ClientDisconnectCancelsUpstream	client aborts mid-download → upstream sees close within 1 s; pool gauge returns to baseline
27	LoadBalancingPolicies	round_robin rotation, ip_hash determinism (djb2), least_connections lowest-active; `weight` ignored
28	ConcurrentLoadSmoke	32 tasks × 100 requests, mixed keep-alive/close; zero errors, bodies intact, upstream conns ≤ pool limit

F1.2 Layer B - Protocol compliance / fuzz (`scripts/proxytest/`)

h2spec is h2-only, N/A. Harness: scripts/proxytest/proxytest.py (Python 3, argparse, per project convention), raw-socket cases against a locally started instance on ephemeral ports, with go-httpbin (static binary / docker) as the upstream for semantic cases.

python3 scripts/proxytest/proxytest.py --target http://127.0.0.1:8080 --host test.local --suite all
python3 scripts/proxytest/proxytest.py --suite smuggling --verbose

Suites: smuggling (CL.TE, TE.CL, TE.TE obfuscated, Transfer-Encoding: xchunked, duplicate CL, chunk-size overflow, negative CL, header injection via \r, oversized head - assert reject + zero smuggled second request via a counting upstream); methods (curl --http1.1 matrix; TRACE/CONNECT → 501/405; internal-path OPTIONS handled, proxied OPTIONS passed); semantics (go-httpbin /status/{code} incl. 5xx-verbatim/3xx-not-followed, /stream-bytes/N, /gzip+/deflate no-double-compress, /redirect/3 not followed, /headers XFF, chunked /drip); keepalive (pipelining, Connection: close, idle timing); hurl (optional, graceful skip like make lint).

Every case self-contained: harness starts the instance, runs, tears down, exits non-zero on failure (CI gate).

F1.3 Layer C - Load / performance / soak

Reuse the documented wrk tiers and scripts/loadtest_mass.py. Acceptance thresholds vs current baselines (v0.2.0 + async fixes):

Metric	Baseline	Threshold
Health 4t/100c	13,391	≥ 13,000 (control plane unchanged; must not regress)
Health 8t/500c	12,825, 0 timeouts	≥ 12,000, 0 timeouts
Health 8t/500c P99	806 ms	≤ 900 ms
rsearch proxy 4t/100c	529-754	≥ 529 (target: improve - streaming)
rsearch proxy P99	305-421 ms	≤ 421 ms
retoor proxy	187	≥ 187, 0 timeouts
Memory after ~270k req	141 M stable	≤ 180 M, no drift
Crashes/backtraces	0	0 (hard fail)
92-domain aggregate	4,358	≥ 4,358

Health RPS is the strongest guardrail (Hummingbird still serves /molohttp/*). Proxied RPS is expected to improve or match (streaming vs buffering) - a regression is a rewrite defect.

Long-soak leak test - new make soak:

scripts/soak.sh --duration 3600 --ws-connections 200 --http "wrk -t4 -c200 -d3600s https://rsearch.app.molodetz.nl/"

Sampled once/min: systemctl show molohttp -p MemoryCurrent (flat ±10 % after warm-up); ls /proc/<pid>/fd | wc -l (plateaus, not monotonic); journal scan for crash markers (empty). The persistent-WS arm targets WS memory growth (200 idle+chatty connections held the whole run; pending/tunnel buffer growth watched).

F2. Compatibility verification - observable-behavior checklist (92 prod domains)

Config/routing: decode the real /etc/molohttp/config.json + .env.json without error (Swift test fixture / on the box pre-cutover); host precedence exact > wildcard(single-label) > config-order; disabled 404; case-sensitive; :port stripped; 400/404/401/rate-limit statuses identical.

Headers on proxied responses: XFF/XRI appended as extra lines (quirk #1); XFP https/http; XFH = client Host; Server: molohttp; Date on HTTPS; hop-by-hop sets stripped.

MUST-PORT list - global middleware behaviors currently applied to proxied traffic via Hummingbird that DO NOT automatically exist in the raw data plane; each reimplemented in the new pipeline with a pinning test:

SecurityHeaders add-if-absent (never overwrite upstream x-frame-options etc.).
CORS add-if-absent; OPTIONS preflight only for internal /molohttp/ paths; proxied OPTIONS pass through; upstream ACAO never overwritten.
Compression predicate (skip if unsupported / content-encoding present / no content-type, else content-type substring match); never double-compress precompressed upstreams - pipeline-placement design dependency (F5 risk #3): the HTTPResponseCompressor sits between the response-relay write and the encoder, sees the relayed .head before body chunks stream.
Access-log line format identical for proxied requests.
Stats recording: every proxied request advances StatsMonitor counters + per-host RPS ring.

Other: ACME challenge route precedence; /favicon.ico, /molohttp/* unchanged (Hummingbird); WS gating/first-upstream; file-server MIME/ETag/range/304/listing/hide/securePath.

F3. Build/verify gates

Keep (all green): make verify (warnings-as-errors, zero), make test (50 existing + ~28 new), make build, make lint. Add:

proxytest: build
	python3 scripts/proxytest/proxytest.py --suite all --start-instance
soak: build
	scripts/soak.sh --duration 3600 --ws-connections 200

Swift 6 strict concurrency for new channel-handler code: handlers are event-loop-confined reference types, not Sendable; follow the repo pattern final class …: @unchecked Sendable only where a NIOLock or loop confinement provides safety (documented in the Exceptions table). Prefer loop confinement (eventLoop.assertInEventLoop()) over locks for per-connection state; cross-loop state behind NIOLock (or the CB atomics). nonisolated(unsafe) for non-Sendable NIO captures across the async boundary, as the existing TLSServer mitigation does. -warnings-as-errors rejects sloppy suppression.

F4. Rollout plan (production box)

Reuse/extend scripts/switchover.py; add scripts/smoke_domains.py (extracts hosts from config.json, curls each with --resolve host:PORT:127.0.0.1).

Step 0 - build + rollback binary:

make verify && make test && make proxytest
make build
sudo cp /usr/local/bin/molohttp /usr/local/bin/molohttp.prev
.build/release/molohttp --version

Step 1 - side-by-side canary on 8080/8443 against real config/upstreams: copy .env.json, override listen ports to 8080/8443, run molohttp run --config /etc/molohttp/config.json --env /tmp/molohttp-canary.env.json &. (Requires making the HTTPS port configurable - see F5 note; otherwise the canary tests HTTP:8080 + routing + proxy only, HTTPS validated post-cutover.)

Step 2 - smoke all reachable domains via the canary: curl -sf http://127.0.0.1:8080/molohttp/health; python3 scripts/smoke_domains.py --port 8080 --scheme http; python3 scripts/proxytest/proxytest.py --target http://127.0.0.1:8080 --suite all; wrk -t4 -c100 -d10s http://127.0.0.1:8080/molohttp/health. Diff per-domain status + response-header set vs current prod (443). Abort cutover on any mismatch. Stop the canary.

Step 3 - systemd cutover (per CLAUDE.md): systemctl stop, cp .build/release/molohttp /usr/local/bin/, copy resources, reset-failed, start, sleep 3.

Step 4 - post-cutover verification: health {"status":"ok"}; https://molodetz.nl/ 200; https://admin.molodetz.nl/ 200; smoke_domains.py --scheme https all 200; WS smoke through devplace.net; systemctl reload + re-check health (SIGHUP); curl /molohttp/api/stats populated; journal crash scan empty; systemctl status active, memory sane, restart counter 0. Re-verify explicitly: admin UI login + one write action; stats/dashboard live WS; SIGHUP reload; ACME challenge path (curl /.well-known/acme-challenge/testtoken → 404 proves routing precedence intact).

Step 5 - rollback (single cp + restart): systemctl stop; cp /usr/local/bin/molohttp.prev /usr/local/bin/molohttp; reset-failed; start; verify health. Trigger on: health failing after 10 s, any crash/backtrace, memory > 1 GB, or a domain that was 200 on the canary returning 5xx in prod. Keep molohttp.prev until a full soak passes post-cutover.

F5. Risk register (top 10, ranked)

#	Risk	Mitigation test
1	WebSocket proxying subtleties (raw-tunnel 101 relay, leftover-bytes flush, close propagation, gating)	Tests 20/21 + soak persistent-WS arm; CT-15/16
2	Streaming body edge cases (chunked/trailers, HEAD/304 no-body, empty body, exact CL, client abort mid-stream)	Tests 10/11/13/25/26; `--suite semantics` drip/stream-bytes
3	Compression × streaming pipeline placement (DESIGN DEPENDENCY) - the compressor needs the `content-type` response header known before the body streams; it sits between the relay-write handler and the encoder; must not double-compress precompressed upstreams	proxy identity `text/html` with `Accept-Encoding: gzip` → valid gzip; proxy `content-encoding: gzip` upstream → single encoding, valid body; assert compressor inserted upstream of the relay-write handler
4	Stats/dashboard breakage (StatsMiddleware was HB middleware; proxied traffic now bypasses it)	fire N proxied requests, assert `StatsMonitor.snapshot()` += N; post-cutover `curl /molohttp/api/stats`
5	Admin UI/internal routes on the new seam (routing precedence between control plane and raw data plane)	`/molohttp/*` and `/molohttp/admin/` win over host dispatch for every Host; post-cutover admin login + one write
6	ACME challenge route priority (must beat host dispatch on both listeners or renewal breaks weeks later)	challenge path on a proxy host served by AcmeChallengeHandler not proxied; post-cutover 404-on-missing-token
7	Keep-alive desync (pooled reuse + streaming; half-read body or mishandled `Connection: close` poisons a pooled conn)	Tests 12/13/28; `--suite keepalive`; soak
8	Config reload race (SIGHUP live index swap mid-flight)	hammer proxy while looping `hostIndex.replace`; assert no crash / no cross-host contamination; post-cutover `systemctl reload` under load
9	Request smuggling via the new parser (rewrite owns TE/CL validation)	Test 23 + `--suite smuggling`; assert zero smuggled second request
10	Memory growth under WS/streaming load (unbounded pending / relay buffers)	`make soak` 30-60 min, RSS flat ±10 %, FD plateau; Test 10 memory-delta

Secondary: file-server ETag process-seed behavior if touched (decide keep vs deterministic and pin); ignored config fields (weight/tls_server_name/health_checks) stay ignored - don't accidentally implement them.

Flagged decisions: (a) HTTPS port hard-coded 443 - make it configurable so the canary also tests TLS; (b) file-server ETag process-seed random - decide keep-vs-fix; (c) compression-pipeline-placement (risk #3) is a genuine design dependency.