Receive Buffers and Flow Control in Rust Multiplexer

Overview

A 1 MiB receive window is used as the baseline scenario. When a peer sends 256 KiB, 768 KiB of credit remains. The receiver removes the buffer from its queue and begins writing it to a socket. If the socket stalls, the queue becomes empty while the buffer still occupies 256 KiB. Returning credit at that point would allow replacement data to arrive while the write is still pending. BibaVPN keeps those bytes accounted for until write_all() completes. This mechanism is part of the TCP multiplexer in an experimental Rust tunnel that carries several logical streams through a shared WebSocket connection.

Receive Path Architecture

The conceptual receive path separates concerns between independent pumps and shared record dispatch. All streams still share the outer TCP transport, but each stream maintains its own receive state and dedicated pumps for read and write directions.

Record Structure

Each mux record begins with a nine-byte header containing:

Field	Type	Description
stream_id	u32, big-endian	Identifies the logical stream
flags	u8	Operation type (OPEN, DATA, CLOSE, RST, WIN)
payload_length	u32, big-endian	Length of the payload
payload	-	The actual payload data

The decoder validates the declared payload length and rejects any trailing bytes. After decoding, the stream ID directs the record to the appropriate dispatcher.

Decoding and Dispatch

Once decoded, the stream ID determines where the record should be sent. Writing directly to a shared destination socket from the shared reader would cause every stream to depend on that socket's speed. Instead, the implementation preserves per-stream receive state and uses separate pumps for bidirectional traffic.

Credit Management and Flow Control

The system employs an optional, negotiated byte-credit extension that governs how much data can remain outstanding before credit must be returned.

Negotiated Credit Mode Example

In a worked example running in negotiated credit mode, no additional data arrives during the sequence. The empty queue triggers publication of accumulated credit after the write completes. The relevant operations in tcp_mux_flow.rs follow this order:

write.write_all(&data).await?;
let len = data.len();
drop(data);
flow.consumed(len);

The window-update logic then batches consumed bytes and publishes credit when either:

• The accumulated amount reaches one eighth of the receive limit, or
• The queue becomes empty

While the socket write is pending, the buffer stays charged to the stream.

Session-Level Limits

Each admitted stream reserves its local receive allowance against a 64 MiB logical receive budget. The following table summarizes the reservation limits:

Local receive window	Reservations that fit in 64 MiB
1 MiB	64
2 MiB	32
3 MiB	21
4 MiB	16

These values represent admission-budget calculations. Additional process memory overhead includes retained allocations, output queues, TLS state, tasks, and other runtime components. The bytes::Bytes type tracks allocation details, and because a short slice can retain the backing allocation, counting only slice.len() can understate the actual memory retained by queued data.

Memory Accounting Considerations

Memory management requires careful tracking of allocations. A backing allocation of 8 KiB is allocated for the slice, with space reserved for the header/padding and payload slice plus any unused capacity. Since keeping the slice prevents the underlying allocation from being deallocated, simply measuring slice.len() underestimates total memory usage.

The receive implementation incorporates compaction and backing-allocation accounting alongside its logical byte limits to ensure accurate resource tracking.

Closing Semantics

Closing introduces a lifetime-related challenge. In negotiated mode, the CLOSE operation is directional-the other pump continues operating independently. In legacy mode, the older behavior persists where both directions are closed upon receiving a close signal.

This distinction affects how the system handles client responses and ensures that per-stream accounting does not alter TCP's ordered delivery guarantees. Even if packet loss occurs on one outer connection, it can still delay data for every logical stream carried by that connection. Separate receive queues handle slow destination sockets within the application, while outer transport loss remains shared across all streams.

Key Implementation Points

For code review, focus on these key areas:

• Flow::enqueue
• Flow::receive
• Flow::consumed
• The downlink write loop

Together, these functions reveal when a buffer enters the accounting system, who owns it while a write is blocked, and when a peer receives permission to send more data. Source code, protocol documentation, and tests are available on BibaVPN on GitHub.