Adding WebRTC to py-libp2p: Two Runtimes, Three Specs, and One Unforgiving Private Slot
DEV Community

Adding WebRTC to py-libp2p: Two Runtimes, Three Specs, and One Unforgiving Private Slot

The problem: why a Python libp2p wants WebRTC at all

libp2p is a modular network stack: identity, transports, security, multiplexing, routing, and pub/sub, each swappable. Its Go and JS implementations have had WebRTC for a while. Python did not - issue #546 tracks the gap.

Why care? Three reasons the ecosystem keeps coming back to:

  • NAT traversal for real users. Most consumer machines sit behind at least one NAT, often more. WebRTC's ICE + STUN + TURN dance is the only widely-deployed answer that browsers speak natively. If your Python peer wants to talk to a browser peer - or two NATed Python peers want to talk directly with only a light-touch relay - WebRTC is the door.
  • Native multiplexing. WebRTC data channels give you real per-stream framing over SCTP. That means the transport itself carries a "muxer," so you can skip the Yamux / Mplex upgrade step that a TCP-based libp2p connection needs.
  • Server-to-browser without an HTTP server. With /webrtc-direct, a Python node can accept an inbound connection from a browser peer purely via a UDP socket + a certificate hash embedded in the multiaddr. No signaling server, no HTTPS termination, no TURN unless traversal forces you there.

That's the "why." Now for the "how" - and specifically, why "how" turned into 6,700 lines of Python that took four review rounds to get right.

What actually landed

The PR body says it, but here it is in one paragraph so you know the surface area:

  • WebRTCDirectTransport (/webrtc-direct) and WebRTCPrivateTransport (/webrtc, relay-signaled).
  • Spec-compliant data-channel framing - uvarint length-prefixed protobuf, with the full FIN / FIN_ACK / STOP_SENDING / RESET state machine.
  • In-band data channels for application streams; the Noise handshake channel (id=0) stays pre-negotiated per spec.
  • Signaling with the bilateral ICE_DONE fix from libp2p/specs#585.
  • Noise XX with a prologue that binds the handshake to the DTLS certificate fingerprints.
  • ECDSA P-256 cert generation with multihash / multibase fingerprint encoding.
  • A DTLS cert pin that actually pins the cert. More on that shortly.
  • A trio ↔ asyncio bridge, because py-libp2p is trio and aiortc is asyncio.
  • The final tests count: 153 / 153 passing, including a real RTCPeerConnection ↔ RTCPeerConnection loopback test that exercises framing, chunked writes over the 16 KiB SCTP ceiling, and the SDP-fingerprint canary.

What's out of scope, and this matters: browser interop is not supported here; go / js /webrtc-direct interop is not wired (they reconstruct the SDP from STUN USERNAME, we don't have that hook yet); the /webrtc dial path is a listener + signaling skeleton only. All of that is deliberate. This is v1. v2 (specs#715) is where browser dial actually happens, and we wanted the v1 skeleton in the tree so it stops being vaporware.

Architecture in one picture

Two flavors of WebRTC show up in py-libp2p, and it took me a while to keep them straight. Here they are, side by side:

flowchart LR
    subgraph Direct["/webrtc-direct (server-reachable)"]
        A[Client] -- SDP offer built<br/>from server multiaddr --> B[Server<br/>publishes cert hash]
        A <-. DTLS + data channels .-> B
    end
    subgraph Private["/webrtc (both peers NATed)"]
        C[Peer A] --- R[Relay v2]
        D[Peer B] --- R
        C -. signaling over relay:<br/>SDP + ICE + ICE_DONE .-> D
        C <-. direct DTLS + data channels .-> D
    end

Two transports, one shared substrate: certs, SDP, Noise binding, stream framing, and the trio↔asyncio bridge. Only signaling differs.

Inside a single peer, the runtime layout looks like this:

flowchart TB
    subgraph Trio["Trio world (py-libp2p)"]
        T1[Host / Swarm]
        T2[WebRTCConnection]
        T3[WebRTCStream × N]
        T4[Application handlers]
    end
    subgraph Bridge["AsyncioBridge<br/>(background daemon thread)"]
        BR[asyncio event loop]
    end
    subgraph Asyncio["Asyncio world (aiortc)"]
        A1[RTCPeerConnection]
        A2[RTCDataChannel × N]
        A3[DTLS / ICE / SCTP]
    end
    T1 --> T2 --> T3
    T2 -- run_coro --> BR
    BR -- runs on --> A1
    A1 --> A2 --> A3
    A2 -. on_data via trio_token .-> T3

The interesting seam is the arrow labelled trio_token: aiortc calls into us from an asyncio thread, and we need those callbacks to hand data to trio memory channels safely. If we skip the token, we get RunFinishedError and worse. That code lives in stream.py:382-410 - I'll come back to it.

Deep dive #1 - framing, or "why bare protobuf was the wrong wire"

If you read the first draft of this PR, application-level writes were serialized as bare protobuf Message bytes and shoved straight into data_channel.send(...). That "works" on the happy path for small payloads. It's also wrong for two independent reasons - and once you understand both, you understand why the spec is written the way it is.

Reason 1: SCTP has a 16 KiB message ceiling

The libp2p WebRTC spec caps each data-channel message at 16,384 bytes. Below that, you're fine. Above, aiortc will happily hand SCTP an oversized message that gets fragmented in ways your peer can't reassemble, or your peer will drop it, or SCTP itself will complain - the failure modes vary by implementation, which is worse than a clean rejection.

The fix is to chunk on the sender's side and let the read side see one wire message per SCTP message.

Reason 2: bare protobuf isn't self-delimiting inside a stream frame

Even inside the 16 KiB limit, a Go or JS peer decoding your message needs a length prefix. The spec is explicit: each SCTP message is one uvarint length prefix + one protobuf Message. Not "raw protobuf." Not "concatenated protobufs." Exactly one framed pair per SCTP datagram.

Here's the on-wire shape:

┌───────────────────┬──────────────────────────────────────────┐
│uvarint length     │protobuf Message (flag? + message? bytes)│
└───────────────────┴──────────────────────────────────────────┘
← up to 16 KiB total, hard ceiling ──────────────────────────→

The uvarint itself is straightforward - 7 bits of payload per byte, top bit for continuation, capped at 10 bytes so it fits uint64:

# libp2p/transport/webrtc/_varint.py
def encode_uvarint(value: int) -> bytes:
    if value < 0:
        raise ValueError("uvarint cannot encode negative values")
    buf = bytearray()
    while value > 0x7F:
        buf.append((value & 0x7F) | 0x80)
        value >>= 7
    buf.append(value & 0x7F)
    return bytes(buf)

The sender side of a stream then becomes:

# libp2p/transport/webrtc/stream.py
async def write(self, data: bytes) -> None:
    if self._state == StreamState.RESET:
        raise WebRTCStreamError("Stream was reset")
    if self._write_closed:
        raise WebRTCStreamError("Write side is closed")
    offset = 0
    while offset < len(data):
        chunk = data[offset:offset + MAX_PAYLOAD_SIZE]
        msg = Message(message=chunk)
        await self._send_message(msg)
        offset += len(chunk)

MAX_PAYLOAD_SIZE is chosen so that even with the largest reasonable varint length prefix plus protobuf tag bytes, the full framed wire message stays under 16 KiB. The chunk boundary is on the sender; the reader just decodes one frame per SCTP message.

And this is worth calling out - there's one spec detail that's easy to miss: The spec allows a message to carry both data and FIN, and the data must be delivered to the reader before the read channel is closed. So the receive path enqueues payload first, then processes the flag. Getting that order wrong gives you a data loss bug that only shows up on the last chunk of a stream close.

From stream.py:352-380:

# libp2p/transport/webrtc/stream.py
def _apply_on_trio_thread() -> None:
    # Enqueue payload BEFORE processing flags - the spec allows a
    # message to carry both data and FIN, and the data must be
    # delivered to the reader before the read channel is closed.
    if has_payload:
        try:
            self._read_send.send_nowait(payload)
        except trio.WouldBlock:
            logger.warning(...)
    if has_flag:
        if flag == Message.FIN:
            self._read_closed = True
            self._enqueue_eof_sentinel_locked()
            self._schedule_send(Message(flag=Message.FIN_ACK))
        elif flag == Message.FIN_ACK:
            self._fin_ack_received.set()
    ...

The comment isn't decoration; it's a spec citation for a future reader who wonders why the ordering matters.

Deep dive #2 - the certificate pin that actually pins

This is my favourite bug in the PR. It's the kind of bug that survives green tests because the failure is silent, the API looks correct, and the wrong value is only observable if you cross-check two places that most people cross-check exactly once - at spec-reading time, then never again.

Setting the scene

/webrtc-direct works by embedding the server's DTLS certificate hash in the multiaddr:

/ip4/127.0.0.1/udp/9090/webrtc-direct/certhash/uEiA…/p2p/12D3K…

The client dials that address, constructs an SDP offer locally (there's no signaling server), and expects the server's DTLS handshake to present a cert whose SHA-256 matches the certhash in the multiaddr. If they don't match, the client bails - that's the whole security story of /webrtc-direct.

Which means: whatever cert the server advertises, it must actually use in the DTLS handshake. If those two paths diverge, you either get a broken connection or, worse, a working connection that isn't actually pinned to anything.

The naïve pin

In aiortc, you'd expect to set the cert on the RTCPeerConnection like this:

pc = RTCPeerConnection()
pc._certificates = [my_cert]  # ← looks fine, right?

That's a silent no-op. RTCPeerConnection reads only its own name-mangled private attribute - self.__certificates inside the class, which mangles to self._RTCPeerConnection__certificates from the outside. The pc._certificates attribute lives on the object, sure, but aiortc never looks at it. It happily generates its own certificate at construction time and uses that in DTLS. Your advertised /certhash/ is a completely different digest.

The maintainer review flagged this indirectly - the loopback test caught it directly. Every test that ran without an actual peer connection passed. The moment we ran an SDP round-trip and grepped the a=fingerprint line, the pinned cert was nowhere.

The fix

# libp2p/transport/webrtc/_aiortc_helpers.py
async def create_peer_connection(
    rtc_cert: RTCCertificate,
    ice_servers: list[str] | None = None,
) -> RTCPeerConnection:
    """
    Create an RTCPeerConnection pinned to the given certificate.

    aiortc >= 1.5 no longer accepts certificates= in RTCConfiguration -
    passing it raises TypeError. We build the peer connection with ICE
    servers only, then replace the auto-generated DTLS certificate before
    any SDP operation triggers the DTLS handshake.

    The fingerprint and the actual handshake cert are read from a
    name-mangled private attribute (self.__certificates inside
    RTCPeerConnection → self._RTCPeerConnection__certificates from the
    outside). Setting the un-mangled pc._certificates is a no-op that
    aiortc never reads.
    """
    config = RTCConfiguration(
        iceServers=list(ice_servers) if ice_servers else []
    )
    pc = RTCPeerConnection(configuration=config)
    # Must use the mangled name - aiortc reads only self.__certificates.
    pc._RTCPeerConnection__certificates = [rtc_cert]  # type: ignore[attr-defined]
    return pc

Two characters (__) that would look like a typo in a code review, guarding a subtle security property.

The canary

Fixing the bug is one thing; making sure it stays fixed is another. The loopback test now includes a canary - after the SDP exchange, it parses the a=fingerprint:sha-256 line out of the offer and asserts it equals the SHA-256 of the pinned cert:

# tests/core/transport/webrtc/test_multiplexing_loopback.py
def _sdp_fingerprint_string(cert: WebRTCCertificate) -> str:
    """
    Render *cert*'s SHA-256 fingerprint in the colon-separated upper-hex
    form aiortc writes into SDP a=fingerprint:sha-256 ... lines.
    """
    return ":".join(f"{b:02X}" for b in cert.fingerprint)

If somebody in the future refactors the pin - or aiortc renames __certificates - this test fails immediately, with a message that points straight at the divergence. That's the whole point of a canary: it wouldn't have been added if the bug hadn't happened, and it stays there so the bug can't come back.

The subplot here isn't "aiortc is buggy" - it's that libraries with public APIs sometimes leak private-attribute contracts into their integrators, and when they do, the only defense is a test that observes the on-wire outcome, not the API you think you called.

Deep dive #3 - bridging two runtimes without blowing up

py-libp2p is trio. aiortc is asyncio. Both are event loops. Neither will await the other's coroutines. This isn't a limitation of Python; it's a design decision by trio (structured concurrency, no thread-safety-by-default) and a reality of asyncio's ecosystem.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.