GitHub doesn't retry webhooks. So I gave my tunnel a database.

Built solo over a weekend for the WeMakeDevs × Zerops challenge. There's a live gateway you can poke at, and a section near the end about what it can't do. I was testing a GitHub integration with a tunnel open to my laptop. I shut the lid to go get coffee. When I came back, the push event was gone. Not delayed. Not sitting in a queue somewhere. Gone. I figured I'd misconfigured something. I hadn't. From GitHub's own docs: "GitHub does not automatically redeliver failed deliveries." - docs.github.com One failed delivery and it's your problem, unless you open the UI and redeliver by hand. I checked other providers, expecting GitHub to be unusual. It isn't: | Provider | What happens when your tunnel is down | |---|---| | GitHub | No automatic retries at all | | Stripe (test mode) | 3 attempts over a few hours | | Shopify | 8 attempts over 4 hours, then deletes your webhook subscription | Read the last one again. Shopify doesn't just give up on the event. It unsubscribes your app. The part nobody tells you about tunnels Here's what I hadn't thought about properly. A tunnel turns your laptop into production infrastructure for somebody else's system. Senders don't wait for you. A customer checks out at 2am. CI finishes after you've left for lunch. Your wifi drops for thirty seconds on a train. When that happens the sender gets a 502 . It burns one of the three or four attempts it was ever going to make, and then the event is gone. Someone is about to say "ngrok already has replay." It does, and it's good. But ngrok can only replay a request it already saw, which means you were connected when it arrived. The ones that hurt are the ones that arrive when you aren't. That's structural, not a missing feature: A tunnel is a pipe, not a mailbox. If nothing is listening at your end, the request has nowhere to go. So I built the mailbox. It's called Doorbell, and it's a tunnel with a database in the path. You can break it right now You have to take my word for everything else in this post. This part you can check yourself. I'm running a public gateway - the always-on server that holds tunnels open. It has a tunnel named shop with nothing connected to it, which is the same state as your laptop with the lid shut. On a phone? Just open the live page and watch the held right now counter. At a keyboard, send it a webhook: curl -i -X POST https://gw-2ad0-3000.prg1.zerops.app/t/shop/hook -d '{"n":1}' HTTP/2 202 202 Accepted , where you'd expect a 502 . There is no laptop on the other end. Your request is now a row in Postgres, and it gets delivered the moment someone connects. Reload that live page and the counter is one higher. That's yours. Now try a name nobody reserved: curl -i -X POST https://gw-2ad0-3000.prg1.zerops.app/t/zzrandom99/hook -d '{"n":1}' HTTP/2 404 That 404 matters as much as the 202 . Only reserved names get held. If any name worked, anyone could fill my database by inventing URLs, and "we store everything" would just mean "free disk for strangers." And here's what you get back when you reconnect: $ doorbell -name demo 3000 https://gw-2ad0-3000.prg1.zerops.app/t/demo/ → forwarding to 127.0.0.1:3000 ▲ requests held while you were away are arriving now ✓ 11:11:23 POST /hooks/github 200 held 4s ✓ 11:11:23 POST /hooks/github 200 held 2s ✓ 11:11:24 POST /hooks/github 200 held <1s Oldest first, each showing how long it waited. Before you get excited: is this for you? One thing you should know 30 seconds in rather than 10 minutes in. A held webhook will fail a signature check. GitHub and Stripe sign each delivery so your app can prove it really came from them. Doorbell strips anything that looks like a signing header before writing the row, so the signature is gone by the time the request reaches you. That's a deliberate trade. I'd rather hand you a request whose signature no longer verifies than keep someone's live signing secret sitting in my database. Held requests carry an X-Doorbell-Replay header, so you can skip verification when you see it. If your handler verifies signatures and you can't branch on that header, this tool will annoy you. Better you know now. Two more things, and then the good part: - You run this yourself. The gateway above is mine, for trying it. For real use you deploy your own from one YAML file, and nobody else's servers sit in the path of your traffic. - It's a development tunnel. Don't put it in front of production traffic. What 202 actually means This is where a lot of "reliable webhook" tooling quietly lies to you. 202 Accepted does not mean your app handled it. It means: I have taken responsibility for this request. The row is on disk before the sender gets any answer at all. Two things make that promise real. Ordering. Each tunnel's queue drains oldest first, one at a time. That's the only order your handler can make sense of. If the delete arrives before the create, your code does the wrong thing. No duplicates. If you reconnect twice quickly, two drains could race for the same stored request and send the same webhook twice. A duplicate payment notification is somebody's real problem. So the database decides, not the application. Claiming a request and marking it taken happen in a single SQL statement, so only one drain can ever win it. There's a test that runs eight of them at once to prove it: if won != 1 { t.Fatalf("%d of %d claimers won the same row; every extra winner is a webhook delivered twice", won, racers) } The whole thing rests on one function When you run doorbell 3000 , the CLI opens a single outbound connection to the gateway and holds it open. Nothing listens on your machine. Nothing gets opened on your router. It's like calling a support line. They can't call you. But while you're on the line, they can talk to you. Each incoming request becomes its own stream on that one connection, using yamux to run many conversations down a single socket. The proxying itself is just Go's standard library, with one substitution: Transport: &http.Transport{ // The whole trick: instead of dialling a network address, open a // new multiplexed stream on the socket the laptop already holds open. DialContext: func(_ context.Context, _, _ string) (net.Conn, error) { return session.Open() }, }, Swap dial an address for open a stream and you're done. Chunked bodies, keep-alive and WebSocket upgrades all still work, because httputil.ReverseProxy is doing the actual HTTP. The bug I shipped, and how I found it Doorbell has a dashboard. It shows the request and response bodies it captured, and it can re-send any stored request to whoever's laptop is connected. Both were behind a token. Good. The problem: that same token also guarded the port the CLI dials to open a tunnel. One check was answering two completely different questions. May you open a tunnel? and may you read everyone's payloads? ran through the same line of code. So anyone I let tunnel through my gateway could also read every captured body on it, and re-send any of them. The file's own doc comment said the opposite: "Tunnels themselves are not gated by this - only the surfaces that expose captured request and response bodies." The comment described the intent. The code did something else. Nobody catches that in review, because both call sites look correct on their own. The fix was to split them: // tokenOK gates the control port: may this client open a tunnel at all? func (g *gateway) tokenOK(presented string) bool { if g.cfg.clientToken == "" { return true } return subtle.ConstantTimeCompare(...) == 1 } // adminTokenOK gates the dashboard. Kept separate on purpose. func (g *gateway) adminTokenOK(presented string) bool { return subtle.ConstantTimeCompare(...) == 1 } Then I nearly made it worse. The dashboard also accepts ?token= in the URL, and that path called tokenOK too. Look at what tokenOK does when no client token is set: it returns true for anything. My public gateway has no client token, on purpose, so anyone can tunnel. Which means ?token=literally-anything would have opened the dashboard to the first person who tried it. I caught it before shipping. There's now a test whose only job is to fail if it comes back: func TestGuessedQueryTokenCannotUnlockTheDashboard(t *testing.T) { g := gatewayWithTokens("", "operator-secret") if got, _ := reachedOperator(g, req("/dashboard?token=anything")); got { t.Fatal("?token=anything unlocked the dashboard on a public gateway") } } Two privileges, two tokens. Opening a tunnel is a small thing to be allowed to do. Reading what went through it isn't. Why this can't run on most of the internet Holding that connection open gives you two requirements you can't negotiate away: - A raw TCP port on the gateway. Plain TCP, not HTTP, because the client speaks its own protocol on it. - A process that never sleeps. It sits there holding the socket for hours. Those two rule out every serverless platform, by definition. A function stops existing the moment it returns a response, so nobody is left holding the line. And a platform that handles HTTP for you can't give you a port that isn't HTTP. I'll be straight with you, because a Fly.io user is already typing: Fly.io gives you a raw TCP port too. So does Railway. Zerops isn't magic here. But the port was never the hard part. The hard part is the database. It's storing captured webhook bodies, so it must never face the internet, and I didn't want to spend a hackathon weekend wiring up a private network by hand to make that true. Paste one YAML file into Zerops and about ninety seconds later you have a gateway, a Postgres, a Valkey and a private network between them, in your own account. (Valkey is a Redis fork. It tracks which gateway container currently owns which tunnel, so this still works when there's more than one.) One assumption sat underneath all of it: that a raw TCP port on Zerops is genuinely reachable from the public internet. The docs said ports 10-65435 were available. But documented isn't measured, and if that line was wrong the whole pro