Your Webhook Receiver Is Offline. Now What?
DEV Community

Your Webhook Receiver Is Offline. Now What?

The Hidden Coupling in a Typical Webhook Integration

A direct webhook integration usually looks like this:

Webhook provider ──────HTTP request──────> Your application

This design couples two separate events: the provider sending the webhook, and your application being ready to process it. For the delivery to succeed, both systems-and the network between them-must be available at the same moment. That creates several possible failure points:

  • The receiving application is deploying or restarting
  • DNS resolution fails temporarily
  • The route between networks is unstable
  • The request times out
  • A firewall rejects the connection
  • The receiver is available only from a private network
  • The sender stops retrying before the receiver recovers

The webhook provider may have its own retry policy, but those policies vary. Some services retry for hours, others make only a few attempts, and some treat certain HTTP responses as permanent failures. Your application has no control over that behavior.

Separate Receiving from Processing

A more resilient design introduces a persistent delivery layer between the webhook provider and the final destination:

Webhook provider
       β”‚
       β”‚ HTTPS
       β–Ό
   Adal Server
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Receive       β”‚
β”‚ Store         β”‚
β”‚ Inspect       β”‚
β”‚ Retry         β”‚
β”‚ Redeliver     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
       β”‚
       β”œβ”€β”€ Direct HTTP ──> Public service
       └── Adal CLI ─────> Private or local service

Instead of sending the webhook directly to your application, the provider sends it to a permanent HTTPS endpoint created in Adal. Adal receives and stores the request first. Delivery to your application becomes a separate operation that can be retried if the destination is unavailable.

This changes the failure model. Your application no longer needs to be online at the exact moment the webhook is created. It only needs to become available before the webhook’s retention period expires.

Setting Up the Receiving Endpoint

After creating an account in Adal, create a Server in the Servers section. The Server receives a permanent HTTPS URL. Use that URL as the webhook endpoint in the sending service.

The flow then becomes:

  1. The provider sends a webhook to the Adal Server
  2. Adal receives and stores the request
  3. The webhook becomes available for inspection and auditing
  4. Adal forwards it to one or more configured destinations
  5. If delivery fails, Adal performs the configured retry attempts

Destinations are configured separately, so the same incoming webhook can be routed according to the needs of each receiving system. There are two delivery options: Direct HTTP and Adal CLI.

Option 1: Direct HTTP

Direct HTTP is the simplest option when the receiving service already has a public endpoint.

Provider ──> Adal ──> https://api.example.com/webhooks

Adal forwards the webhook directly to the configured HTTPS address. The request is delivered as received, with one intentional change: Adal identifies itself in the User-Agent header. This makes the delivery path transparent and helps investigate reports of unwanted or malicious requests.

Direct HTTP is useful when:

  • The receiver has a public IP address or hostname
  • The service is reachable from the internet
  • The endpoint occasionally goes offline
  • Network routing between the original sender and receiver is unreliable
  • You want stored requests, delivery history, and controlled retries

Each destination has its own retry configuration. If the receiving service does not respond, Adal makes the configured number of automatic delivery attempts.

Example: A Temporarily Unavailable Payment Handler

Imagine that a payment provider sends this event:

{
  "event": "payment.completed",
  "payment_id": "pay_84b7d1",
  "amount": 4900,
  "currency": "USD"
}

At that moment, your payment handler is being deployed and cannot accept requests. With direct delivery, the result depends entirely on the payment provider’s retry policy. With Adal in the middle, the event is accepted and stored first. Adal then attempts to deliver it to your handler. If the handler recovers during the automatic retry window, delivery continues without requiring the original provider to send the event again.

Option 2: Adal CLI

Not every webhook receiver should be publicly accessible. You may need to deliver events to:

  • An application running on a developer’s computer
  • A test environment inside a private network
  • An internal business service
  • A machine behind NAT
  • A service protected by a corporate router
  • A system without a public IP address

Opening inbound ports for these systems may be inconvenient, unsafe, or prohibited by network policy. Adal CLI handles this by establishing a persistent, encrypted outbound WebSocket connection to Adal:

Provider ──> Adal <════ outbound WebSocket ════ Adal CLI ──> Local app

Because the connection is initiated by Adal CLI, you do not need to:

  • Expose the local application to the internet
  • Configure port forwarding
  • Open an inbound firewall port
  • Assign a public IP address to the machine

When the Adal Server receives a webhook, it sends the request through the established connection. Adal CLI then forwards it to the local endpoint configured by the user. This is especially useful during development, but it can also work for internal services that are intentionally unavailable from the public internet.

What Happens When Automatic Retries Run Out?

Automatic retries solve temporary failures, but no retry schedule can continue forever. A destination might remain offline longer than expected. A hardware failure may take several hours to repair. A deployment may need to be rolled back manually. An office network may stay disconnected until the next morning.

If all automatic delivery attempts are exhausted before the receiver returns, Adal keeps the stored webhook available until the end of its retention period. Once the destination is working again, you can restart the delivery manually.

Automatic retries exhausted
       β”‚
       β–Ό
Webhook remains stored in Adal
       β”‚
Receiver recovers
       β”‚
       β–Ό
User starts redelivery manually

The number of previous automatic attempts does not prevent manual redelivery. The only requirement is that the original webhook must still be stored in Adal. After the retention period expires, the webhook is permanently deleted. At that point, it can no longer be inspected, recovered, or delivered again.

A Practical Recovery Scenario

Suppose an internal order-processing service loses its network connection at 10:00. During the outage:

  • An e-commerce platform sends several order events
  • Adal receives and stores them
  • Automatic delivery attempts fail because the internal service is offline
  • The retry limit is reached at 10:30
  • The network is restored at 12:00
  • An operator verifies that the order service is healthy
  • The failed deliveries are restarted manually
  • The original events can still be delivered because they remain within their retention period

Without an intermediate storage layer, recovery might require asking the original platform to resend the events-assuming that it supports manual redelivery and still retains them.

Design Your Receiver for Duplicate Delivery

Reliable delivery and exactly-once processing are not the same thing. Any system that retries HTTP requests can potentially deliver the same event more than once. A receiver may successfully process a webhook but fail to return a response before the sender times out. From the sender’s perspective, the result is unknown, so it retries. Manual redelivery creates the same possibility.

Webhook handlers should therefore be idempotent whenever possible. A common approach is to store a unique event ID before performing business operations:

Receive webhook
       β”‚
       β–Ό
Extract event ID
       β”‚
       β–Ό
Was this ID already processed?
       β”‚
    β”Œβ”€β”€β”΄β”€β”€β”
   Yes    No
    β”‚      β”‚
Return    Process event
  2xx     Store event ID

A simplified handler might follow this logic:

async function handleWebhook(event) {
  if (await eventStore.wasProcessed(event.id)) {
    return { status: 200, message: "Already processed" };
  }
  await processEvent(event);
  await eventStore.markAsProcessed(event.id);
  return { status: 200, message: "Processed" };
}

The exact implementation depends on your database and consistency requirements, but the principle is important: retries should not create duplicate payments, orders, emails, or other irreversible side effects.

Security Considerations

Adding a delivery layer does not remove the need to secure the receiving endpoint. Consider the following before going to production:

  • Validate webhook signatures when the provider supports them
  • Treat all webhook payloads as untrusted input
  • Use HTTPS for public destinations
  • Restrict what the webhook handler is allowed to do
  • Avoid storing secrets in webhook URLs
  • Monitor repeated failures and unexpected delivery patterns
  • Test how signature validation behaves when a request passes through an intermediary
  • Define an appropriate retention period for the sensitivity of your data

Adal stores webhook contents in encrypted form. Stored webhooks are available only to the user who owns the corresponding Server. Adal does not sell or use webhook contents for unrelated purposes. The data is processed to store requests and deliver them to destinations configured by the user.

Adal Is a Transport Layer, Not a Policy Bypass

There is an important distinction between improving delivery reliability and bypassing network restrictions. Adal is designed to transport webhooks between systems with unstable or limited connectivity. It is not intended to circumvent security controls, regional restrictions, or organizational network policies.

Before connecting an internal system, confirm that the setup is permitted by the network owner or system administrator. This is particularly important in corporate environments where outbound connections, third-party data processing, and webhook payload retention may be subject to security or compliance requirements.

Choosing a Delivery Method

The choice between Direct HTTP and Adal CLI depends primarily on whether the receiving service is publicly reachable.

Requirement Direct HTTP Adal CLI
Public receiving endpoint Required Not required
Public IP address Usually required Not required
Open inbound port Required Not required
Works with local development Possible with extra networking Yes
Works behind NAT Only with forwarding or a proxy Yes
Automatic retry support Yes Yes
Manual redelivery during retention Yes Yes

Use Direct HTTP when your service already exposes a public HTTPS endpoint. Use Adal CLI when the destination is local, private, behind NAT, or unable to accept inbound internet connections.

Production Checklist

Before relying on any webhook delivery pipeline in production, verify the complete failure path:

  1. Send a test webhook successfully
  2. Confirm that the HTTP method, headers, and body arrive as expected
  3. Stop the receiving service
  4. Confirm that failed delivery attempts are recorded
  5. Start the service again during the retry window
  6. Verify that automatic delivery resumes
  7. Exhaust the automatic retries in a test environment
  8. Restart the delivery manually
  9. Test duplicate events and confirm idempotent processing
  10. Verify signature validation
  11. Review the webhook retention period
  12. Confirm the setup with the network or security administrator
  13. Add monitoring for repeated delivery failures

Testing failure behavior is just as important as testing the successful path. The middle of an outage is a bad time to discover that a retry policy, signature check, or deduplication mechanism does not behave as expected.

Final Thoughts

The main reliability problem with direct webhooks is timing: the sender and receiver must both be available at the same moment. A persistent delivery layer breaks that dependency. Adal receives and stores the webhook first, then attempts to deliver it through Direct HTTP or Adal CLI. Temporary failures are handled with automatic retries. Longer outages can be handled with manual redelivery, provided the webhook is still within its retention period.

That does not eliminate every integration failure. Your handlers still need authentication, validation, monitoring, and idempotency. But it gives you something extremely valuable during an outage: time to restore the receiving service without immediately losing the original request.

How do you currently handle webhooks when a destination remains offline longer than the sender’s retry window?

Comments

No comments yet. Start the discussion.