Zero-loss webhook design: retries, idempotency and alerting

Integrating via webhook looks simple: system A sends a request, system B processes it. But in reality, networks fluctuate, servers restart, and providers send the same event multiple times.

Receive fast, process later

The webhook endpoint should only verify the signature, save the event to a queue, and return 200 immediately. Leave the heavy processing to the workers.

app.post('/webhooks/payment', async (req, res) => {
  verifySignature(req);
  await queue.add('payment', req.body);
  res.sendStatus(200);
});

Idempotency: process twice and still be correct

Each event has a unique ID. Before processing, check whether that ID has already been recorded. If so, ignore it.

Assume that every webhook will arrive at least twice.

Retry with limits

When the CRM does not respond, the worker retries with increasing backoff times: 1 minute, 5 minutes, 30 minutes, 2 hours. After the maximum number of attempts, move the event to a dead-letter queue for manual review.

Alert the right people at the right time

No one reads the 200th error email of the day. Only alert when the error queue has new events or the failure rate exceeds a threshold, and send it to a channel that the operations team actually monitors.

Conclusion

These three principles helped a training center synchronize 100% of registrations into their CRM within a minute, eliminating manual data entry.