Webhooks don't wait for cold starts

GitHub delivers a webhook once. If your endpoint doesn’t answer within its timeout window, the delivery is marked failed and GitHub does not automatically retry — redelivery is a manual button in the UI or an API call you’d have to script. That policy is fine for backends that are always up. It is silently fatal for backends on free tiers that spin down when idle: the webhook fires, your Render/Railway/Fly instance is cold, the platform takes longer to wake the process than the producer is willing to wait, and the event is simply gone. No error in your logs — your process wasn’t up to log anything. I hit this wiring GitHub push events into a side project (ooh-curses, which auto-checks-in users when they commit): commits happened, check-ins didn’t, and the only evidence was GitHub’s delivery log showing timeouts nobody was watching.

The tempting fixes are both bad. Keeping the backend warm with a ping cron defeats the point of a scale-to-zero free tier. Moving the webhook handler itself to the edge means moving the app’s database access and logic there too. What the problem actually calls for is narrower: something always-on that accepts the delivery instantly and takes over the producer’s retry responsibility.

A relay, not a message bus

The result is do-relay, a Cloudflare Worker + Durable Object that draws a deliberately thin line: it decouples acceptance from delivery, and does nothing else. It answers the producer with 202 Accepted immediately, persists the request — method, headers, exact body bytes — and redelivers to the real backend with retries until it wakes. It is not a message bus: no fan-out, no transforms, no subscriptions, no inspection of payloads. One incoming request becomes one outgoing request, byte-identical, eventually.

Two consequences of that thinness are the design’s best properties. First, the relay never needs the webhook secret: GitHub’s X-Hub-Signature-256 HMAC is computed over the raw body bytes, and since the relay forwards those bytes and that header untouched, the backend verifies the signature end-to-end exactly as if GitHub had called it directly. A relay that parsed and re-serialized JSON would break the signature — which is why the code carries the body as arrayBuffer() → byte array, never as text. Second, the relay itself is allowed to sleep: Durable Objects are evicted when idle and their alarms wake them, so the whole arrangement stays on free-tier economics with nothing kept warm anywhere.

The obvious alternative was Cloudflare Queues, and the repo still carries an explored Queues-based prototype next to the shipped design. Queues hands you retry machinery, exponential backoff, and dead-letter queues out of the box — genuinely attractive — but three things decided against it. Queues requires the paid Workers plan, while SQLite-backed Durable Objects run on the free tier, and “free without keeping a server warm” was the point of the exercise. Queues makes no ordering guarantee, while one Durable Object per destination URL gives strict per-backend FIFO by construction — and webhook consumers that replay events into application state care about order. And the DO’s alarm loop keeps the retry policy in code you own, which is what makes the response classification below possible at all. The honest cost of that choice: the retry machinery Queues would have provided now has to be built — and its gaps owned — by hand, which is exactly what the failure-behavior section has to account for.

Two files at the edge

The edge half (src/worker.js) is small enough to quote nearly whole. The destination comes from ?to= or an X-Forward-To header, and — the key routing decision — each destination URL gets its own Durable Object, which gives per-backend FIFO ordering and serialized delivery for free:

export default {
  async fetch(req, env) {
    const to = req.headers.get('x-forward-to') || new URL(req.url).searchParams.get('to');
    if (!to) return json({ error: 'missing ?to= or X-Forward-To' }, 400);

    const body = await req.arrayBuffer();          // exact bytes, never re-parsed

    // Route = hash of upstream URL (one DO per upstream)
    const id = env.RELAY_DO.idFromName(to);
    const stub = env.RELAY_DO.get(id);

    await stub.fetch('https://do/queue', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ to, m: req.method, h: [...req.headers], b: [...new Uint8Array(body)] }),
    });

    return json({ status: 'accepted', to }, 202);  // producer is done in one round-trip
  },
};

The Durable Object (src/RelayQueue.js) is a persisted queue plus a drain loop. The parts that earn their lines: state is reloaded from storage on wake (blockConcurrencyWhile), a single-flight flag stops concurrent drains, delivery outcomes are classified into three buckets, and a failed attempt schedules an alarm — the mechanism that lets the DO be evicted and still come back to retry:

async deliver() {
  if (this.processing) return;
  this.processing = true;

  while (this.queue.length) {
    const item = this.queue[0];
    const { ok, clientErr } = await this.trySend(item);
    if (ok || clientErr) {
      this.queue.shift();                            // done, or permanently rejected
      await this.state.storage.put('q', this.queue);
    } else {
      await this.state.storage.put('q', this.queue); // transient: persist and back off
      await this.state.storage.setAlarm(Date.now() + 15_000);
      break;
    }
  }
  this.processing = false;
}

async alarm() { await this.deliver(); }

async trySend({ to, m, h, b }) {
  const headers = new Headers(h.filter(([k]) => {
    const kl = k.toLowerCase();
    return kl !== 'connection' && kl !== 'host' && kl !== 'accept-encoding' && kl !== 'cf-ray';
  }));
  const body = b.length ? Uint8Array.from(b).buffer : undefined;
  try {
    const r = await fetch(to, { method: m, headers, body, redirect: 'follow' });
    return { ok: r.ok, clientErr: r.status >= 400 && r.status < 500 };
  } catch {
    return {};                                       // network error → retry bucket
  }
}

Two details are load-bearing. The header filter strips hop-by-hop headers — forwarding the original Host would break routing and TLS at the destination, and cf-ray/accept-encoding describe the first hop, not the second. And the three-bucket classification: 2xx acknowledges, 4xx drops (a client error is the destination saying “this request is wrong” — retrying it forever poisons the queue and blocks everything behind it), while 5xx and network failures retry, because those are exactly what a cold start looks like from outside.

There’s one more question every relay must answer before it ships publicly: abuse. An open relay is a free proxy for making Cloudflare’s IPs hit arbitrary URLs, so the public deployment refuses anything that doesn’t carry GitHub’s delivery fingerprint before it queues a byte:

const ua = req.headers.get('user-agent') || '';
const evt = req.headers.get('x-github-event');
const delId = req.headers.get('x-github-delivery');
if (!evt || !delId || !ua.startsWith('GitHub-Hookshot/')) {
  return jsonError('forbidden', 403);
}

That’s abuse reduction, not authentication — all three headers are spoofable — which is precisely why real authentication stays where it belongs, in the backend’s HMAC verification of the forwarded signature. If you deploy your own, tighten the gate to your own needs (an allowlist of to hosts is the strongest cheap option).

Adapting this: the whole thing is two files plus a wrangler.jsonc binding the DO class; point your webhook at https://<your-worker>/?to=<your-backend-endpoint> and change nothing in the backend except, ideally, one thing — see below.

At-least-once is your problem now

The design converts GitHub’s at-most-once into at-least-once, in order, per destination — and at-least-once means duplicates are now your backend’s problem. If the backend processes a delivery but the response is lost (timeout on the response leg, worker eviction mid-flight), the relay retries something that already happened. The fix costs one line in the consumer: treat x-github-delivery (a GUID per delivery, forwarded untouched) as an idempotency key and skip already-seen IDs.

The other honest edges: a destination that stays dead accumulates queue entries in DO storage and an alarm that fires every 15 seconds indefinitely — bounded in cost by the free tier’s generosity but not by the design; a production-hardened fork should add a max-age after which items are dropped or dead-lettered. A 4xx from a half-awake backend (e.g., a 404 while routes are still mounting) is treated as permanent and dropped — the classification trusts the destination to mean what it says. And the relay is a new single point of failure between producer and consumer; the difference is that Cloudflare’s edge being down is a rarer event than a free-tier dyno being asleep, which is the entire trade.

Honestly serverless

The backend gets to be honestly serverless: it can scale to zero without an event-loss tax, and “was the webhook delivered?” becomes a question with an inspectable answer (the DO’s queue) instead of a race against a producer’s patience. Debugging inverts, too — before, a missed event left no trace on your side; now a stuck delivery is visible state sitting in a queue you own, with the retry loop as your witness. And because the relay is producer-agnostic plumbing (any method, any body, signature preserved), the next webhook integration — Stripe, a CI system, anything that fires-and-forgets — starts from “point it at the relay” rather than from rediscovering, one vanished event at a time, that the producer was never going to wait for you.