Skip to content

Handling Retries

Narad deliberately does not have a retry engine. It gives you three durable primitives (visibility leases, acks, and topics), and every retry policy you've ever wanted composes out of them in your consumer, where you can see it, log it, and change it without a broker upgrade. This page is the complete menu.

One rule before the menu: your handler must be idempotent. Crashes, timeouts, nacks, and network blips all cause redelivery: that's the at-least-once contract. Every pattern below assumes processing the same message twice is safe. (Deduplicate on a business ID in the payload if it isn't.)

Pattern 0: do nothing (the lease is already a retry)

If your consumer crashes, hangs, or just never acks, the message becomes visible again after the topic's visibility_timeout_ms (default 30s) and someone else gets it. You wrote zero lines of retry code and you already survive process death.

  • Retry spacing: constant, = the visibility timeout.
  • Attempts: unbounded; a permanently failing message loops forever until a human notices. Fine for prototypes; keep reading for production.

Pattern 1: sync retries (for blips)

The dependency timed out once? Just call it again, in-process, while you still hold the lease:

for attempt in 1..3:
    if handle(msg) succeeds: ack; done
ack-with-extend or fall through to a requeue (below)

Cheap, immediate, and invisible to the broker. If your processing time approaches the visibility window, heartbeat while you retry: POST /ack?receipt_handle=$H&extend=true restarts your lease.

Sync retries solve transient failures. They do nothing for "this message will fail for the next ten minutes"; that needs one of the patterns below.

Pattern 2: nack (retry soon, on someone else)

POST /ack?receipt_handle=$H&extend=0 gives the message back immediately: the lease drops, the message is available again on the next consume. Use it when this worker is the problem (shutting down, missing a local dependency) rather than the message.

Same caveat as pattern 0: the redelivered message is byte-identical, so nothing counts attempts for you. Which brings us to the load-bearing pattern:

Pattern 3: the envelope (bounded retries, your way)

Wrap your payload at produce time:

{"payload": {"order_id": "ord_123"}, "delivery_count": 0}

On failure, produce a new copy with delivery_count + 1, then ack the original, in that order. A crash between the two steps yields a duplicate (idempotency absorbs it), never a loss. The reversed order can lose the message; don't reverse it.

on failure:
    if msg.delivery_count >= MAX:  produce to jobs-dlq;  ack;  done
    produce {payload, delivery_count+1} to the same topic;  ack

What this buys you:

  • Bounded attempts: the counter travels with the message, no broker state needed.
  • A dead-letter queue: jobs-dlq is just a topic you create. Alert on its depth, inspect it with replay, re-produce from it to the main topic when the bug is fixed. No special broker feature, full API available on it.
  • No head-of-line blocking: the failed message re-enters at the tail; the partition's frontier moves on immediately.
  • Evolvable policy: want per-error-type max attempts? Different DLQs per failure class? It's your JSON and your code.

This is the pattern the classic worker frameworks (Sidekiq, and the SQS redrive policy itself) implement: the counter just lives in the message instead of the broker.

Pattern 4: spaced backoff (the retry-topic pair)

Requeue-at-tail retries again immediately (an empty queue redelivers in milliseconds). When failures need breathing room (a rate-limited API, a database failing over), use Narad's native delay machinery as your backoff.

First, the rule that makes the shape below make sense: you cannot produce into a delayed child. Narad rejects it (409) because a direct write would bypass the delay the topic promises. A delayed child has exactly one source (its parent) and it receives everything the parent receives. So the parent is your write side, the child is your read side, and the parent must be a dedicated retry topic that carries nothing but retries:

jobs                       ← main topic: produce work here, consume from here

jobs-retry-30s             ← tier PARENT: produce failed messages here…
  └── jobs-retry-30s-run   ← …its delayed child (fanout_delay_ms: 30000):
                              consume retries from here, 30s later

Your consumers consume jobs and jobs-retry-30s-run. On failure, produce the message (envelope counter incremented) to jobs-retry-30s; its copy becomes consumable exactly 30 seconds later: durably, surviving restarts of everything, no in-process timers holding your fate.

Want tiers? One parent + one child per tier, picked by delivery_count:

jobs-retry-30s   └── jobs-retry-30s-run    (delay 30s)   ← delivery_count 1–2
jobs-retry-5m    └── jobs-retry-5m-run     (delay 5m)    ← delivery_count 3–4
jobs-retry-30m   └── jobs-retry-30m-run    (delay 30m)   ← delivery_count 5+

Do not attach several delay children to one retry parent to "get tiers": every child copies every parent message, so each failed message would come back once per tier. One tier, one pair. Fixed tiers, not arbitrary per-message delays; in practice two or three cover every real policy.

The one discipline that is not optional: retry your acks

An ack can fail: a node restarting, or a 503 because the partition's out-of-order ack window (max_acked_ahead_per_partition) is momentarily full. A failed ack is not a failed job. Retry the ack itself with a short backoff until it returns 204 or 410.

We say this from experience: our own long-running soak once ignored failed acks, and the visibility timeout dutifully redelivered 623,000 duplicates over seven hours. The broker was fine. The messages were fine. The harness just didn't retry a 503. Retry your acks.

(410 Gone on ack means the lease already lapsed and the message was handed to someone else; that's not an error to retry, it's Narad telling you the work may run twice. Idempotency, again.)

Choosing, quickly

You want Use
Survive crashes Nothing (leases do it)
Ride out a blip Sync retries (+ extend=true heartbeat)
"Not me, try another worker" Nack (extend=0)
Bounded attempts + DLQ Envelope counter + requeue-and-ack
Backoff between attempts Retry-topic pair with delayed children
Recover after fixing a bug Replay the DLQ, re-produce

Compose freely: production setups typically run sync retries → envelope requeue with a 30s retry tier → DLQ after 5, and alert on DLQ depth. Total broker features required: the ones on this site already.