Queues and async processing

What it is

A queue lets a request hand off work and return immediately. The producer writes a message; a consumer picks it up later. The user gets a fast response, and the slow work happens out of band.

When it is worth it

Good reasons:

• The work is slow and the user does not need the result now — sending email, generating a PDF, transcoding video, syncing to a third party.
• The work can fail and should be retried without failing the user's request.
• You need to absorb bursts. A queue is a shock absorber: 10,000 signups in a minute become a backlog that drains at a rate your database can survive.
• You need to fan out one event to several consumers.

Bad reasons: "it feels more scalable", or the user genuinely needs the result before they can continue. Async work turns one request into a distributed workflow with its own failure modes, monitoring and debugging story. That is a real cost.

At-least-once, and why exactly-once is mostly a myth

Queues offer one of three guarantees:

• At-most-once — deliver and forget. Messages can be lost. Rarely acceptable.
• At-least-once — deliver until acknowledged. Messages can be delivered twice. This is the practical default everywhere.
• Exactly-once — the marketing claim. Genuinely impossible end-to-end in the general case, because acknowledgement itself can fail. Some systems provide it within their own boundary; the moment your consumer writes to an external database or calls a payment API, you are back to at-least-once.

Duplicates are not an edge case. They happen when a consumer processes a message, then crashes before acknowledging; the broker redelivers, correctly.

The full path including the parts people skip: redelivery, retries and the dead-letter queue.

So make consumers idempotent

This is the whole discipline, and it is not optional.

• Give every message a stable id. On receipt, record it in a processed_messages table inside the same transaction as the work. If the insert violates the unique constraint, you have already done this one — acknowledge and move on.
• Prefer naturally idempotent operations. SET status = 'paid' is safe to repeat; balance = balance + 100 is not.
• For external calls, pass an idempotency key so the other system dedupes. See rate limiting and idempotency.

Retries and dead letters

A failed message should retry with exponential backoff, not immediately and not forever. After N attempts it goes to a dead-letter queue — a holding pen you can inspect and replay.

A DLQ with no alert on it is a silent data-loss machine. Alert on depth greater than zero.

Distinguish failure types: a malformed message will never succeed, so retrying it 50 times is waste — fail it straight to the DLQ. A timeout from a downstream service is worth retrying.

Ordering

Most queues do not guarantee global order. If order matters, you need per-key ordering — Kafka partitions by key, SQS FIFO uses message group ids — and that limits parallelism, because one key is processed by one consumer at a time. Often the better answer is to design so order does not matter: include a timestamp or version and ignore stale updates.

Gotchas

• Set visibility timeouts longer than the slowest realistic job, or the broker will redeliver while you are still working.
• Monitor queue depth and consumer lag. Depth rising steadily means consumers cannot keep up, and the backlog is now unbounded.
• Never put large payloads in a message. Store the blob, queue the reference.
• Version your message schemas. Old messages will be in flight during a deploy.
• The user needs feedback: a job id and a status endpoint, or a websocket push. "It's processing" with no way to check is worse than a slow response.

Prove you know it

Take a slow endpoint, move the work to a queue, then deliberately kill the consumer halfway through a job. Confirm the message is redelivered and that running it twice does not produce two of anything. If it does, your consumer is not idempotent yet.