Add a queue to something

The brief

Take something in an existing project that is slow or unreliable and move it behind a queue. Then deliberately break it, repeatedly, until you have seen every failure mode with your own eyes.

Good candidates: sending a welcome email, generating a PDF or report, resizing an uploaded image, calling a flaky third-party API, or a nightly aggregation.

The theory is in queues and async processing. This is the part where it stops being theory.

Pick a broker

Any of these work. Do not spend a day choosing:

• Redis + BullMQ / Sidekiq / RQ — easiest if Redis is already there. Retries, delays, dead-letter handling and a dashboard included.
• Postgres as a queue — SELECT ... FOR UPDATE SKIP LOCKED. No new infrastructure, and genuinely fine to tens of thousands of jobs per hour. Underrated.
• SQS / Azure Service Bus — managed, durable, the closest to what you will meet at work.
• RabbitMQ or Kafka — more machinery than this exercise needs. Kafka in particular is a log, not a task queue.

Build it

• Producer: the endpoint enqueues a message and returns immediately — 202 Accepted with a job id.
• Message contains a reference, not a payload. { user_id, report_id }, never the whole 4 MB document.
• Every message has a stable unique id.
• Consumer runs as a separate process from the web server. This matters: if it runs in the same process, you have not actually decoupled anything.
• Consumer is idempotent — a processed_messages table with a unique constraint on the message id, written in the same transaction as the work.
• Retries with exponential backoff and jitter, capped at 3–5 attempts.
• Dead-letter queue for messages that exhaust their retries.
• The user can check status: a GET /jobs/:id endpoint returning queued / processing / done / failed.
• Graceful shutdown: on SIGTERM, stop taking new messages, finish the current one, then exit.

Now break it on purpose

This is the actual exercise. Each experiment teaches one thing that reading cannot.

1. Kill the consumer mid-job. Start a job, wait until it is halfway, kill -9 the process. Restart it.

Expected: the message is redelivered after the visibility timeout, and reprocessing produces no duplicate side effects. If you get two emails, your consumer is not idempotent — and you have just reproduced the single most common async bug in production.

2. Make the job fail every time. Throw unconditionally. Watch the backoff intervals grow, count the attempts, confirm it lands in the DLQ and stops. Then confirm you can replay it from the DLQ after fixing the cause.

3. Enqueue 10,000 messages at once. Watch queue depth climb and drain. How long? Does the database survive it? Does one slow consumer become the bottleneck? Now add a second consumer and watch throughput.

4. Make the job slower than the visibility timeout. Sleep for longer than the timeout. The broker redelivers while you are still working, and now two consumers are processing the same message simultaneously. This one surprises everyone the first time.

5. Deploy while jobs are in flight. Send SIGTERM during processing. Without graceful shutdown, the job dies mid-work and retries. With it, it finishes cleanly. Compare both.

6. Change the message schema. Deploy a consumer expecting a new field while old messages are still queued. Watch it fail. This is why messages need a version field.

Instrument it

• Queue depth as a metric, with an alert if it grows steadily — that means consumers cannot keep up and the backlog is unbounded.
• Job duration, p50 and p99.
• Failure rate, and an alert on DLQ depth above zero.
• Trace context propagated from the producing request into the job, so one trace spans both. See observability.

Prove you know it

Run experiment 1 and confirm exactly one email was sent. Then run experiment 4 and explain out loud why two consumers ended up on the same message. If you can do both, you understand at-least-once delivery in a way that reading about it does not give you.