Phase 4 · Chapter 4

Queues

Why they exist, how BullMQ works inside Redis, and what makes a job bulletproof

What this chapter does to you: You already use BullMQ on AttemptIQ. You know it works. After this chapter, you will know why it works — the exact Redis data structures underneath, how a job survives a server crash, why BRPOP is the right primitive for workers, and what makes a job safe to retry versus one that causes double-billing. This is the chapter where your existing intuition gets a foundation.

Part 1: The Problem Queues Solve

In Chapter 1, you learned that Node.js cannot do CPU-heavy work without blocking the event loop. But the problem queues solve is broader than that.

Three distinct problems — all solved by the same mechanism:

Problem 1: Slow work that shouldn't block the user A student submits a test. Grading takes 3 seconds. Do you make them wait 3 seconds for an HTTP response? No. You respond instantly — "submitted" — and grade asynchronously.

Problem 2: Work that must survive crashes You're sending 10,000 welcome emails after a marketing campaign. Your server crashes halfway through. Emails 1-5,000 sent. 5,001-10,000 lost. A queue persists the work in Redis — surviving the crash, resumable on restart.

Problem 3: Rate-controlling work against external systems Your SMS provider allows 10 messages/second. You have 100,000 users. A queue acts as a buffer — jobs pile in at any rate, workers drain at 10/second. No queue: 100,000 simultaneous HTTP calls to SMS provider = provider bans you. With queue: smooth, controlled output.

All three problems have the same solution: decouple the act of requesting work from the act of doing work. The queue sits between them.

Part 2: The Naive Approach and Why It Fails

Before queues, engineers tried simpler solutions. Understanding why they fail makes the queue design obvious.

Naive approach 1: Do it synchronously

app.post('/submit-test', async (req, res) => {
  await gradeTest(req.body)           // 3 seconds
  await updateLeaderboard(req.body)   // 1 second
  await sendResultEmail(req.body)     // 2 seconds
  res.json({ status: 'done' })        // user waited 6 seconds
})

User waits 6 seconds. Unacceptable UX. Blocks the event loop during CPU work. Server crash mid-process = partial state.

Naive approach 2: Fire and forget with setTimeout

app.post('/submit-test', async (req, res) => {
  res.json({ status: 'submitted' })   // instant response
  setTimeout(async () => {
    await gradeTest(req.body)         // runs "later"
  }, 0)
})

Better UX. But: server crash = job vanishes. No retry on failure. No visibility into what's running. No rate control. RAM-only — restarts lose everything.

Naive approach 3: Database as queue

// On submit:
await db.query(`INSERT INTO jobs (type, payload, status) VALUES ('grade', $1, 'pending')`, [data])

// Worker polls:
setInterval(async () => {
  const job = await db.query(`SELECT * FROM jobs WHERE status='pending' LIMIT 1 FOR UPDATE SKIP LOCKED`)
  if (job) {
    await process(job)
    await db.query(`UPDATE jobs SET status='done' WHERE id=$1`, [job.id])
  }
}, 1000)

Works. Production systems have used this. But:

• SELECT FOR UPDATE creates row locks — DB contention at scale
• Polling every second = constant DB load even when no jobs
• SKIP LOCKED helps but is complex to reason about
• Postgres is optimized for queries, not queue operations

Redis is optimized for exactly these access patterns. BullMQ exists because Redis primitives map perfectly to queue operations.

Part 3: Redis Primitives That Make Queues Work

Before understanding BullMQ, you need to understand two Redis primitives:

LPUSH / BRPOP — The Foundation of a Simple Queue

LPUSH myqueue "job1"    → pushes to the LEFT of a list
LPUSH myqueue "job2"    → list is now: ["job2", "job1"]
BRPOP myqueue 30        → blocking pop from RIGHT, waits up to 30 seconds
                        → returns "job1" (FIFO — first in, first out)

BRPOP is the key primitive. "B" = blocking.

If the list is empty, BRPOP blocks — the Redis connection sits there waiting, consuming zero CPU. The moment any other client pushes a job with LPUSH, Redis wakes up the blocked BRPOP caller and hands them the job.

This is how BullMQ workers wait for jobs without polling. No setInterval. No wasted CPU. Pure event-driven — exactly like Node.js's event loop.

ZADD — The Foundation of Delayed and Priority Jobs

Sorted sets (from the Caching chapter) store members with a float score, always sorted by score.

ZADD delayed_jobs 1705312800 "job:abc"   // score = Unix timestamp = "run at this time"
ZADD delayed_jobs 1705316400 "job:def"
ZADD delayed_jobs 1705309200 "job:ghi"

ZRANGEBYSCORE delayed_jobs 0 1705310000  // all jobs ready to run as of timestamp 1705310000
→ ["job:ghi"]                            // only job:ghi is ready

BullMQ runs a periodic check (every 5 seconds by default): "Which delayed jobs have score <= NOW? Move them to the wait list."

This is how delay: 5000 in BullMQ works. The job is ZADDed with score = Date.now() + 5000. When that timestamp passes, the scheduler moves it to the wait list where workers can pick it up.

Part 4: BullMQ Internals — The Full Picture

BullMQ maintains several Redis keys per queue. Let's say your queue is named grading.

The Data Structures

bull:grading:wait       → List    (jobs waiting to be processed, FIFO)
bull:grading:active     → List    (jobs currently being processed by a worker)
bull:grading:completed  → ZSet    (completed jobs, score = completion timestamp)
bull:grading:failed     → ZSet    (failed jobs, score = failure timestamp)
bull:grading:delayed    → ZSet    (delayed jobs, score = scheduled run timestamp)
bull:grading:priority   → ZSet    (priority jobs, score = inverse priority)
bull:grading:events     → Stream  (event log for monitoring)
bull:grading:{jobId}    → Hash    (job data, attempts, options, timestamps)

Each job is stored as a Redis Hash:

HSET bull:grading:abc123
  name        "gradeSubmission"
  data        '{"submissionId":"xyz","userId":"user:123"}'
  opts        '{"attempts":3,"backoff":{"type":"exponential","delay":1000}}'
  timestamp   1705226400000
  attemptsMade 0
  processedOn  0
  finishedOn   0

The Job Lifecycle — Exact State Transitions

Step 1: queue.add('gradeSubmission', data)

-- BullMQ runs this Lua script atomically:
LPUSH bull:grading:wait "abc123"           -- add job ID to wait list
HSET bull:grading:abc123 ...data...        -- store job data

One atomic operation: job is either in wait with its data, or not at all. No partial state possible.

Step 2: Worker picks up job

-- Worker calls BRPOP, gets "abc123"
-- BullMQ Lua script atomically:
BRPOP bull:grading:wait                    -- take from wait list
LPUSH bull:grading:active "abc123"         -- move to active list
HSET bull:grading:abc123 processedOn NOW   -- record start time

The job moves from wait → active atomically via Lua. At no point is the job in neither list (lost) or both lists (double-processed).

Step 3a: Job succeeds

LREM bull:grading:active 1 "abc123"        -- remove from active
ZADD bull:grading:completed NOW "abc123"   -- add to completed
HSET bull:grading:abc123 finishedOn NOW    -- record finish time

Step 3b: Job fails (exception thrown in worker)

-- Increment attempts
HINCRBY bull:grading:abc123 attemptsMade 1

-- If attempts < maxAttempts: reschedule with backoff
ZADD bull:grading:delayed (NOW + backoffDelay) "abc123"
LREM bull:grading:active 1 "abc123"

-- If attempts >= maxAttempts: move to failed
LREM bull:grading:active 1 "abc123"
ZADD bull:grading:failed NOW "abc123"

Why Lua scripts for every transition?

These operations are multi-step. Without Lua:

• Move wait → active: BRPOP then LPUSH. Crash between them = job lost.
• Move active → completed: LREM then ZADD. Crash = job stays in active forever.

Lua scripts run atomically on Redis. The entire state transition is one indivisible operation. Either it fully completes or fully doesn't.

This is the exact same lesson from Chapter 2 (Rate Limiting) — multi-step Redis operations require Lua for correctness.

Part 5: Worker Concurrency — How Node.js Handles Multiple Jobs

const worker = new Worker('grading', async (job) => {
  await gradeSubmission(job.data)
}, {
  concurrency: 5   // process up to 5 jobs simultaneously
})

concurrency: 5 means the worker maintains up to 5 in-flight jobs at once.

But Node.js is single-threaded. How can 5 jobs run simultaneously?

They don't run simultaneously — they run concurrently.

Remember Chapter 1: Node.js can have many I/O operations in flight. If gradeSubmission is mostly I/O (DB queries, Redis reads), then while job 1 is waiting for Postgres to respond, the event loop picks up job 2, starts it, waits for its I/O, picks up job 3, and so on.

5 jobs in-flight = 5 Postgres queries happening in parallel at the OS level, managed by libuv, while Node.js's single JS thread coordinates between them.

When concurrency helps and when it doesn't:

Job involves: DB query + Redis read + HTTP call
→ Mostly I/O waiting
→ High concurrency is good (10, 20, 50)
→ Event loop handles coordination, libuv handles parallel I/O

Job involves: Parsing a 50MB JSON, generating a PDF, running ML inference
→ Mostly CPU computation
→ High concurrency HURTS — blocks the event loop
→ Use concurrency: 1 with multiple separate Worker processes
→ Or use Worker Threads for CPU isolation

Multiple worker processes:

For CPU-heavy jobs, run multiple worker processes:

# Run 4 separate worker processes (PM2 cluster mode)
pm2 start worker.js -i 4

# Or manually:
# worker.js runs: new Worker('grading', handler, { concurrency: 1 })
# 4 instances = 4 parallel workers, each with its own Node.js process
# Redis BRPOP is atomic — only one worker gets each job

4 separate processes = 4 CPU cores utilized = true parallelism. Each process competes for jobs via BRPOP — Redis ensures only one wins each job.

Part 6: Retries and Backoff — Making Jobs Bulletproof

The Retry Configuration

queue.add('sendEmail', data, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 2000    // base delay in ms
  }
})

If the job fails, BullMQ reschedules it:

• Attempt 1 fails → wait 2s → retry
• Attempt 2 fails → wait 4s → retry
• Attempt 3 fails → wait 8s → retry
• Attempt 4 fails → wait 16s → retry
• Attempt 5 fails → job moves to failed queue

Why exponential backoff?

If your email provider is down, hammering it every second makes things worse. Exponential backoff gives the service time to recover. It also prevents thundering herd — if 1,000 jobs all fail at the same moment, they're retried at staggered intervals, not all simultaneously.

The Dead Letter Queue

After maxAttempts, BullMQ moves jobs to the failed sorted set. This is your dead letter queue — a holding area for jobs that couldn't be processed.

You should always handle failed jobs:

worker.on('failed', (job, error) => {
  // job.attemptsMade === job.opts.attempts — this is the final failure
  logger.error({ jobId: job.id, data: job.data, error: error.message })
  // alert via Sentry, PagerDuty, or whatever monitoring you use
})

// Or manually inspect and retry from your admin panel:
const failedJobs = await queue.getFailed(0, 100)
await failedJobs[0].retry()   // move back to wait list for one more attempt

Failed jobs with their full data and error history are stored in Redis. This is your audit trail — you can see exactly what failed, when, and why.

Here's why attempts and a failed set matter. Watch one malformed job take out a worker, then switch on retries with backoff and a DLQ.

Part 7: Idempotency — The Most Important Job Design Principle

If a job fails mid-execution and retries, it runs twice. Partially or fully.

Idempotent: Running the job multiple times has the same effect as running it once. Non-idempotent: Running twice causes double effects — double charges, duplicate emails, etc.

The Dangerous Pattern

// NON-IDEMPOTENT — do not do this
async function sendWelcomeEmail(job) {
  await emailProvider.send({
    to: job.data.email,
    subject: 'Welcome to AttemptIQ!'
  })
  await db.query('UPDATE users SET welcome_sent = true WHERE id = $1', [job.data.userId])
}

If the email sends successfully but the server crashes before the DB update, the job retries, sends the email again. User gets two welcome emails.

The Correct Pattern

// IDEMPOTENT — check before acting
async function sendWelcomeEmail(job) {
  // Check if already done
  const user = await db.query('SELECT welcome_sent FROM users WHERE id = $1', [job.data.userId])

  if (user.welcome_sent) {
    return   // already sent, skip silently — this is correct behavior on retry
  }

  await emailProvider.send({
    to: job.data.email,
    subject: 'Welcome to AttemptIQ!'
  })

  await db.query('UPDATE users SET welcome_sent = true WHERE id = $1', [job.data.userId])
}

Now retrying is safe. The check-then-act pattern makes the operation idempotent.

For financial operations — use unique transaction IDs:

async function chargeUser(job) {
  // Include job.id as idempotency key to payment provider
  await stripe.charges.create({
    amount: job.data.amount,
    idempotencyKey: `charge-${job.id}`   // Stripe deduplicates on this key
  })
}

Even if your job retries, Stripe sees the same idempotencyKey and returns the original charge instead of creating a new one. No double-billing.

The rule: Every job that produces side effects (emails, charges, DB writes) must be designed for idempotency before it's designed for anything else.

Part 8: Job Deduplication — Preventing Duplicate Work

Sometimes the same job gets enqueued multiple times before the first one runs. User clicks "Submit" twice. Two queue.add() calls. Two grading jobs for the same submission.

BullMQ solves this with job IDs:

await queue.add('gradeSubmission', data, {
  jobId: `grade-${submissionId}`   // deterministic ID based on content
})

If a job with jobId = 'grade-xyz' already exists in wait, active, or delayed, BullMQ silently skips the duplicate. One job, regardless of how many times you call add().

This is the queue equivalent of a database INSERT ... ON CONFLICT DO NOTHING.

Part 9: Queue Patterns — Fan-out, Pipeline, Competing Consumers

Pattern 1: Competing Consumers

Multiple workers on the same queue. BRPOP is atomic — each job goes to exactly one worker.

Scale workers horizontally to match throughput. Queue depth is your backpressure indicator — if it keeps growing, add workers.

Pattern 2: Fan-out

One event produces multiple independent jobs:

// After test submission:
await Promise.all([
  gradingQueue.add('grade', { submissionId }),
  analyticsQueue.add('recordAttempt', { submissionId }),
  notificationQueue.add('notifyUser', { userId, testId })
])

Three separate queues, three separate workers. They process independently. A failure in analytics doesn't affect grading. Each concern is isolated.

Why separate queues instead of one queue? Different jobs have different priorities, retry policies, and worker requirements. Your grading jobs are CPU-light and need low latency. Your analytics jobs are I/O-heavy and can tolerate delay. Mixing them in one queue means a pile of analytics jobs can delay grading jobs. Separate queues = separate throughput control.

Pattern 3: Pipeline (Job Chaining)

Each job's completion triggers the next job:

// In the grading worker:
worker = new Worker('grading', async (job) => {
  const result = await gradeSubmission(job.data)

  // Trigger the next step
  await leaderboardQueue.add('updateLeaderboard', {
    userId: job.data.userId,
    score: result.score,
    testId: job.data.testId
  })

  return result
})

BullMQ also has a built-in Flow API for this:

const flow = new FlowProducer()
await flow.add({
  name: 'sendResult',
  queueName: 'notifications',
  data: { userId },
  children: [{
    name: 'updateLeaderboard',
    queueName: 'leaderboard',
    data: { userId, score },
    children: [{
      name: 'gradeSubmission',
      queueName: 'grading',
      data: { submissionId }
    }]
  }]
})

The tree executes leaves first. gradeSubmission runs → on success, updateLeaderboard runs → on success, sendResult runs. If any step fails, downstream steps don't execute.

Pattern 4: Priority Queue

Some jobs are more urgent than others:

// High priority — student waiting for result
await queue.add('grade', data, { priority: 1 })

// Low priority — batch analytics, can wait
await queue.add('grade', data, { priority: 10 })

BullMQ uses a sorted set for priority jobs with priority as score. Workers always drain higher-priority jobs first.

Use priority carefully. If high-priority jobs keep arriving, low-priority jobs can starve — never processed. Design your priority levels to guarantee that lower-priority work eventually runs.

Part 10: What Happens When Things Go Wrong

Scenario 1: Worker Crashes Mid-Job

Worker picks up job → starts processing → server crashes. Job is in the active list. Nobody is processing it. It's stuck.

BullMQ uses a stall detection mechanism.

Every active job has a "lock" — a Redis key with a TTL:

bull:grading:abc123:lock   (TTL: 30 seconds)

The worker renews this lock every few seconds while processing. A separate BullMQ process checks for locks that have expired (not renewed). If a lock expires, the job is assumed stalled — worker died. BullMQ automatically moves it back to the wait list for re-processing.

Your job will be retried. Design for idempotency.

Scenario 2: Redis Goes Down

Jobs cannot be added. Workers cannot pick up jobs. Everything stops.

Options:

• Redis Sentinel: automatic failover to replica — minimal downtime
• Redis Cluster: sharded Redis — higher throughput, more complex
• Graceful degradation: if queue.add() throws, handle it — don't crash the API

For AttemptIQ: a brief Redis outage means grading pauses, not that submissions are lost. The submission is in Postgres. When Redis recovers, you can re-enqueue unprocessed submissions. Always design your system so the authoritative record lives in Postgres, not in the queue.

Scenario 3: Queue Depth Explodes

More jobs arriving than workers can process. Queue grows unboundedly. Eventually Redis RAM fills up.

Monitor: await queue.getJobCounts() — shows waiting, active, completed, failed counts. Alert when waiting > threshold. Scale workers horizontally.

Redis maxmemory + eviction policy interacts badly with queues. If Redis starts evicting your queue keys, jobs silently vanish. Use a dedicated Redis instance for queues. Never share the queue Redis with your cache Redis if memory pressure is a concern.

Part 11: The Vocabulary Map

Part 12: Practice Problems

Problem 1 (Design — AttemptIQ): A student submits a test on AttemptIQ. Design the complete queue architecture:

• Which jobs get created?
• What are their queue names?
• What are their priorities?
• What are their retry policies?
• Which ones must be idempotent and how do you make them so?
• What is the correct order of execution?

Draw the fan-out and pipeline relationships.

Problem 2 (Idempotency): Your grading worker does this:

async function gradeSubmission(job) {
  const answers = await db.query(`SELECT * FROM answers WHERE submission_id=$1`, [job.data.id])
  const score = calculateScore(answers)
  await db.query(`INSERT INTO scores (submission_id, score) VALUES ($1, $2)`, [job.data.id, score])
  await db.query(`UPDATE submissions SET status='graded', score=$1 WHERE id=$2`, [score, job.data.id])
  await notificationQueue.add('notify', { userId: job.data.userId, score })
}

The worker crashes after the INSERT INTO scores but before the UPDATE submissions.

What state is the DB in? What happens when the job retries? How many notification jobs get created if this job retries 3 times? Fix the code to be idempotent.

Problem 3 (Internals): A job is in the active list. The worker processing it crashes. The job's Redis lock TTL is 30 seconds.

Walk through exactly what happens:

• What does BullMQ check to detect the stall?
• Who runs the stall detection — the crashed worker, a surviving worker, or BullMQ itself?
• What Redis commands are executed to recover the job?
• How does BullMQ ensure two workers don't both pick up the recovered job?

Problem 4 (Architecture): AttemptIQ sends email notifications for test results. Your email provider (Brevo) allows 10 emails/second on your plan.

Design the rate-controlled queue system:

• Which BullMQ option controls the drain rate?
• If 1,000 test results come in at once, how does the queue behave?
• What happens to queue depth over time?
• How do you monitor whether you're falling behind?
• What is the maximum acceptable queue depth before you alert?

Problem 5 (Connect Everything): Trace the complete journey of a test submission on AttemptIQ:

User submits → your API receives it → grading job enqueued → worker processes → result stored

For each step, identify:

1. Which layer from our curriculum is responsible (HTTP, Node.js event loop, Redis, Postgres, BullMQ)?
2. What would happen if that layer fails at exactly that moment?
3. What mechanism ensures the system recovers?

This is a complete resilience analysis — the kind asked in senior engineering interviews.

Where We Are in Phase 4

Phase 4 — Backend Patterns & Production Infrastructure
  ✅ Chapter 1: Caching
  ✅ Chapter 2: Rate Limiting
  ✅ Chapter 3: Authentication
  ✅ Chapter 4: Queues
  ⬜ Chapter 5: Observability
  ⬜ Chapter 6: Load Balancing

You now have the core backend patterns. Every standard web application you will build leans on these four chapters.

Next: Chapter 5 — Observability. Once work is spread across API servers, queues, and workers, console.log stops telling you what happened. Chapter 5 is how you see inside a running distributed system. Then Chapter 6 opens the load balancer, and Phase 5 applies everything to full system designs.

If you jumped here from the Kafka chapter (Phase 3, Chapter 4), go back to it now — it builds directly on the BullMQ model you just learned.

End of Phase 4, Chapter 4. Estimated reading time: 70-85 minutes. Practice problems: 60-90 minutes. Do not proceed to Chapter 5 until you can answer Problem 5 completely — the full resilience trace. That analysis is the exact format of senior system design interviews. If you can do it for AttemptIQ's test submission, you can do it for any system.