BullMQ: Background Jobs That Actually Work

Some things shouldnt happen during a request. Sending emails, processing images, generating reports, syncing data. These need to happen in the background.

BullMQ is my go-to for Node.js job queues. Its fast, reliable, and has features you actually need in production.

Why BullMQ?

BullMQ advantages:

Redis-backed (fast, persistent)
Automatic retries with backoff
Job priorities
Rate limiting
Delayed jobs
Repeatable jobs (cron-like)
Dashboard for monitoring

Basic Setup

import { Queue, Worker } from 'bullmq';

// Create a queue
const emailQueue = new Queue('emails', {
  connection: {
    host: 'localhost',
    port: 6379
  }
});

// Add jobs to the queue
await emailQueue.add('welcome-email', {
  to: 'user@example.com',
  subject: 'Welcome!',
  template: 'welcome'
});

// Create a worker to process jobs
const worker = new Worker('emails', async (job) => {
  console.log(`Processing job ${job.id}: ${job.name}`);

  await sendEmail(job.data);

  return { sent: true };
}, {
  connection: {
    host: 'localhost',
    port: 6379
  }
});

// Handle events
worker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed:`, result);
});

worker.on('failed', (job, error) => {
  console.error(`Job ${job.id} failed:`, error.message);
});

Retry Strategies

Jobs fail. Network issues, API rate limits, temporary outages. Configure retries:

await emailQueue.add('important-email', { to: 'user@example.com' }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000  // 1s, 2s, 4s, 8s, 16s
  }
});

Different backoff strategies:

// Exponential: 1s, 2s, 4s, 8s...
backoff: { type: 'exponential', delay: 1000 }

// Fixed: 5s, 5s, 5s...
backoff: { type: 'fixed', delay: 5000 }

// Custom strategy
backoff: {
  type: 'custom',
  delay: (attemptsMade) => {
    // More aggressive for first retries
    return Math.min(attemptsMade * 1000, 30000);
  }
}

Job Priorities

Some jobs are more important than others:

// Higher priority = processed first (lower number)
await queue.add('urgent', data, { priority: 1 });
await queue.add('normal', data, { priority: 5 });
await queue.add('low', data, { priority: 10 });

Delayed Jobs

Schedule jobs for later:

// Process in 5 minutes
await queue.add('reminder', { userId: '123' }, {
  delay: 5 * 60 * 1000
});

// Process at specific time
const delay = new Date('2025-01-01T00:00:00').getTime() - Date.now();
await queue.add('new-year-email', data, { delay });

Repeatable Jobs (Cron)

For recurring tasks like daily reports:

// Every day at 9am
await queue.add('daily-report', {}, {
  repeat: {
    pattern: '0 9 * * *'  // Cron syntax
  }
});

// Every 5 minutes
await queue.add('health-check', {}, {
  repeat: {
    every: 5 * 60 * 1000
  }
});

// List all repeatable jobs
const repeatableJobs = await queue.getRepeatableJobs();

// Remove a repeatable job
await queue.removeRepeatableByKey(repeatableJobs[0].key);

Rate Limiting

Dont overwhelm external APIs:

const worker = new Worker('api-calls', processJob, {
  connection,
  limiter: {
    max: 10,      // Max 10 jobs
    duration: 1000 // Per second
  }
});

Concurrency

Process multiple jobs simultaneously:

const worker = new Worker('image-processing', processImage, {
  connection,
  concurrency: 5  // Process 5 jobs at once
});

Be careful with concurrency and external resources. Database connections, API rate limits, memory usage.

Job Progress

Report progress for long-running jobs:

const worker = new Worker('video-encoding', async (job) => {
  const frames = 1000;

  for (let i = 0; i < frames; i++) {
    await encodeFrame(i);
    await job.updateProgress(Math.floor((i / frames) * 100));
  }

  return { encoded: true };
});

// Listen for progress
worker.on('progress', (job, progress) => {
  console.log(`Job ${job.id} is ${progress}% complete`);
});

Handling Failures

Sometimes jobs fail permanently. Handle that:

worker.on('failed', async (job, error) => {
  if (job.attemptsMade >= job.opts.attempts) {
    // All retries exhausted
    await notifyAdmin({
      jobId: job.id,
      error: error.message,
      data: job.data
    });

    // Maybe move to dead letter queue
    await deadLetterQueue.add('failed-job', {
      originalJob: job.data,
      error: error.message,
      failedAt: new Date()
    });
  }
});

Dashboard with Bull Board

Visualize your queues:

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(imageQueue),
  ],
  serverAdapter
});

app.use('/admin/queues', serverAdapter.getRouter());

Production Setup

import { Queue, Worker, QueueScheduler } from 'bullmq';

const connection = {
  host: process.env.REDIS_HOST,
  port: parseInt(process.env.REDIS_PORT),
  password: process.env.REDIS_PASSWORD,
};

// Queue for adding jobs
const queue = new Queue('tasks', { connection });

// Scheduler for delayed/repeatable jobs (one per queue)
const scheduler = new QueueScheduler('tasks', { connection });

// Worker(s) for processing
const worker = new Worker('tasks', processJob, {
  connection,
  concurrency: 5,
  limiter: {
    max: 100,
    duration: 1000
  }
});

// Graceful shutdown
process.on('SIGTERM', async () => {
  await worker.close();
  await scheduler.close();
  await queue.close();
  process.exit(0);
});

Quick Checklist

[ ] Connection pooling configured
[ ] Retry strategy with exponential backoff
[ ] Failed job handling (dead letter queue)
[ ] Graceful shutdown
[ ] Monitoring dashboard
[ ] Rate limiting for external APIs
[ ] Appropriate concurrency settings
[ ] Job timeout configured