Queue Worker

You are a backend engineer setting up a production-ready job queue system. The user wants to implement Bull/BullMQ to handle asynchronous jobs with retry logic, concurrency control, and error handling.

What to check first

• Run npm list bull bullmq redis to verify Bull or BullMQ is installed (use BullMQ for new projects, Bull v3 is legacy)
• Confirm Redis server is running: redis-cli ping should return PONG
• Check Node.js version is 12+ with node --version

Steps

1. Install BullMQ and Redis client: npm install bullmq redis (or bull for legacy)
2. Create a Queue instance by importing Queue from bullmq and specifying the queue name and Redis connection
3. Define a processor function that handles individual job data and returns success or throws errors
4. Register the processor with queue.process() or use Worker class for better separation (BullMQ pattern)
5. Add jobs to the queue using queue.add(jobName, jobData, { options }) with retry and delay settings
6. Listen to job events (completed, failed, progress) to track execution status
7. Set concurrency limits in processor options to control parallel execution
8. Implement exponential backoff retry strategy in job options for resilience

Code

import { Queue, Worker, QueueEvents } from 'bullmq';
import { createClient } from 'redis';

// Redis connection config
const redisConnection = {
  host: process.env.REDIS_HOST || 'localhost',
  port: process.env.REDIS_PORT || 6379,
};

// Create queue instance
const emailQueue = new Queue('email-jobs', { connection: redisConnection });

// Define processor with concurrency and timeout
const emailWorker = new Worker(
  'email-jobs',
  async (job) => {
    console.log(`Processing job ${job.id}: ${job.name}`);
    
    if (job.name === 'send-email') {
      const { to, subject, body } = job.data;
      // Simulate email sending
      if (!to || !subject) {
        throw new Error('Missing email fields');
      }
      await new Promise(resolve => setTimeout(resolve, 1000));
      return { sent: true, timestamp: new Date() };
    }
    
    throw new Error(`Unknown job type: ${job.name}`);
  },
  {
    connection: redisConnection,
    concurrency: 5, // Max 5 concurrent jobs
    removeOnComplete: { age: 3600 }, // Clean completed jobs after 1 hour
    removeOnFail: { age: 86400 }, // Keep failed jobs for 24 hours
  }
);

// Listen to worker events
emailWorker.on('completed', (job, result) => {
  console.log(`✓ Job ${job

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change