Background Job

You are a backend engineer implementing asynchronous background job processing. The user wants to create a reliable system for queuing, processing, and tracking jobs that run outside the request/response cycle.

What to check first

• Verify you have a message queue (Redis, RabbitMQ) or job queue library installed: npm list bull or npm list bullmq
• Check your Node.js version supports Worker Threads: node --version (v10.5.0+)
• Confirm database connectivity for job persistence and status tracking

Steps

1. Install Bull or BullMQ: npm install bullmq redis (Bull is simpler for small projects, BullMQ is more robust)
2. Set up a Redis connection in your config file with host, port, and optional password
3. Create a queue instance by calling new Queue('jobName', { connection: redis }) with your Redis config
4. Define job producers that add jobs to the queue using queue.add('jobType', data, { options })
5. Create job consumers (workers) that process jobs with .process() or by instantiating a Worker class
6. Implement error handling with retry logic using { attempts: 3, backoff: { type: 'exponential' } }
7. Add event listeners for job completion (queue.on('completed')) and failures (queue.on('failed'))
8. Track job status in your database or use Bull's built-in job tracking with job.progress() for long-running tasks

Code

const Queue = require('bullmq').Queue;
const Worker = require('bullmq').Worker;
const Redis = require('redis');

// Initialize Redis connection
const redisConnection = {
  host: process.env.REDIS_HOST || 'localhost',
  port: process.env.REDIS_PORT || 6379,
};

// Create queue instance
const emailQueue = new Queue('emails', { connection: redisConnection });

// Producer: Add job to queue
async function sendEmailJob(userEmail, subject, body) {
  const job = await emailQueue.add(
    'send-email',
    { email: userEmail, subject, body },
    {
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: true,
      removeOnFailed: false,
    }
  );
  return job.id;
}

// Consumer: Process jobs with Worker
const worker = new Worker('emails', async (job) => {
  console.log(`Processing job ${job.id}: ${job.data.email}`);
  
  try {
    // Simulate email sending
    await new Promise((resolve) => setTimeout(resolve, 2000));
    
    if (job.data.email === 'invalid@test.com') {
      throw new Error('Invalid email address');
    }
    
    job.progress(100);
    return { status: 'sent',

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change