Email Queue

You are a backend engineer implementing a reliable email queuing system. The user wants to queue emails for batch sending with retry logic and persistence.

What to check first

• Verify a message queue service is available (Redis, RabbitMQ, or AWS SQS)
• Check that your email service credentials are configured (SendGrid, AWS SES, or Mailgun)
• Confirm Node.js version supports async/await (v10+)

Steps

1. Install queue library: npm install bull (for Redis-backed queues) or npm install aws-sdk for SQS
2. Set up Redis connection or AWS credentials in your environment variables
3. Create a queue instance with retry configuration (maxAttempts, backoff strategy)
4. Define the email job processor function that calls your email service API
5. Add error handling with exponential backoff for failed jobs
6. Create a route/function to enqueue emails instead of sending directly
7. Monitor queue health with completed, failed, and delayed job counts
8. Test with mock email sends before connecting to production service

Code

const Queue = require('bull');
const nodemailer = require('nodemailer');

// Initialize queue with Redis connection
const emailQueue = new Queue('emails', {
  redis: {
    host: process.env.REDIS_HOST || '127.0.0.1',
    port: process.env.REDIS_PORT || 6379
  }
});

// Configure email transporter
const transporter = nodemailer.createTransport({
  service: 'gmail',
  auth: {
    user: process.env.EMAIL_USER,
    pass: process.env.EMAIL_PASSWORD
  }
});

// Process queued emails with retry logic
emailQueue.process(10, async (job) => {
  const { to, subject, html, text } = job.data;
  
  try {
    const result = await transporter.sendMail({
      from: process.env.EMAIL_FROM,
      to,
      subject,
      html,
      text
    });
    
    return { messageId: result.messageId, timestamp: new Date() };
  } catch (error) {
    console.error(`Email job ${job.id} failed:`, error.message);
    throw error; // Bull will retry based on backoff settings
  }
});

// Queue email for sending
async function queueEmail(emailData) {
  const job = await emailQueue.add(emailData, {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 2000
    },
    removeOnComplete: true,
    removeOnFail: false
  });
  
  return job.id;
}

// Event handlers for monitoring
emailQueue.on('completed', (job) => {
  console.log(`Email job ${job.id} sent successfully`);
});

emailQueue.on('failed', (job, err) => {
  console.

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change