Batch Email Validation with Background Jobs: Sidekiq, Bull, and Celery Patterns
50,000 Emails and a Frozen Web Server
Your marketing team just uploaded a CSV with 200,000 contacts. They click “Validate All.” Your web process starts looping through addresses, firing API calls one by one. Thirty seconds in, the request times out. The browser shows a 502. Half the list is unvalidated, and nobody knows which half.
Synchronous bulk validation doesn’t work. At 200ms per API call, 50,000 emails take nearly three hours of wall-clock time. Your web process can’t hold a connection that long. Your email validation API provider will throttle you before you hit 1,000. And if anything fails mid-run, you start over.
The answer is background jobs. Chunk the list, enqueue the work, process it asynchronously, and report progress back to the user. The pattern is identical whether you’re running Sidekiq in Ruby, BullMQ in Node.js, or Celery in Python. The framework changes. The architecture doesn’t.
The Universal Pattern
Every background validation system follows the same five steps, regardless of language:
- Client submits a list of emails.
- Server chunks the list into batches of 100-500 addresses.
- Each chunk becomes a job on a queue.
- Workers pull jobs, call the validation API with rate limiting, and store results.
- Frontend polls (or listens via WebSocket) for progress updates.
Redis sits at the center of all three frameworks. It’s the queue backend for Sidekiq and BullMQ, and a common broker for Celery. It also works as a result cache, so you don’t re-validate emails you checked yesterday. One dependency, three roles.
Why chunk into 100-500 instead of sending all 200,000 as one job? Smaller jobs are restartable. If a worker crashes processing email #47 out of 200,000, you lose everything. If it crashes on chunk #12 out of 400, you lose 500 emails and retry that chunk. Smaller jobs also distribute across multiple workers for parallel processing.
Sidekiq (Ruby): The Rails Way
Most Rails apps already have Sidekiq running. Adding batch validation means writing a job, a chunking service, and a progress tracker.
Here’s the worker:
# app/jobs/validate_email_chunk_job.rb
class ValidateEmailChunkJob
include Sidekiq::Job
sidekiq_options queue: :validation, retry: 3
sidekiq_retry_in do |count|
(count + 1) * 30 # 30s, 60s, 90s
end
def perform(batch_id, emails)
batch = ValidationBatch.find(batch_id)
emails.each do |email|
next if ValidationResult.exists?(batch_id: batch_id, email: email)
result = MailCop.validate(email, timeout: 10)
ValidationResult.create!(
batch_id: batch_id,
email: email,
status: result.status,
disposable: result.disposable,
catch_all: result.catch_all
)
batch.increment!(:processed_count)
rescue MailCop::RateLimitError
sleep 2
retry
rescue MailCop::TimeoutError => e
ValidationResult.create!(
batch_id: batch_id, email: email,
status: "error", error_message: e.message
)
batch.increment!(:processed_count)
end
end
end
The next if ValidationResult.exists? line makes the job idempotent. If Sidekiq retries after a crash, it skips emails already processed. Without this, retries create duplicates and inflate your API bill.
Now the chunking service that kicks everything off:
# app/services/batch_validation_service.rb
class BatchValidationService
CHUNK_SIZE = 250
def self.start(emails, user:)
batch = ValidationBatch.create!(
user: user,
total_count: emails.size,
processed_count: 0,
status: "processing"
)
emails.uniq.each_slice(CHUNK_SIZE) do |chunk|
ValidateEmailChunkJob.perform_async(batch.id, chunk)
end
batch
end
end
The emails.uniq call deduplicates before enqueuing. A 200,000-row CSV with 15% duplicates means 30,000 fewer API calls. That’s real money.
Progress tracking is straightforward. The processed_count column gets incremented per email. Your frontend polls a simple endpoint:
# In your controller
def progress
batch = ValidationBatch.find(params[:id])
render json: {
total: batch.total_count,
processed: batch.processed_count,
percent: (batch.processed_count.to_f / batch.total_count * 100).round(1),
status: batch.status
}
end
BullMQ (Node.js): Queue + Worker
BullMQ gives you more granular control over concurrency and job events than Sidekiq does out of the box. If you’re already running a Node.js email validation service or an email validation microservice, BullMQ slots right in.
The queue setup:
// src/queues/validation.js
import { Queue, Worker } from "bullmq";
import IORedis from "ioredis";
const connection = new IORedis(process.env.REDIS_URL, {
maxRetriesPerRequest: null,
});
export const validationQueue = new Queue("email-validation", {
connection,
defaultJobOptions: {
attempts: 3,
backoff: { type: "exponential", delay: 5000 },
removeOnComplete: { age: 86400 },
},
});
The worker processes each chunk and reports progress per email:
// src/workers/validation.js
import { Worker } from "bullmq";
const worker = new Worker(
"email-validation",
async (job) => {
const { batchId, emails } = job.data;
const results = [];
for (let i = 0; i < emails.length; i++) {
const email = emails[i];
try {
const result = await validateWithRateLimit(email);
results.push({ email, ...result });
} catch (err) {
results.push({ email, status: "error", error: err.message });
}
await job.updateProgress(((i + 1) / emails.length) * 100);
}
await storeResults(batchId, results);
return { batchId, count: results.length };
},
{ connection, concurrency: 5 }
);
That concurrency: 5 means five chunks process simultaneously. With 250 emails per chunk, that’s 1,250 emails in flight across your workers at any moment. Pair this with rate limiting (covered below) to avoid hammering your validation API.
Enqueueing the chunks from your API endpoint:
// src/routes/batch.js
app.post("/validate/batch", async (req, res) => {
const { emails } = req.body;
const batchId = crypto.randomUUID();
const chunks = chunkArray([...new Set(emails)], 250);
const jobs = chunks.map((chunk, i) => ({
name: `batch-${batchId}-${i}`,
data: { batchId, emails: chunk },
}));
await validationQueue.addBulk(jobs);
res.status(202).json({ batchId, totalEmails: emails.length, chunks: chunks.length });
});
function chunkArray(arr, size) {
const result = [];
for (let i = 0; i < arr.length; i += size) {
result.push(arr.slice(i, i + size));
}
return result;
}
addBulk enqueues all chunks in a single Redis round-trip. For 200,000 emails split into 800 chunks, that’s one Redis call instead of 800.
BullMQ’s built-in exponential backoff handles transient failures. First retry at 5 seconds, second at 10, third at 20. If the validation API is down for a minute, your jobs survive. If it’s down for an hour, they land in the failed set where you can inspect and retry manually.
Celery (Python): Tasks and Chords
Python shops typically reach for Celery. The pattern maps cleanly: a task per chunk, a group for parallel execution, and a chord callback to finalize the batch.
The task definition:
# tasks/validation.py
from celery import shared_task
from celery.utils.log import get_task_logger
import time
logger = get_task_logger(__name__)
@shared_task(
bind=True,
max_retries=3,
default_retry_delay=30,
rate_limit="10/m",
)
def validate_chunk(self, batch_id, emails):
results = []
for i, email in enumerate(emails):
try:
result = call_validation_api(email)
results.append({"email": email, **result})
except RateLimitError:
time.sleep(2)
result = call_validation_api(email)
results.append({"email": email, **result})
except ApiTimeoutError as exc:
results.append({"email": email, "status": "error"})
update_progress(batch_id, i + 1, len(emails))
store_results(batch_id, results)
return {"batch_id": batch_id, "processed": len(results)}
The rate_limit="10/m" on the task decorator tells Celery: don’t execute this task more than 10 times per minute per worker. With 4 workers, that’s 40 chunks per minute. At 250 emails per chunk, you’re processing 10,000 emails per minute while staying well within typical API rate limits.
Kick off all chunks in parallel with a group:
# services/batch_validation.py
from celery import group, chord
from tasks.validation import validate_chunk, finalize_batch
def start_batch_validation(emails, user_id):
unique_emails = list(set(emails))
batch = create_batch_record(user_id, len(unique_emails))
chunks = [
unique_emails[i:i + 250]
for i in range(0, len(unique_emails), 250)
]
callback = finalize_batch.s(batch.id)
job = chord(
[validate_chunk.s(batch.id, chunk) for chunk in chunks]
)(callback)
return batch
The chord waits for all chunks to finish, then calls finalize_batch with the collected results. That callback updates the batch status to “completed” and triggers any downstream work (sending notification emails, generating reports).
How does this compare to the Sidekiq and BullMQ approaches? Celery’s rate_limit is built into the task decorator. Sidekiq needs a separate gem (like sidekiq-rate-limiter) or manual sleep calls. BullMQ handles it through the limiter option on the queue. Same outcome, different API surfaces.
Rate Limiting: Don’t Hammer Your Provider
All three frameworks need rate limiting against the upstream API. Without it, 20 workers firing simultaneously will blow past any provider’s rate limit and get you throttled (or banned).
The approaches differ by framework, but the goal is identical: rate limiting email validation calls to stay under your provider’s threshold.
Sidekiq doesn’t have built-in rate limiting. Use Redis directly:
# app/services/rate_limited_validator.rb
class RateLimitedValidator
LIMIT = 50 # requests per second
KEY = "validation_rate_limit"
def self.validate(email)
loop do
count = Redis.current.incr(KEY)
Redis.current.expire(KEY, 1) if count == 1
break if count <= LIMIT
sleep 0.1
end
MailCop.validate(email)
end
end
BullMQ has a built-in limiter on the queue:
const queue = new Queue("email-validation", {
connection,
limiter: { max: 50, duration: 1000 },
});
Celery uses the task decorator as shown above, or a custom semaphore with Redis for finer control.
The number 50 requests per second isn’t arbitrary. Most validation APIs allow 50-100 RPS on standard plans. Check your provider’s docs and set your limit 20% below the actual cap. Hitting the ceiling triggers 429 responses that waste time on retries.
Progress Tracking That Actually Works
Users don’t care about queue internals. They want a progress bar. Here’s how to surface it.
The simplest approach: store progress in your database. Every chunk updates a counter. Your frontend polls every 2-3 seconds. This works for batch vs real-time validation because the polling interval matches the pace of batch processing. Nobody needs sub-second updates for a job that runs 20 minutes.
For real-time updates without polling, use Server-Sent Events:
// Node.js SSE endpoint
app.get("/validate/:batchId/progress", (req, res) => {
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
const interval = setInterval(async () => {
const progress = await getBatchProgress(req.params.batchId);
res.write(`data: ${JSON.stringify(progress)}\n\n`);
if (progress.status === "completed") {
clearInterval(interval);
res.end();
}
}, 2000);
req.on("close", () => clearInterval(interval));
});
SSE over WebSockets for this use case. Why? It’s one-directional (server to client), works through most proxies without special config, and reconnects automatically. WebSockets are overkill for a progress bar.
Job Design Principles That Save You at 3 AM
Three rules that prevent debugging sessions at unholy hours.
Make every job idempotent. If a worker crashes and retries, it shouldn’t re-validate emails it already checked. Use a unique constraint on (batch_id, email) or check before processing. Duplicate API calls cost money and skew your results.
Second, deduplicate before enqueuing. Run uniq (Ruby), new Set() (JavaScript), or set() (Python) on the email list before chunking. A 200,000-row CSV exported from a CRM commonly has 10-20% duplicates. According to Gartner, organizations estimate that bad data costs them an average of $12.9 million per year. Deduplication alone can cut your validation costs by thousands of dollars annually on large lists.
Third, handle partial failures gracefully. If one email in a 250-email chunk throws an unrecoverable error, log it and move on. Don’t let one bad address kill the entire chunk. Store the error, let the user see it in results, and keep processing. HubSpot’s research shows that email databases degrade by about 22.5% every year from bounces, job changes, and domain expirations. You’ll see errors. Plan for them.
Redis: The Thread That Connects Everything
Redis isn’t just the queue backend. It’s the shared infrastructure across your entire batch validation pipeline.
In Sidekiq, Redis stores the job queue, retry state, and dead-letter jobs. In BullMQ, it stores the queue, job data, progress updates, and completed results. In Celery with Redis as broker, it handles task routing and (optionally) result storage.
Layer a validation cache on top and Redis pulls quadruple duty: queue, retry store, result cache, and rate-limit counter. A single Redis instance handles all four for most workloads. At 100,000 cached validation results (roughly 50-80MB of memory), you’re nowhere near Redis’s limits.
The cache alone pays for itself. Mailtrap reports that re-validating previously checked addresses accounts for 30-40% of API calls in apps without caching. A 24-hour TTL on valid results and a 1-hour TTL on invalid results (to catch temporary failures) cuts your API spend significantly.
Pick Your Stack, Ship the Pattern
The framework doesn’t matter as much as the pattern. Chunk your list. Enqueue the chunks. Rate limit the API calls. Make jobs idempotent. Track progress. Store results.
Whether you’re writing Ruby, JavaScript, or Python, the architecture is the same. Redis in the middle, workers on the edges, your app collecting results. The code samples above are production-ready starting points. Grab the one that matches your stack, wire it to your validation API, and stop freezing your web server on CSV imports.
Start with 250 emails per chunk and 50 requests per second. Tune from there based on your provider’s rate limits and your Redis memory. A 200,000-email list at those settings finishes in under an hour. Your marketing team won’t even have time to get coffee.
Well, maybe one cup.