Real-Time vs Bulk Email Validation: When to Use Each and How to Architect Both
Two Problems That Look the Same But Aren’t
A user types their email into your signup form. You need to know if it’s real before they hit submit. That’s one problem.
Your marketing team uploads a CSV of 50,000 contacts from a trade show. You need to clean it before the next campaign. That’s a completely different problem.
Both involve validating email addresses. But the architecture, latency constraints, and failure modes are nothing alike. Treating them the same way is how you end up with a signup form that takes 8 seconds to respond or a bulk import that crashes at row 12,000 with no recovery.
This guide covers both patterns with working code (Node.js and Rails), the technical decisions behind sync vs async, and the queue design that ties them together.
Real-Time Validation: The Synchronous Path
Real-time validation happens inline with form submission. The user clicks “Sign up,” your server validates the email, and the form either proceeds or shows an error. The entire round trip needs to complete before the user loses patience.
How fast is fast enough? Google recommends server response times under 200ms. Amazon found that every 100ms of added latency costs 1% in sales. For a signup form, you’ve got roughly 200-400ms of budget for validation before users notice the delay. Beyond 500ms, you’re hurting conversion.
That budget forces a technical decision: you can’t run a full SMTP handshake synchronously. Syntax checks take under 5ms. MX lookups run 50-200ms. But SMTP verification? 500ms to 3 seconds per address. Greylisted servers push that to 15 minutes. Not an option for a form submission.
The production pattern: run syntax + MX checks inline, defer SMTP to a background job.
Node.js: Real-Time Validation at Signup
// routes/signup.js
const express = require("express");
const router = express.Router();
router.post("/signup", async (req, res) => {
const { email, password } = req.body;
// Synchronous: syntax + MX (fast checks only)
const quickCheck = await truemail.validate(email, {
checks: ["syntax", "mx"],
timeout: 400,
});
if (quickCheck.status === "invalid") {
return res.status(422).json({
error: "That email doesn't look right. Check for typos.",
});
}
const user = await db.user.create({
data: { email, password: await hash(password), emailStatus: "pending" },
});
// Async: full SMTP verification in the background
await queue.add("deep-verify-email", {
userId: user.id,
email: user.email,
});
res.status(201).json({ user: { id: user.id, email: user.email } });
});
The 400ms timeout is intentional. If the email validation API doesn’t respond in time, the check fails open and the user gets through. You verify in the background. Never let an external service block signups.
Rails: Real-Time Validation with Fail-Open
# app/validators/realtime_email_validator.rb
class RealtimeEmailValidator < ActiveModel::EachValidator
def validate_each(record, attribute, value)
return if value.blank?
result = MailCop.validate(value, checks: %w[syntax mx], timeout: 0.4)
if result.status == "invalid"
record.errors.add(attribute, "doesn't look deliverable")
end
rescue MailCop::TimeoutError, MailCop::ConnectionError
# Fail open: accept the email, verify later
Rails.logger.warn("Email validation timeout for #{value}")
end
end
# app/controllers/registrations_controller.rb
class RegistrationsController < ApplicationController
def create
@user = User.new(user_params)
if @user.save
DeepEmailVerificationJob.perform_later(@user.id)
redirect_to dashboard_path
else
render :new, status: :unprocessable_entity
end
end
end
Two layers working together. The validator catches obvious garbage (typo domains, missing MX records) in under 400ms. The background job handles the slow SMTP check after the user is already in. Mailgun’s SLA confirms this split makes sense: they guarantee single validation responses within 200ms for 95% of volume, but that’s syntax + MX only.
Handling Timeouts Without Losing Users
What happens when the validation API is slow or down? You’ve got two choices: fail open or fail closed.
Fail closed means rejecting the signup. Safe for your email list, brutal for conversion. If your API has 99.9% uptime, that’s still 8.7 hours of downtime per year where nobody can register.
Fail open means accepting the email and verifying later. You let a few bad addresses through, but no real user gets blocked. Every production system I’ve seen chooses fail open for the synchronous path.
Cache recent results in Redis to reduce API calls and survive brief outages:
// middleware/email-cache.js
async function getCachedValidation(email) {
const cached = await redis.get(`email:validation:${email}`);
if (cached) return JSON.parse(cached);
const result = await truemail.validate(email, {
checks: ["syntax", "mx"],
timeout: 400,
});
// Cache for 24 hours
await redis.set(
`email:validation:${email}`,
JSON.stringify(result),
"EX",
86400
);
return result;
}
Same email hitting your signup form twice in a day? Serve the cached result. Zero API calls. Zero latency risk.
Bulk Validation: The Async Path
Bulk validation is a different animal. You’re not blocking a user. You’re processing a list. The constraints shift from “respond in 400ms” to “process 50,000 rows reliably without losing progress.”
The architecture: submit the batch, track progress, receive results via webhook callbacks. No long-lived HTTP connections. No polling loops.
Why You Can’t Just Loop Through the List
Tempting approach: iterate through the CSV, call the API once per row, wait for each response. Don’t.
At 200ms per validation (the fast path), 50,000 rows takes 2.7 hours sequentially. One network hiccup at row 30,000 and you start over. No parallelism, no checkpointing, no rate limit awareness.
Most validation APIs allow 10-100 concurrent requests. Sidekiq can process 20,000+ jobs per second with a single Redis instance. The queue-based approach isn’t just more reliable. It’s 50-100x faster.
Rails: Queue-Based Batch Processing
# app/jobs/validate_email_batch_job.rb
class ValidateEmailBatchJob < ApplicationJob
queue_as :bulk_validation
retry_on StandardError, wait: :polynomially_longer, attempts: 5
def perform(batch_id, email_ids)
batch = ValidationBatch.find(batch_id)
Contact.where(id: email_ids).find_each do |contact|
result = MailCop.validate(contact.email, timeout: 10)
contact.update!(
email_status: result.status,
disposable: result.disposable,
catch_all: result.catch_all,
validated_at: Time.current
)
batch.increment!(:processed_count)
end
end
end
# app/services/bulk_validation_service.rb
class BulkValidationService
SLICE_SIZE = 100
def self.enqueue(contacts, user:)
batch = ValidationBatch.create!(
user: user,
total_count: contacts.size,
processed_count: 0,
status: "processing"
)
contacts.pluck(:id).each_slice(SLICE_SIZE) do |ids|
ValidateEmailBatchJob.perform_later(batch.id, ids)
end
batch
end
end
Slicing into batches of 100 keeps individual jobs fast and recoverable. If job #7 fails, jobs #1-6 and #8+ still complete. The retry logic handles transient failures with polynomial backoff.
Node.js: BullMQ with Rate Limiting
// workers/bulk-validation.js
const { Queue, Worker } = require("bullmq");
const validationQueue = new Queue("bulk-validation", {
connection: { host: "localhost", port: 6379 },
});
// Rate limit: max 80 jobs per second
const worker = new Worker(
"bulk-validation",
async (job) => {
const { batchId, emails } = job.data;
for (const entry of emails) {
const result = await truemail.validate(entry.email, { timeout: 10000 });
await db.contact.update({
where: { id: entry.id },
data: {
emailStatus: result.status,
disposable: result.disposable,
catchAll: result.catch_all,
validatedAt: new Date(),
},
});
}
await db.validationBatch.update({
where: { id: batchId },
data: { processedCount: { increment: emails.length } },
});
},
{
connection: { host: "localhost", port: 6379 },
limiter: { max: 80, duration: 1000 },
concurrency: 5,
}
);
BullMQ’s built-in rate limiter caps throughput at 80 jobs per second across all workers. If the validation API returns 429 (Too Many Requests), you can dynamically throttle using worker.rateLimit(). No custom token bucket logic needed.
Progress Tracking for Bulk Jobs
Your users uploaded 50,000 contacts. They want to know how it’s going. Don’t make them refresh.
# app/controllers/api/validation_batches_controller.rb
module Api
class ValidationBatchesController < ApplicationController
def show
batch = current_user.validation_batches.find(params[:id])
render json: {
id: batch.id,
status: batch.status,
total: batch.total_count,
processed: batch.processed_count,
percent: batch.progress_percent,
estimated_minutes_remaining: batch.eta_minutes
}
end
end
end
Poll this endpoint every 5 seconds from your frontend. Or go further and push updates over WebSockets. Either way, the user sees real progress instead of a spinner with no ETA.
For a 50,000-contact list at 80 validations per second, that’s about 10 minutes of wall time. Set that expectation in your UI. Surprises frustrate users more than wait times.
When to Go Sync vs Async
The decision tree is simpler than it sounds.
Go synchronous when you’re validating a single email during a user-initiated action. Signup forms, checkout fields, profile updates. The user is waiting. Response time matters. Run syntax + MX inline, defer SMTP to background.
Go async when you’re processing a list. CSV imports, CRM exports, scheduled list hygiene. Nobody’s staring at a loading spinner. Reliability and throughput matter more than latency.
There’s a gray zone: validating 5-20 emails at once (a small API batch from a Django form or Next.js server action). For these, you can parallelize synchronous calls with a short timeout:
// Parallel validation for small batches (under 20 emails)
async function validateSmallBatch(emails) {
const results = await Promise.allSettled(
emails.map((email) =>
truemail.validate(email, {
checks: ["syntax", "mx"],
timeout: 400,
})
)
);
return results.map((r, i) => ({
email: emails[i],
status: r.status === "fulfilled" ? r.value.status : "unknown",
}));
}
Promise.allSettled is key here. If 3 out of 20 validations time out, you still get results for the other 17. Promise.all would throw away everything on the first failure.
The Combined Architecture
In production, you don’t pick one pattern. You build a system that handles both. Here’s how the pieces fit together:
Your signup form calls the real-time path: syntax + MX check with a 400ms timeout, fail open on errors, background SMTP verification via Sidekiq or BullMQ.
Your bulk import page accepts a CSV, creates a validation batch record, slices the list into chunks of 100, and enqueues each chunk as a background job. Progress updates stream back through a polling endpoint or WebSocket.
Both paths write to the same contacts table with the same status fields. Both use the same validation API. The only difference is the orchestration layer: synchronous with aggressive timeouts vs queued with rate limiting.
One API client. Two calling patterns. Shared result storage.
Sound too simple? That’s the point. The complexity lives in the queue configuration, retry logic, and failure handling, not in the validation logic itself. Keep the validation layer thin and push the orchestration concerns outward.
What to Watch in Production
Three metrics that tell you if your validation architecture is healthy.
P95 latency on the synchronous path. If your real-time validation consistently exceeds 400ms, investigate. DNS resolution issues, API throttling, and cold connection pools are the usual suspects.
Bulk queue depth and processing rate. If jobs pile up faster than workers drain them, you need more workers or lower concurrency per worker. Sidekiq’s dashboard shows this in real time.
Fail-open rate on the sync path. Track how often your timeout/error rescue fires. If it’s above 1%, your API client configuration needs tuning. If it’s above 5%, something is wrong with the API or your network path to it.
Build alerts for all three. The first sign of trouble is usually the fail-open rate creeping up, not users complaining.
Start Simple, Add Queues When You Need Them
If you’re pre-launch with 100 signups a day, inline validation with a timeout is all you need. Ten lines of code.
When you hit your first bulk import request, add the queue layer. Sidekiq for Rails, BullMQ for Node.js. Slice into batches of 100, rate-limit to stay under your API quota, track progress.
When bulk jobs start competing with signup validation for API quota, split them into separate queues with independent rate limits. Priority to real-time. Bulk can wait.
That’s it. Three stages that map to three growth phases. Don’t build stage three on day one.