WorkCommunication infrastructureflagship
High-Volume Campaign Platform
Evolving a mature, multi-tenant Laravel campaign platform into an asynchronous, observable sending system engineered for billion-scale workloads, without stopping the product.
System and tool names have been generalised.
Stack as evidence
Laravel · Redis · MySQL · ClickHouse · Elasticsearch · AWS · Docker
Concerns
Multi-tenancy · Queue orchestration · Recipient streaming · Deferred webhooks · Columnar analytics · Incremental migration
- 01
Recipients
Segments resolved per tenant
- 02
Streaming
Cursor-based, never loaded whole
- 03
Queue orchestration
Chunk, order, pace
- 04
Redis
Queues · dedup · throttles · cooldowns
- 05
Batch processing
Workers pull bounded batches
- 06
Send infrastructure
Providers, failover, webhooks deferred
- 07
Analytics
Events, not row updates
- 08
ClickHouse
Per-campaign send data, columnar
01The problem
A mature system meeting workloads it was never shaped for
The platform was already a working, revenue-carrying Laravel application serving many tenants. Campaigns had grown from lists into very large audiences, and the paths that were fine at the previous scale started to bind: recipient sets loaded eagerly, sends coupled to the request lifecycle, analytics written as row updates against the same MySQL the product ran on, and provider webhooks processed synchronously as they arrived.
The hard part was not designing a high-volume sending system on a whiteboard. It was doing it inside a system that customers were using every day.
02The constraints
What could not simply be changed.
No rewrite
The business could not pause for a new platform. Every change had to ship into the running application.
Tenancy is load-bearing
Isolation between tenants was part of the product contract, so every new execution path had to respect it.
Existing behaviour is the spec
Campaign semantics customers relied on had to survive intact while the machinery underneath them changed.
Providers are outside our control
Sending providers rate limit, fail and deliver webhooks in bursts. The system had to absorb that rather than pass it on.
03The architecture
The redesign treats a campaign as a stream rather than a set. Recipients are resolved through Elasticsearch segmentation and streamed with cursors, so no worker ever holds an audience in memory. Streaming feeds a queue orchestration layer that chunks recipients into bounded batches, orders them, and paces dispatch onto Redis-backed queues.
Redis does more than hold jobs. It carries the coordination state that keeps a distributed send correct: deduplication keys so a recipient is never sent twice when a batch is retried, per-tenant and per-provider throttles, and cooldowns that back off when a provider pushes back. Workers pull batches, hand them to the send infrastructure, and record events.
Provider webhooks are accepted immediately and processed later. Acknowledging fast and deferring the work turns a bursty external input into a smooth internal queue. Per-campaign send data and engagement events flow into ClickHouse, where columnar storage answers the analytical questions the product asks without touching the transactional MySQL database.
All of it was introduced incrementally: new paths ran beside old ones, were switched on per workload, and were watched through the observability that had to be built first.
04Engineering decisions
01
Stream recipients instead of loading them
Memory ceilings were the first wall. Cursor-based streaming made the audience size irrelevant to worker footprint and let batching become a tuning parameter instead of a limit.
02
Put coordination state in Redis, not MySQL
Deduplication, throttling and cooldowns are high-frequency, short-lived and latency-sensitive. Redis is the right home for that; a relational table under row locks is not.
03
Defer webhook processing
Providers deliver events in bursts that had nothing to do with our capacity. Accept, enqueue, process. The database stopped being hammered by someone else’s traffic pattern.
04
Move analytics to ClickHouse
Send data is append-heavy and read in aggregates. A columnar store answers those queries orders of magnitude more comfortably than the OLTP database, and removing them from MySQL made the product faster too.
05
Segment in Elasticsearch, own truth in MySQL
Elasticsearch is excellent at answering “who matches this” and poor as a system of record. It was used for exactly one of those jobs.
05The trade-offs
What complexity I introduced, on purpose.
- More moving parts. Redis, ClickHouse and Elasticsearch each add operational surface, so observability had to precede them rather than follow.
- Eventual consistency in analytics and segmentation. The product now has a small, explicit lag where it previously had none, in exchange for not sharing a database with billions of potential writes.
- Two execution paths during migration. Running old and new side by side is safer than a cut-over, but it means carrying both until the last workload moves.
06The outcome
- Sending moved off the request lifecycle into orchestrated, throttled, deduplicated batch execution.
- Analytics and segmentation left the transactional database for stores built for those workloads.
- Provider failures and webhook bursts became queue depth rather than outages.
- The product kept shipping features throughout. Customers did not experience a migration.
07The lesson
The system was already mature. The goal was never to rewrite it. The goal was to isolate bottlenecks, introduce better execution paths, and evolve the architecture without breaking existing functionality. Most of the engineering judgement was in choosing what not to touch.