Queue Job Batching in Laravel: How It Works

Laravel 8 introduced a neat feature: you can dispatch a group of jobs to the queue, run them in parallel, and then execute specific logic when any fail or when all complete. Here is a look under the hood.

A Quick Example

From the official docs:

$batch = Bus::batch([
    new ProcessPodcast(Podcast::find(1)),
    new ProcessPodcast(Podcast::find(2)),
    new ProcessPodcast(Podcast::find(3)),
    new ProcessPodcast(Podcast::find(4)),
    new ProcessPodcast(Podcast::find(5)),
])->then(function (Batch $batch) {
    // All jobs completed successfully...
})->catch(function (Batch $batch, Throwable $e) {
    // First batch job failure detected...
})->finally(function (Batch $batch) {
    // The batch has finished executing...
})->name('Process Podcasts')
  ->allowFailures(false)
  ->onConnection('redis')
  ->onQueue('podcasts')
  ->dispatch();

Storing the Batch

When you call dispatch(), the batch information is written to the database first. The store() method in Illuminate\Bus\DatabaseBatchRepository inserts a row:

$this->connection->table($this->table)->insert([
    'id'            => $id,
    'name'          => $batch->name,
    'total_jobs'    => 0,
    'pending_jobs'  => 0,
    'failed_jobs'   => 0,
    'failed_job_ids'=> '[]',
    'options'       => serialize($batch->options),
    'created_at'    => time(),
    'cancelled_at'  => null,
    'finished_at'   => null,
]);

Each batch gets a UUID and an optional name. The options array holds everything else: the closures passed to then(), catch(), and finally(), plus allowFailures, onConnection, and onQueue settings. Closures are serialized using Opis\Closure\SerializableClosure so they survive the database round trip.

Dispatching the Jobs

Once the batch record exists, Laravel attaches the batch ID to every job and dispatches them all at once using the queue's bulk() method:

$jobs->each->withBatchId($this->id);

$this->repository->incrementTotalJobs($this->id, count($jobs));

$this->queue
    ->connection($this->options['connection'] ?? null)
    ->bulk($jobs->all(), $data = '', $this->options['queue'] ?? null);

The total job count is incremented in the database so Laravel can later determine when the batch is done. Using bulk() means all jobs are sent to the queue store in a single transaction, rather than dispatching them one by one.

Monitoring the Batch

On Success

After each successful job execution, recordSuccessfulJob() is called. It decrements the pending count and checks whether callbacks should fire:

$counts = $this->decrementPendingJobs($jobId);

if ($counts->pendingJobs === 0) {
    $this->repository->markAsFinished($this->id);
}

if ($counts->pendingJobs === 0 && $this->hasThenCallbacks()) {
    collect($this->options['then'])->each(/* Invoke */);
}

if ($counts->allJobsHaveRanExactlyOnce() && $this->hasFinallyCallbacks()) {
    collect($this->options['finally'])->each(/* Invoke */);
}

When there are no more pending jobs, Laravel marks the batch as finished by writing to the finished_at timestamp. Any then callbacks are triggered if everything ran successfully, and any finally callbacks fire once every job has been attempted at least once.

On Failure

When a job exhausts all its attempts, recordFailedJob() is called:

$counts = $this->incrementFailedJobs($jobId);

if ($counts->failedJobs === 1 && ! $this->allowsFailures()) {
    $this->cancel();
}

if ($counts->failedJobs === 1 && $this->hasCatchCallbacks()) {
    $batch = $this->fresh();
    collect($this->options['catch'])->each(/* Invoke */);
}

if ($counts->allJobsHaveRanExactlyOnce() && $this->hasFinallyCallbacks()) {
    $batch = $this->fresh();
    collect($this->options['finally'])->each(/* Invoke */);
}

If the batch does not allow failures, the first failure cancels the entire batch, and the cancelled_at field gets set to the current timestamp. The catch callback fires only on the first failure, and finally fires once all jobs have had at least one attempt.

Both recordSuccessfulJob() and recordFailedJob() are called from within Illuminate\Queue\CallQueuedHandler.

Exploring Further

If you want to dig deeper into the batch system, the key classes are:

• Illuminate\Bus\Dispatcher::batch(): entry point for creating batches.
• Illuminate\Bus\PendingBatch: the fluent builder you chain methods on.
• Illuminate\Bus\DatabaseBatchRepository: handles database storage and tracking.
• Illuminate\Bus\Batch: the in-memory representation of a running batch.