Queue Spikes And Scaling

In this chapter, we will cover how to deal with many jobs in your queue.

Setup

1. First let's seed 200 users, add the following line to your DatabaseSeeder run method:

database/seeders/DatabaseSeeder.php

\App\Models\User::factory(200)->create();

2. Run migrations and seeder:

php artisan migrate:fresh --seed

3. Create a new GenerateReport job:

php artisan make:job GenerateReport

app/Jobs/GenerateReport.php

namespace App\Jobs;
 
use App\Models\User;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
 
class GenerateReport implements ShouldQueue
{
    protected $user;
 
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
    public function __construct(User $user)
    {
        $this->user = $user;
    }
 
    public function handle(): void
    {
        // There should be report generation logic
        //Now we only imitate that
        sleep(rand(2, 5));
 
        info('Report for user ' . $this->user->id . ' has been generated.');
    }
}

Instead of actually generating a report we simulate that using the sleep function randomly between 2 and 5 seconds. This will be enough for that part.

When the job is completed success message will be logged to the storage/logs/laravel.log file so we can see when jobs were processed.

4. To make it easier to dispatch jobs let's create a new MakeReports command:

php artisan make:command MakeReports

It will dispatch a GenerateReport job for every user. Contents of the file should be as follows:

app/Console/Commands/MakeReports.php

namespace App\Console\Commands;
 
use App\Jobs\GenerateReport;
use App\Models\User;
use Illuminate\Console\Command;
 
class MakeReports extends Command
{
    protected $signature = 'app:make-reports';
 
    protected $description = 'Generates reports for all users.';
 
    public function handle()
    {
        $this->components->info('Dispatching jobs.');
 
        $this->withProgressBar(User::all(), function ($user) {
            GenerateReport::dispatch($user);
 
            info('Job for user ' . $user->id . ' has been dispatched.');
        });
 
        $this->line('');
        $this->line('');
        $this->components->info('Command completed successfully.');
    }
}

Monitoring Queue Worker

1. Now let's setup the supervisor with the following configuration:

[program:laravel-worker]
directory=/home/web/laravel-queues
command=php artisan queue:work --max-time=3600
 
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=web
numprocs=1
redirect_stderr=true
stdout_logfile=/home/web/.supervisor/laravel-worker.log
stopwaitsecs=3600

Then reread and update the supervisor configuration.

supervisorctl reread
supervisorctl update

2. Queue worker should be running by now and we can dispatch jobs using our artisan command:

php artisan app:make-reports

To see the current status we can run the queue:monitor command, and the argument default is the queue name we want to monitor:

php artisan queue:monitor default

Queue name ................................................... Size / Status
[database] default ................................................ [198] OK

Here we can see the remaining jobs left in the queue which is 198 and the status is OK.

3. If your queue receives a sudden influx of jobs, it could become overwhelmed, leading to a long wait time for jobs to complete. This is what actually is happening, all these small jobs will take over 11 minutes to complete on average. If you wish, Laravel can alert you when your queue job count exceeds a specified threshold.

queue:monitor accepts --max flag for job count threshold:

php artisan queue:monitor default --max=100

Queue name ................................................... Size / Status
[database] default ............................................. [198] ALERT

Now we can see that status has changed to ALERT, but by default, nothing else will happen.

4. First we need to create a new notification, let's call it QueueHasLongWaitTime by running the make:notification artisan command:

art make:notification QueueHasLongWaitTime

With the following content:

app/Notifications/QueueHasLongWaitTime.php

namespace App\Notifications;
 
use Illuminate\Notifications\Messages\MailMessage;
use Illuminate\Notifications\Notification;
 
class QueueHasLongWaitTime extends Notification
{
    protected $connection;
 
    protected $queue;
 
    protected $size;
 
    public function __construct($connection, $queue, $size)
    {
        $this->connection = $connection;
        $this->queue      = $queue;
        $this->size       = $size;
    }
 
    public function via(object $notifiable): array
    {
        return ['mail'];
    }
 
    public function toMail(object $notifiable): MailMessage
    {
        return (new MailMessage)
            ->subject(sprintf(
                '[Alert] %s app has %s queue spike',
                config('app.name'),
                $this->queue
            ))
            ->line(sprintf(
                "'%s' queue on '%s' connection has reached %s jobs.",
                $this->queue,
                $this->connection,
                $this->size
            ));
    }
 
    public function toArray(object $notifiable): array
    {
        return [
            'connection' => $this->connection,
            'queue'      => $this->queue,
            'size'       => $this->size,
        ];
    }
}

5. To send a notification we can bind this notification in the boot method of EventServiceProvider to the QueueBusy event, this event is fired when you run the queue:monitor command and it reaches the job threshold:

app/Providers/EventServiceProvider.php

use Illuminate\Support\Facades\Event;
use Illuminate\Queue\Events\QueueBusy;
use Illuminate\Support\Facades\Notification;
use App\Notifications\QueueHasLongWaitTime;
 
 
// ...
 
public function boot(): void
{
    Event::listen(function (QueueBusy $event) {
        Notification::route('mail', '[email protected]')
            ->notify(new QueueHasLongWaitTime(
                $event->connection,
                $event->queue,
                $event->size
            ));
    });
}

Now if you run queue:monitor default --max=100 again while you have more than 100 jobs pending in the default queue and have Mailtrap configured email should be delivered informing you about a spike of jobs.

php artisan queue:monitor default --max=100

6. At the time notification will be sent only when you manually run the queue:monitor command. It can be automated by adding it to the scheduler to run every minute.

We need to add $schedule->command('queue:monitor default --max=100')->everyMinute(); line into schedule method of Kernel.php file:

app/Console/Kernel.php

namespace App\Console;
 
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;
 
class Kernel extends ConsoleKernel
{
    protected function schedule(Schedule $schedule): void
    {
        $schedule->command('queue:monitor default --max=100')->everyMinute();
    }
 
    protected function commands(): void
    {
        $this->load(__DIR__ . '/Commands');
 
        require base_path('routes/console.php');
    }
}

To verify it is recognized run the schedule:list Artisan command:

php artisan schedule:list

Output:

* * * * *  php artisan queue:monitor ......... Next Due: 14 seconds from now

7. Now that we have learned how to define a scheduled task it is time to actually run it on a server, just defining tasks in Laravel itself won't run them automatically.

There are several ways we can do that:

Cron On Server

To run scheduled tasks we need to add a single cron configuration entry to our server that runs the schedule:run command every minute.

* * * * * cd /path-to-your-project && php artisan schedule:run >> /dev/null 2>&1

For cron configuration please refer to your Linux distribution's documentation.

Locally

To quickly run the scheduler locally without configuring cron we can run the schedule:work Artisan command:

php artisan schedule:work

Using Laravel Forge

If you're using Laravel Forge service it can manage cron entries for you. In the server section, there's a Scheduler menu entry and the command may look as follows php8.1 /home/forge/example.org/artisan schedule:run, consider updating the path to your actual application and make sure the Every Minute option is selected.

Scaling workers

So far so good, now we are automatically notified if jobs reach a certain threshold, in a perfect scenario we would like to receive as least as possible such emails.

As you may notice we have only one process running to the process queue in our supervisor configuration numprocs=1. Let's increase it to numprocs=10. By increasing the number of processes we allow queue jobs to be processed in parallel by 10 at a time and all 200 jobs will be processed in a bit more than a minute on average instead of 11+ minutes.

In the laravel.log file we can see that output is no longer sequential and this indicates that workers work in parallel.

storage/logs/laravel.log

Report for user 5 has been generated.
Report for user 8 has been generated.
Report for user 3 has been generated.
Report for user 9 has been generated.
Report for user 1 has been generated.
Report for user 7 has been generated.

Sometimes it is not very efficient to keep a lot of workers running idle, especially when your application experiences queue spikes at certain periods of the day.

We can add another configuration file to the supervisor as follows:

[program:laravel-worker-busy-hours-pool]
directory=/home/web/laravel-queues
command=php artisan queue:work --max-time=3600
 
process_name=%(program_name)s_%(process_num)02d
autostart=false
autorestart=true
stopasgroup=true
killasgroup=true
user=web
numprocs=5
redirect_stderr=true
stdout_logfile=/home/web/.supervisor/laravel-worker-busy-hours-pool.log
stopwaitsecs=3600

Notice autostart value is set to false as we do not want to start it automatically. Then we can add a cron job to automatically start and stop that pool of workers. Let's see this example:

0 9 * * 1-5 root supervisorctl start laravel-worker-busy-hours-pool:*
0 10 * * 1-5 root supervisorctl stop laravel-worker-busy-hours-pool:*

For cron configuration please refer to your Linux distribution's documentation.

This means At 09:00 on every day of the week from Monday through Friday start additional 5 workers and stop them at 10:00.

Laravel Horizon

1. Let's set up Supervisor for Horizon first with the following configuration:

[program:laravel-horizon]
directory=/home/web/laravel-queues
command=php artisan horizon
 
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=web
numprocs=1
redirect_stderr=true
stdout_logfile=/home/web/.supervisor/laravel-horizon.log
stopwaitsecs=3600

And change queue driver:

.env

QUEUE_CONNECTION=redis

For more details and Horizon setup please refer to Chapter 3.

2. The way to get notifications depending on job count using Horizon works in the same way as described in the previous section by listening for the QueueBusy event, but Horizon itself provides that functionality in a bit different manner.

The main difference here is that it depends not on job count, but on wait time, which means how long a job has to wait in the queue until it is being processed. This is more useful than job count because it is easier to decide how long is too long than try to estimate how many jobs are too much.

This method can be called from the boot method of your application's HorizonServiceProvider:

app/Providers/HorizonServiceProvider.php

public function boot(): void
{
    parent::boot();
 
    Horizon::routeMailNotificationsTo('[email protected]');
}

3. Threshold wait time in seconds is defined in Horizon configuration.

config/horizon.php

'waits' => [
    'redis:default' => 60,
],

4. Now with Horizon let's dispatch 600 jobs by running the app:make-reports command 3 times.

php artisan app:make-reports
php artisan app:make-reports
php artisan app:make-reports

5. In the Dashboard we can immediately see that Horizon estimates Max Wait Time for a new job to start being processed in 11 minutes:

And not long enough email notification arrives:

Scaling workers

Now let's look up to Horizon how it handles queue scaling and how we can reduce wait times. The default configuration looks as follows:

config/horizon.php

// ...
 
'defaults' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'auto',
            'autoScalingStrategy' => 'time',
            'maxProcesses' => 1,
            'maxTime' => 0,
            'maxJobs' => 0,
            'memory' => 128,
            'tries' => 1,
            'timeout' => 60,
            'nice' => 0,
        ],
    ],
 
    'environments' => [
        'production' => [
            'supervisor-1' => [
                'maxProcesses' => 10,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
            ],
        ],
 
    // ...
 
    ],

By default, the scaling strategy is set by the balance key which is set auto value. The worker count will be increased or decreased depending on how busy is queue.

When the strategy is set to auto the balanceMaxShift and balanceCooldown values determine how quickly Horizon will scale to meet worker demand.

In this example, it means every 3 seconds increase or decrease worker count by 1. This helps to quickly deal with unexpected spikes and save resources when the load is low.

The autoScalingStrategy configuration value determines if Horizon will assign more worker processes to queues based on the total amount of time it will take to clear the queue (time strategy) or by the total number of jobs on the queue (size strategy).

Optionally you can specify minProcesses, default is 1.

When the queue is empty we can see that there's only one process waiting for any job to be enqueued.

If we dispatch 200 jobs again, over time worker count will increase to the maxProcesses value.

In this scenario we won't get a notification because Horizon handled a long queue automatically, so no actions were needed to be taken.

simple strategy just splits processes between the queues.

For example, if we have this configuration:

// ...
 
'defaults' => [
        'supervisor-1' => [
            // ...
            'queue' => ['default', 'invoices'],
            'balance' => 'simple',
            // ...
        ],
    ],
 
    'environments' => [
        'production' => [
            'supervisor-1' => [
                'processes' => 20
            ],
        ],
 
    // ...
 
    ],

All workers will be split between 2 queues (default and invoices) and 10 processes will be running for each one all the time.