Health Check - Queue Errors

Self-Hosted | v5
1

The system health check (Self-Hosted v5, latest version, Mac Client APP) shows a Red triangle in the Queue section with 634 pending jobs and 8 failed jobs. Clicking on “View Last Queue Error” points to SMTP timeout error: (Illuminate\Queue\TimeoutExceededException: App\Listeners\Invoice\InvoiceEmailedNotification has timed out in /public_html/vendor/laravel/framework/src/Illuminate/Queue/TimeoutExceededException.php:15)

How do I track the pending jobs, they do not seem to go down automatically?

I am running the queue cronjob every minute as recommended:
* * * * * php8.2 /public_html/artisan schedule:run >> /dev/null 2>&1

I also ran the queue by hand like below:
php8.2 artisan schedule:run
and it runs without any errors.

How do I track this? When will these errors disappear?

Thanks!

2

HI,

Since the queue has been enabled you need to add the queue:work cron process the jobs.

Note: you may want to clear the jobs otherwise once it’s fixed many invoices will be processed. I believe you can use this command: php artisan queue:clear

1 Like
3
  • How do I know the queue is enabled? ( I do see the .env variable defined as QUEUE_CONNECTION=database. Is that it?)
  • Your answer gave me a hint to check the queue related artisan commands:
 queue
  queue:clear                           Delete all of the jobs from the specified queue
  queue:failed                          List all of the failed queue jobs
  queue:flush                           Flush all of the failed queue jobs
  queue:forget                          Delete a failed queue job
  queue:listen                          Listen to a given queue
  queue:monitor                         Monitor the size of the specified queues
  queue:prune-batches                   Prune stale entries from the batches database
  queue:prune-failed                    Prune stale entries from the failed jobs table
  queue:restart                         Restart queue worker daemons after their current job
  queue:retry                           Retry a failed queue job
  queue:retry-batch                     Retry the failed jobs for a batch
  queue:work                            Start processing jobs on the queue as a daemon
  • php8.2 artisan queue:failed lists the 8 failed jobs:

  2023-12-02 06:13:02 7518f165-f46e-4893-84b4-52e449758847 ..... database@default Illuminate\Events\CallQueuedListener
  2023-12-02 05:49:01 b7ec4c23-e9fe-44e8-9632-18162f1b329b ..... database@default Illuminate\Events\CallQueuedListener
  2023-12-02 05:46:02 a039871a-a214-4f25-837e-eae4bc2e01e6 ..... database@default Illuminate\Events\CallQueuedListener
  2023-12-02 03:03:02 fba2d8c7-bc5f-4a3d-940f-53dee99b4ebf ..... database@default Illuminate\Events\CallQueuedListener
  2023-12-02 02:57:02 1fa7b0a9-ab62-4528-8bf2-cd74b51c9514 ..... database@default Illuminate\Events\CallQueuedListener
  2023-12-02 02:39:08 b8aeee49-bae4-436e-8aef-9f3aab6a3f56 ..... database@default Illuminate\Events\CallQueuedListener
  2023-12-02 02:37:05 194078e7-887c-4f72-9f21-ac5a0a40adfd ..... database@default Illuminate\Events\CallQueuedListener
  2023-10-11 00:25:46 38f9ae89-5a20-4ad6-8c8d-855a83a10c46 ...... database@default App\Jobs\Util\ImportStripeCustomers
  • So it looks like they are very old jobs.
  • I ran php8.2 artisan queue:work and it took a while to run, but it did not seem to clear the queue or do anything to process the jobs?

  2024-09-11 14:45:04 App\Jobs\Ninja\QueueSize ............................................................... RUNNING
  2024-09-11 14:45:04 App\Jobs\Ninja\QueueSize .......................................................... 28.53ms DONE
  • I ran php8.2 artisan queue:flush to flush the old failed jobs and now it doesn’t list any failed jobs.
  • php8.2 artisan queue:work still took a good 4 minutes and 46 seconds and it did not clear any of the pending jobs (still 634!), so running it every 5 minutes seems dangerous?
  • So should both php artisan schedule:run and php artisan queue:work be run in cron (once a minute and once every 5 minutes)?
  • Got some answers, but still confused as hell!

Thanks!

4

Correct, setting QUEUE_CONNECTION to database means the database is used to queue the jobs. You can set it to sync to disable the queue.

I’m not sure you’ll want all of those jobs processed. If you can’t use the command line you may want to manually empty the jobs table (after taking a backup of the database).

You can use supervisor instead of the cron but it’s a bit more complicated to setup.

1 Like
5

I think the documentation that you linked talks about supervisor setup.
I also realized that passing --stop-when-empty makes it exit in 4-5 seconds. So I think that’s what I will use in cron.

However I am still confused about the 634 pending jobs and the Yellow triangle in the health check. Are these jobs scheduled to run in future? Any way to list them? If this is normal circumstance, then why the Yellow triangle in Health check with 634 Pending jobs?

6

My guess is you enabled the queue a while ago but didn’t realize it wasn’t working. We recently added more info to the health check which is why you’re only seeing the error now.

You would need to view the jobs table in the database to see what’s pending.

7

Checked the db tables, jobs tables does indeed have 634 rows. Here is a sample row (payload column is redacted):

| id | queue | payload | attempts | reserved_at | available _at | created_at | | 1413 | default | {"uuid":"41332c80-883d-4848-a0da-6648229d88c7", "displayName":"App\\Listeners\\Misc\\InvitationViewedListener","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null} | 1 | 1694147618 | 1694147618 | 1694147618 |

So how do I make sense of this?

8

I printed out the latest and it is timestamp 1701497398 (December 2, 2023). I guess, I am ready to purge all of these jobs and move on?

9

I think that would probably be best

10

Dang it!

That did bring up the green checkboxes on the Health check!

1 Like