Stuck pending jobs in System Health with Laravel workers

charles · November 10, 2024, 5:30pm

Version <v5.10.44-W171>

Hi,

I had my VPS ported from Ubuntu 20.04 to a new instance running Ubuntu 22 and now I’m having a few issues with the queue : when checking System Health, there always seems to be one or several pending jobs in the queue. However, my database queue seems to be working OK with Laravel workers, so I don’t know what might be going wrong.

First, let’s start clean:

Everything looks good for now:

Just checking that Supervisor is working fine, and that seems to be the case (I just redid a clean install of Supervisor just to be sure):

Now here is the Supervisor configuration ported from my old server, everything seems OK as far as I’m concerned:

The Laravel worker log feel does get populated as expected:

2024-11-10 17:10:02 App\Jobs\Ninja\QueueSize ....................... RUNNING
2024-11-10 17:10:02 App\Jobs\Ninja\QueueSize ................... 2.80ms DONE
2024-11-10 17:10:02 App\Jobs\Ninja\TaskScheduler ................... RUNNING
2024-11-10 17:10:02 App\Jobs\Ninja\TaskScheduler .............. 18.31ms DONE
2024-11-10 17:15:03 App\Jobs\Ninja\QueueSize ....................... RUNNING
2024-11-10 17:15:03 App\Jobs\Ninja\QueueSize ................... 3.91ms DONE
2024-11-10 17:20:03 App\Jobs\Ninja\QueueSize ....................... RUNNING
2024-11-10 17:20:03 App\Jobs\Ninja\QueueSize ................... 3.92ms DONE

I also have a cron running this every minute from the invoice ninja installation directory:
php artisan schedule:run

Now let’s try downloading some invoices, the logs get updated properly:

2024-11-10 17:22:32 App\Jobs\Invoice\ZipInvoices ................... RUNNING
2024-11-10 17:22:39 App\Jobs\Invoice\ZipInvoices .................. 6 s DONE
2024-11-10 17:22:39 App\Jobs\Mail\NinjaMailerJob ................... RUNNING
2024-11-10 17:22:40 App\Jobs\Mail\NinjaMailerJob .................. 1 s DONE
2024-11-10 17:22:40 App\Listeners\Mail\MailSentListener ............ RUNNING
2024-11-10 17:22:40 App\Listeners\Mail\MailSentListener ....... 13.22ms DONE

I get the confirmation email, the button in it works and it properly downloads a zip file with the requested invoices:

But now if I go back to the System Health, I see a pending job:

Am I missing something in my config? Could this be a false positive and the job did complete successfully, or could some process be stuck somewhere? I’m really scratching my head with this one.

Thanks in advance!

hillel · November 10, 2024, 7:16pm

Hi,

I’m not sure. You can check the jobs table in the database for more info, the app is just checking the number of rows in the table.

charles · November 12, 2024, 5:14am

OK thanks, it looks like it does get cleared out after a while because when I look from time to time the pending jobs are gone and the jobs table in the database is empty.

Some jobs do fail however, and this is also new since my server swap. Here is one out of 4 currently (they all look quite similar), any idea what might be going wrong?

> PDOException: SQLSTATE[HY000] [1045] Access denied for user 'forge'@'localhost' (using password: NO) in /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:66
> Stack trace:
> #0 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php(66): PDO->__construct('mysql:host=loca...', 'forge', Object(SensitiveParameterValue), Array)
> #1 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php(44): Illuminate\Database\Connectors\Connector->createPdoConnection('mysql:host=loca...', 'forge', '', Array)
> #2 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/MySqlConnector.php(24): Illuminate\Database\Connectors\Connector->createConnection('mysql:host=loca...', Array, Array)
> #3 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/ConnectionFactory.php(185): Illuminate\Database\Connectors\MySqlConnector->connect(Array)
> #4 [internal function]: Illuminate\Database\Connectors\ConnectionFactory->Illuminate\Database\Connectors\{closure}()
> #5 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(1231): call_user_func(Object(Closure))
> #6 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(1267): Illuminate\Database\Connection->getPdo()
> #7 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(512): Illuminate\Database\Connection->getReadPdo()
> #8 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(469): Illuminate\Database\Connection->getPdoForSelect(true)
> #9 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(812): Illuminate\Database\Connection->Illuminate\Database\{closure}('select * from `...', Array)
> #10 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(779): Illuminate\Database\Connection->runQueryCallback('select * from `...', Array, Object(Closure))
> #11 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(461): Illuminate\Database\Connection->run('select * from `...', Array, Object(Closure))
> #12 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3352): Illuminate\Database\Connection->cursor('select * from `...', Array, true)
> #13 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(800): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}()
> #14 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(445): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #15 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(800): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #16 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(445): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #17 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php(256): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #18 /home/charleso/public_html/admin/app/PaymentDrivers/Rotessa/Jobs/TransactionReport.php(55): Illuminate\Support\LazyCollection->each(Object(Closure))
> #19 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\PaymentDrivers\Rotessa\Jobs\TransactionReport->handle()
> #20 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
> #21 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
> #22 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
> #23 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Container.php(690): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
> #24 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call(Array)
> #25 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #26 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #27 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then(Object(Closure))
> #28 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(124): Illuminate\Bus\Dispatcher->dispatchNow(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport), false)
> #29 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #30 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #31 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\Pipeline\Pipeline->then(Object(Closure))
> #32 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(71): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\Queue\Jobs\DatabaseJob), Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #33 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\DatabaseJob), Array)
> #34 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(440): Illuminate\Queue\Jobs\Job->fire()
> #35 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(390): Illuminate\Queue\Worker->process('database', Object(Illuminate\Queue\Jobs\DatabaseJob), Object(Illuminate\Queue\WorkerOptions))
> #36 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(177): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\DatabaseJob), 'database', Object(Illuminate\Queue\WorkerOptions))
> #37 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(148): Illuminate\Queue\Worker->daemon('database', 'default', Object(Illuminate\Queue\WorkerOptions))
> #38 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(131): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'default')
> #39 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle()
> #40 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
> #41 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
> #42 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
> #43 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Container.php(690): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
> #44 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\Container\Container->call(Array)
> #45 /home/charleso/public_html/admin/vendor/symfony/console/Command/Command.php(279): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
> #46 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
> #47 /home/charleso/public_html/admin/vendor/symfony/console/Application.php(1047): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #48 /home/charleso/public_html/admin/vendor/symfony/console/Application.php(316): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #49 /home/charleso/public_html/admin/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #50 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(197): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #51 /home/charleso/public_html/admin/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #52 {main}
> 
> Next Illuminate\Database\QueryException: SQLSTATE[HY000] [1045] Access denied for user 'forge'@'localhost' (using password: NO) (Connection: db-ninja-02, SQL: select * from `company_gateways` where `gateway_key` = 91be24c7b792230bced33e930ac61676 and `company_gateways`.`deleted_at` is null) in /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php:825
> Stack trace:
> #0 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(779): Illuminate\Database\Connection->runQueryCallback('select * from `...', Array, Object(Closure))
> #1 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connection.php(461): Illuminate\Database\Connection->run('select * from `...', Array, Object(Closure))
> #2 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3352): Illuminate\Database\Connection->cursor('select * from `...', Array, true)
> #3 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(800): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}()
> #4 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(445): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #5 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(800): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #6 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/LazyCollection.php(445): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #7 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php(256): Illuminate\Support\LazyCollection->Illuminate\Support\{closure}()
> #8 /home/charleso/public_html/admin/app/PaymentDrivers/Rotessa/Jobs/TransactionReport.php(55): Illuminate\Support\LazyCollection->each(Object(Closure))
> #9 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\PaymentDrivers\Rotessa\Jobs\TransactionReport->handle()
> #10 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
> #11 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
> #12 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
> #13 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Container.php(690): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
> #14 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call(Array)
> #15 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #16 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #17 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then(Object(Closure))
> #18 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(124): Illuminate\Bus\Dispatcher->dispatchNow(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport), false)
> #19 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #20 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #21 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\Pipeline\Pipeline->then(Object(Closure))
> #22 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(71): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\Queue\Jobs\DatabaseJob), Object(App\PaymentDrivers\Rotessa\Jobs\TransactionReport))
> #23 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\DatabaseJob), Array)
> #24 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(440): Illuminate\Queue\Jobs\Job->fire()
> #25 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(390): Illuminate\Queue\Worker->process('database', Object(Illuminate\Queue\Jobs\DatabaseJob), Object(Illuminate\Queue\WorkerOptions))
> #26 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(177): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\DatabaseJob), 'database', Object(Illuminate\Queue\WorkerOptions))
> #27 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(148): Illuminate\Queue\Worker->daemon('database', 'default', Object(Illuminate\Queue\WorkerOptions))
> #28 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(131): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'default')
> #29 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle()
> #30 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
> #31 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
> #32 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
> #33 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Container/Container.php(690): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
> #34 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\Container\Container->call(Array)
> #35 /home/charleso/public_html/admin/vendor/symfony/console/Command/Command.php(279): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
> #36 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
> #37 /home/charleso/public_html/admin/vendor/symfony/console/Application.php(1047): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #38 /home/charleso/public_html/admin/vendor/symfony/console/Application.php(316): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #39 /home/charleso/public_html/admin/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #40 /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(197): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #41 /home/charleso/public_html/admin/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #42 {main}

hillel · November 12, 2024, 5:38am

Are the database user and name correct: forge and db-ninja-02?

charles · November 14, 2024, 4:14pm

No, I think “forge” is the default Laravel username and “db-ninja-02”, I have no idea where that comes from.

Obviously my database can be successfully read and written to since the rest of the app works fine (or so it seems), so where could those credientals come from? Not from my env file, that’s for sure.

hillel · November 14, 2024, 4:29pm

Is it possible these are old errors from when you first setup the app?

You can run php artisan queue:clear to clear failed jobs

charles · November 14, 2024, 6:41pm

I’m pretty sure those errors are new-ish, unless somehow the app fetched them after I ran artisan queue: clear previously. I’ll try that once more and I’ll report back if I have other failed tasks! Thanks

Edit: I had to actually run php artisan queue:flush to clear failed jobs, so I did that.

charles · November 19, 2024, 6:07pm

Hi,

I have 10 new failed jobs now over the last few days (after starting with a clean slate).

All of them have in common that something, somewhere, tries to connect to the database (I think) using default credentials, eg.:

PDOException: SQLSTATE[HY000] [1045] Access denied for user 'forge'@'localhost' (using password: NO) in /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:66

So it’s definitely not old jobs, I can confirm that.

hillel · November 19, 2024, 8:26pm

I suggest checking the jobs table to find more information about the failed jobs

charles · November 23, 2024, 6:16pm

There is nothing in the jobs table for now, as I have six workers who quickly go through it as soon as there is a new job.

I checked the failed jobs table however and I now have 18 entries over the last few days and they all share the same exception, which is starting with this:

PDOException: SQLSTATE[HY000] [1045] Access denied for user 'forge'@'localhost' (using password: NO) in /home/charleso/public_html/admin/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:66

I decided to go have a look at the line 66 of Connector.php and this is it:

? new PDO($dsn, $username, $password, $options)

Looks like $username is fetching the default Laravel username forge for some reason and then it won’t work. Any idea what can be causing this? Apparently this happens when DB_username is not properly set in the .env file, but mine has the correct credentials, as the jobs are working properly most of the time and there are no other issues that I’m aware of with the database, the app connects to it, reads from it and writes to it as intended.

In any case, I have my .env set with DB_DATABASE1, DB_USERNAME1 DB_PASSWORD1 variables (I don’t remember why I set it up this way, it’s been a few years), so I’ll try removing the 1 suffix and see if it helps.

hillel · November 23, 2024, 7:08pm

You may need to restart the queue after updating the .env file