Clarification on Queue (Supervisor) Service Setup

One of the last things remaining for my v5 installation is to get the Laravel queue working. I’ve been reading the documentation here, but there’s not much information. The documentation says:

If you have root access to your system, then simply follow the Laravel guide to configure the supervisor service to start and restart your queue.

Ok, got it, but this is where the instructions on the Laravel page are just generic and don’t specifically tell you what you need to do with InvoiceNinja v5. When I go to the Supervisor configuration page, I see I can install Supervisor with sudo apt-get install supervisor, and then it looks like I need to create a laravel-worker.conf file inside of /etc/supervisor/conf.d, and they give an example of what the conf file might look like. But unless I missed it, there’s no documentation on the InvoiceNinja page about how to actually configure Supervisor specifically for InvoiceNinja. It just says the next thing I need to do is change the .env file to read QUEUE_CONNECTION=database.

Could this page be improved, or am I missing something obvious? InvoiceNinja v5 seems to be running fine without it, but it’s the one thing left in the Health Check screen that’s not showing a green check mark, so naturally I want to make it a green check mark.

Also, will this supervisor/queue configuration still matter once the React app is the main web UI? If not, I’ll just skip this and use InvoiceNinja v5 the way it is until then so it’s not something I need to decommission later.

Hi,

A simpler option may be to use the queue:work cron instead instead.

The queue is a backend service, it’s used by both the Flutter and React apps.

A simpler option may be to use the queue:work cron instead instead.

Thanks @hillel . Ideally, I’d like to do whichever way is the most proper method, seeing as I have full access to this server. Will this method also satisfy the Health Check “Queue Not Enabled” status?

The queue is a backend service, it’s used by both the Flutter and React apps.

Got it, that’s what I figured, but wanted to double check. Thanks!

It should, @david can you advise on the differences?

Best practice here would be to setup supervisor to monitor your queue’s

The queue:work that we include in the cron, does “work” but the supervisor service would be the best choice if full server access is available.

1 Like

Excellent, thanks @david ! So, back to my original question above, what exactly needs to be configured? The InvoiceNinja documentation seems to be lacking on specific information related to how InvoiceNinja is supposed to be configured with supervisor.

@link470

Supervisor configuration can be found here:

We’ll update the docs.

Thanks @david , maybe I’m not being clear about my question.

There’s already a link to that exact page on the InvoiceNinja documentation. My concern is that the InvoiceNinja documentation just says:

If you have root access to your system, then simply follow the Laravel guide to configure the supervisor service

followed by

You will then need to update the QUEUE_CONNECTION variable in the .env file

What the InvoiceNinja documentation doesn’t appear to include is how the Laravel supervisor is actually supposed to be configured. Are you saying that if I simply follow the guide here and do exactly as it says, complete with the below config file they mention exactly as written below:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=forge
numprocs=8
redirect_stderr=true
stdout_logfile=/home/forge/app.com/worker.log
stopwaitsecs=3600

that I’ll have a working setup for InvoiceNinja? If not, then something needs to be updated on the documentation to explain how to set up supervisor. For example, my guess is that at least the command directive above needs to be changed, but to what exactly? Does anything else need to be modified? This should be in the InvoiceNinja documentation I think.

The sample configuration is a template so the only required modifications would be

command=php /home/forge/app.com/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600

changes to the path to your invoice ninja installation

user=forge

To the web user ? www-data

stdout_logfile=/home/forge/app.com/worker.log

optionally you can remove this if you don’t want to log this output

Thanks @david . Finally getting around to trying this on my InvoiceNinja 5.5.102 installation. To be totally clear, here’s what I’ve done:

•Installed Supervisor:
sudo apt-get install supervisor

•Created a configuration file for InvoiceNinja:

cd /etc/supervisor/conf.d
sudo vi my-laravel-worker.conf

•Used the example config file I posted from my last post, but changed the following 3 lines as you suggested:

command=php /path/to/invoiceninja/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600
user=<WebServerUser>
stdout_logfile=/log/path/my-laravel-worker.log

•Created the log file

sudo touch /log/path/my-laravel-worker.log
sudo chown <WebServerUser>:<WebServerUser> /log/path/my-laravel-worker.log

•Now that Supervisor was set up, ran:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start my-laravel-worker:*

Next, I went ahead and changed the Invoice Ninja .env file and edited the line for QUEUE_CONNECTION to read:
QUEUE_CONNECTION=database

I then restarted PHP FPM and Apache. I refreshed Invoice Ninja in my browser and it hung for a very long time, and during this time, server resources completely skyrocketed and maxed out memory usage and the system became unresponsive. After a few minutes, I gained control back and Invoice Ninja finished loading in the web browser.

If I run sudo supervisorctl status I see 8 workers with a status of RUNNING (numbers 00 through 07). If I check the log file I created, I see this over and over (which also seems to be the exact issue reported here):

In Connection.php line 760:

  SQLSTATE[HY000] [2002] No such file or directory (SQL: select * from inform
  ation_schema.tables where table_schema = <InvoiceNinjaDatabase> and table_nam
  e = accounts and table_type = 'BASE TABLE')


In Connector.php line 70:

  SQLSTATE[HY000] [2002] No such file or directory

Lastly, if I run a health check on Invoice Ninja, it still shows Queue: Sync.

What am I missing?

It sounds the config has not been updated, i would attempt:

php artisan optimize

then restart your queues with

php artisan queue:restart

Thanks @david , that appears to have done it!

One last question about this. My installation is running 5.5.102, and I see on Github the latest version is 5.5.103. My installation isn’t offering me 5.5.103 yet as an update in the bottom left corner. This made me think there was something else wrong with a cron job or queue or something, but everything is showing as OK.

Does it take awhile for a new version to become available via the auto update feature, or should I see 5.5.103 available now?

@link470

If you inspect the accounts table of your database, can you advise the value of the latest_version column?

@david - latest_version in the accounts table is 5.5.102

Hi @david , I see that 5.5.104 is now out. My installation still doesn’t show anything new. I still just see a contact envelope in the bottom right where I’d usually expect to see the blue triangle showing a new version is available. If I click the About button I’m running v5.5.102-C113, and if I click the Health Check button, I’ve got 3 green check marks.

If I examine the database again today, the accounts table still shows a value of 5.5.102 for the latest_version column.

Are you sure your crons are running? The daily version check should pick this up.

Also, ensure your crons are set to run every minute

Ahh, thanks @david ; double checked the cron path and it was still set to my original path (I the application path after I made v5 the production app after migrating from v4). I’ve edited the cron to the correct path, and after a minute of saving, the server ground to a halt for roughly 5 minutes or so. I’m guessing it was catching up on tasks? It’s been running fine for a few minutes now, so I’m assuming maybe it was busy running lots of queued tasks or something.

However, even after leaving it for a few more minutes and refreshing Invoice Ninja in the web, I’m still seeing no new version being offered, and 5.5.102 is still the latest version showing in the accounts database table.

Anything else I should be checking?

EDIT

I changed the cron to output to a file, and here’s the last 3 minutes output:

  2023-04-16 00:03:02 Running ['artisan' queue:work database --stop-when-empty --memory=256]  3,664ms DONE
  ⇂ '/usr/bin/php8.1' 'artisan' queue:work database --stop-when-empty --memory=256 > '/dev/null' 2>&1


  2023-04-16 00:04:02 Running ['artisan' queue:work database --stop-when-empty --memory=256]  3,652ms DONE
  ⇂ '/usr/bin/php8.1' 'artisan' queue:work database --stop-when-empty --memory=256 > '/dev/null' 2>&1


  2023-04-16 00:05:02 Running [queue-size-job] ..................... 33ms DONE
  2023-04-16 00:05:02 Running [Callback] ........................... 17ms DONE
  2023-04-16 00:05:02 Running ['artisan' queue:work database --stop-when-empty --memory=256]  3,708ms DONE
  ⇂ '/usr/bin/php8.1' 'artisan' queue:work database --stop-when-empty --memory=256 > '/dev/null' 2>&1
  2023-04-16 00:05:06 Running ['artisan' queue:restart] ........... 734ms DONE
  ⇂ '/usr/bin/php8.1' 'artisan' queue:restart > '/dev/null' 2>&1

EDIT 2

Ah, just refreshed again after seeing that extra long entry there after the third minute, and sure enough, it’s now showing 5.5.104 available. Looks like I just needed to give it some time.

Does it not actually check for updates every minute, but instead, every, say, 10 or 15 minutes? Was this delay expected, as well as that extra long run the first time where the system load became very high for a few minutes?

We only run the version check daily… so sometimes - depending on when we release - it could take up to 48 hours for the update to appear. You can always use the Force Update option if you know that a release is available.

Over on this post, I was having issues with excessive logs being stored and was wondering what was going on. The problem was that in the Supervisor documentation from Laravel, they include an example command. They also leave in the text sqs in that example command, which I, not having a clue how Supervisor worked, thought needed to be there. Turns out, it doesn’t.

I really think the Invoice Ninja Supervisor documentation needs to be updated since clearly these people all made the same mistake.


Invoice Ninja’s documentation page has this:

If you have root access to your system, then simply follow the Laravel guide to configure the supervisor service to start and restart your queue.

You will then need to update the QUEUE_CONNECTION variable in the .env file as follows:

QUEUE_CONNECTION=database

While technically true, it’s very vague, and this left a lot of questions which I was able to get help with answering in this topic with the help of @david. I thought I’d share the bit from my own documentation on just the Supervisor setup. I hope it helps someone. Sorry about the random syntax highlighting in the commands, I have no idea what Discourse is doing.

Setting Up Supervisor for Invoice Ninja v5
Written for Ubuntu 22.04 LTS

First, we need to install the Supervisor application:

$ sudo apt-get install supervisor

Next, create a configuration file for Invoice Ninja by navigating to the Supervisor configuration directory and starting up your text editor of choice with the file name to use. You can set any file name here ending in .conf. I’m using vi.

$ cd /etc/supervisor/conf.d
$ sudo vi invoiceninja-worker.conf

In the new invoiceninja-worker.conf file, enter the following:.

[program:invoiceninja-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/invoiceninja/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/invoiceninja-worker.log
stopwaitsecs=3600

There’s 4 main things to change, shown in bold above:

  • program: The program name you’ll use for the worker
  • command: The path to the artisan file in the root of the Invoice Ninja application directory
  • user: The user that’ll be running Supervisor. Typically, this is the web server user (the same user that has ownership over the Invoice Ninja files)
  • stdout_logfile: Optionally, include the path to a log file for the Supervisor worker

Once these changes are made to suit your environment, save the file.

If you chose to set a log file path, ensure the log file is created and that the user account configured has access to it. For this example, the log is stored in /var/log.

$ cd /var/log
$ sudo touch invoiceninja-worker.log
$ sudo chown www-data:www-data invoiceninja-worker.log

Now that Supervisor is set up, we can read the configuration file:

$ sudo supervisorctl reread

This should tell us that the name of our config (invoiceninja-worker) is available. The above command will also need to be re-ran any time that the config file is updated.

Next, add the new config to the process group for Supervisor (this command also needs to be ran any time the configuration is updated)

$ sudo supervisorctl update

Now, start the Supervisor worker

$ sudo supervisorctl start invoiceninja-worker:*

Running the following will show worker processes as RUNNING (on my installation, I see 8 worker processes, numbered 00 through 07)

$ sudo supervisorctl status

Now that the Supervisor is configured, we need to tell Invoice Ninja to actually use it.

Edit the .env file for Invoice Ninja

$ sudo vi /path/to/invoiceninja/.env

Edit the following line to change it from sync to database, and save the file

QUEUE_CONNECTION=database

Finally, now that everything is set up, reload the config for Invoice Ninja and restart the queue. I’m using www-data for this example as the web server user.

$ cd /path/to/invoiceninja/
$ sudo -u www-data php artisan optimize
$ sudo -u www-data php artisan queue:restart

That’s it! You should now have a functioning Supervisor setup for Invoice Ninja.

1 Like

@link470

Thanks, i’ve updated the docs also with your guide :slight_smile: