Unusual errors/pdf preview stops working/unable to download invoice

Self-Hosted | v5
1

Hi Folks,

Just came back from a week holiday, and following up on invoices for couple clients.
All of a sudden getting different error messages, and certain functions stop working randomly.

System:
IN V5.5.83-W109 (running in Windows 10 Docker Desktop 4.17.0 (99724) )
WSL 2 = Ubuntu 20.04 LTS /Focal
Host Name = invoice.localhost.test
IP: 192.168.50.174

Error log from invoiceninja_prod-app-1

023-03-09 08:21:31 172.20.0.4 -  09/Mar/2023:00:21:31 +0000 "GET /index.php" 302
2023-03-09 08:21:33 172.20.0.4 -  09/Mar/2023:00:21:31 +0000 "GET /index.php" 200
2023-03-09 08:21:33 2023-03-09 00:21:33,892 INFO reaped unknown pid 132 (exit status 0)
2023-03-09 08:21:33 2023-03-09 00:21:33,892 INFO reaped unknown pid 133 (exit status 0)
2023-03-09 08:21:45 2023-03-09 00:21:45,041 INFO reaped unknown pid 179 (exit status 0)
2023-03-09 08:21:45 2023-03-09 00:21:45,041 INFO reaped unknown pid 180 (exit status 0)
2023-03-09 08:21:47 2023-03-09 00:21:47,079 INFO reaped unknown pid 225 (exit status 0)
2023-03-09 08:21:47 2023-03-09 00:21:47,079 INFO reaped unknown pid 226 (exit status 0)
2023-03-09 08:21:47 172.20.0.4 -  09/Mar/2023:00:21:42 +0000 "PUT /index.php" 200
2023-03-09 08:21:47 172.20.0.4 -  09/Mar/2023:00:21:47 +0000 "POST /index.php" 200
2023-03-09 08:21:50 172.20.0.4 -  09/Mar/2023:00:21:48 +0000 "POST /index.php" 200
2023-03-09 08:21:50 2023-03-09 00:21:50,777 INFO reaped unknown pid 272 (exit status 0)
2023-03-09 08:21:50 2023-03-09 00:21:50,777 INFO reaped unknown pid 273 (exit status 0)

504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->

504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>

Error log from invoiceninja_prod-server-1 nginx

2023-03-09 08:20:55 2023/03/09 00:20:55 [notice] 1#1: start worker process 44
2023-03-09 08:26:34 2023/03/09 00:26:34 [error] 42#42: *29 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.20.0.1, server: _, request: "POST /api/v1/live_preview?entity=invoice&entity_id=k8mepp6eMy HTTP/1.1", upstream: "fastcgi://172.20.0.3:9000", host: "invoice.localhost.test"

Now, a temporary workaround, has been to ping the hostname both in ubuntu server, and then ping the hostname in the nginx server but after i perform the following:

apt-get update
apt-get install iputils-ping

is there anyway that NGINX server can do a DNS lookup? it is odd because it works some times, then other times it doesn’t and reverts to some ‘default’ ip that has never been used.

Thank you.
Kind regards

2

@dingobongo

Is the IP statically configured?

in your /etc/hosts file have you defined the host / ip ?

Given it is intermittent, it does sound like an internal DNS lookup issue.

3

Hi @david,

Yes it is defined both in the host file app-1 server (invoiceninja/invoiceninja:5)
and windows (host machine) host file.

I just found it was missing from the nginx host file (/etc/hosts)
I added an entry there. (although after reboot it is no longer there and problem reoccurs)

It is odd that the problem has recently started to occur, when it has been in operation for 12+ months.

Is there a way to insure that entry is in the NGINX host file by way of the docker-compose.yml file?

Anyway, thank you for the reminder about host file on the nginx server. Have to find a solution that creates a entry in host file for the NGINX server.

Kind regards,

4

Think I have found a way to auto insert entries into host file via docker-compose

extra_hosts:
  - "yourdomain.com:127.0.0.1"

Will report back

5

Update:

So I have added entry to extra hosts in the docker-compose.yml.

However, upon reboot of servers, and launching the IN windows App error in log occurs

2023-03-09 15:13:28 invoiceninja_prod-server-1  | 2023/03/09 07:13:28 [error] 31#31: *4 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.21.0.1, server: _, request: "POST /api/v1/live_preview?entity=invoice&entity_id=O5xe7xja7r HTTP/1.1", upstream: "fastcgi://172.21.0.3:9000", host: "invoice.localhost.test"
2023-03-09 15:13:28 invoiceninja_prod-server-1  | 172.21.0.1 - - [09/Mar/2023:07:13:28 +0000] "POST /api/v1/live_preview?entity=invoice&entity_id=O5xe7xja7r HTTP/1.1" 504 167 "-" "Dart/2.19 (dart:io)" "-"
2023-03-09 15:13:29 invoiceninja_prod-app-1     | 172.21.0.4 -  09/Mar/2023:07:12:28 +0000 "POST /index.php" 500

I then select a couple of invoice entries and within a minute I am able to preview the invoice.

However, I then wait several minutes and try again and error reoccurs.

I then, via docker windows, access terminal of the nginx server and attempt to ping server name (invoice.localhost.test) and error in console “ping not found”.
So, I have to then perform apt update and then apt install utilis-ping, then ping the server, and it is working again.

Any ideas?

Thank you.

6

Update:

Still not working 100%. Very intermittent when it works.

Randomly get following error in Brave browser:

504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->

nginx log

2023-03-10 15:23:09 172.22.0.1 - - [10/Mar/2023:07:23:09 +0000] "PUT /api/v1/invoices/oBDbDDnbl2?include=activities.history HTTP/1.1" 504 569 "http://invoice.localhost.test/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" "-"
2023-03-10 15:09:51 2023/03/10 07:09:51 [warn] 30#30: *71 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/1/00/0000000001 while reading upstream, client: 172.22.0.1, server: invoice.localhost.test, request: "POST /api/v1/refresh?&first_load=true&include_static=true HTTP/1.1", upstream: "fastcgi://172.22.0.3:9000", host: "invoice.localhost.test", referrer: "http://invoice.localhost.test/"
2023-03-10 15:14:59 2023/03/10 07:14:59 [error] 30#30: *76 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.22.0.1, server: invoice.localhost.test, request: "POST /api/v1/invoices/bulk?per_page=100 HTTP/1.1", upstream: "fastcgi://172.22.0.3:9000", host: "invoice.localhost.test", referrer: "http://invoice.localhost.test/"
2023-03-10 15:20:59 2023/03/10 07:20:59 [error] 30#30: *82 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.22.0.1, server: invoice.localhost.test, request: "POST /api/v1/live_preview?entity=invoice HTTP/1.1", upstream: "fastcgi://172.22.0.3:9000", host: "invoice.localhost.test", referrer: "http://invoice.localhost.test/"
2023-03-10 15:22:17 2023/03/10 07:22:17 [error] 30#30: *82 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.22.0.1, server: invoice.localhost.test, request: "POST /api/v1/live_preview?entity=invoice HTTP/1.1", upstream: "fastcgi://172.22.0.3:9000", host: "invoice.localhost.test", referrer: "http://invoice.localhost.test/"
2023-03-10 15:22:52 2023/03/10 07:22:52 [error] 30#30: *90 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.22.0.1, server: invoice.localhost.test, request: "POST /api/v1/live_preview?entity=invoice&entity_id=xYRdGJ0bDz HTTP/1.1", upstream: "fastcgi://172.22.0.3:9000", host: "invoice.localhost.test", referrer: "http://invoice.localhost.test/"
2023-03-10 15:23:09 2023/03/10 07:23:09 [error] 30#30: *97 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.22.0.1, server: invoice.localhost.test, request: "PUT /api/v1/invoices/oBDbDDnbl2?include=activities.history HTTP/1.1", upstream: "fastcgi://172.22.0.3:9000", host: "invoice.localhost.test", referrer: "http://invoice.localhost.test/"