502: Bad Gateway errors

Ph03n1ks · November 16, 2023, 7:58am

I’m getting lots of 502: Bad Gateway errors since the last update (v5.7.51-M125). Can’t save or view quote/invoice PDFs or save changes.

hillel · November 16, 2023, 8:00am

Hi,

Which version did you upgrade from?

@david do you have any suggestions?

Ph03n1ks · November 16, 2023, 8:02am

I’ve recreated the docker container(s) and it’s up and running again. I’ve upgraded from v5.7.50 btw.

hillel · November 16, 2023, 8:04am

Glad to hear it, thanks for the update!

Ph03n1ks · February 21, 2024, 10:47am

I’m reviving this thread as the Bad Gateway issue is back again unfortunately.

v5.8.27-M155

hillel · February 21, 2024, 10:59am

There may be more details in the web server error logs.

Ph03n1ks · February 21, 2024, 11:05am

I have tried to strip any sensitive info from the error logs. Hopefully this is helpful:

2024/02/21 10:39:59 [error] 30#30: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:39:59 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "10.6.66.107"
2024/02/21 10:39:59 [error] 30#30: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:39:59 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "10.6.66.107"
172.19.0.1 - - [21/Feb/2024:10:41:13 +0000] "GET /storage/Lifu01EDJ7Ut6TddSYHjp3UIU68at3w1/Iy8Fu9HqgDtnP8UfSfGKaue42thpneF5TND6fB3g.png HTTP/1.1" 200 17678 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
172.19.0.1 - - [21/Feb/2024:10:41:13 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:41:13 [error] 30#30: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
2024/02/21 10:41:18 [error] 29#29: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:41:18 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:41:20 [error] 30#30: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:41:20 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:41:22 [error] 30#30: *12 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:41:22 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:41:24 [error] 29#29: *14 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:41:24 +0000] "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:41:27 [error] 29#29: *16 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:41:27 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:41:30 [error] 30#30: *18 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/invoices/wMvbmqpbYA?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:41:30 +0000] "PUT /api/v1/invoices/wMvbmqpbYA?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:43:28 [error] 30#30: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:43:28 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
172.19.0.1 - - [21/Feb/2024:10:43:28 +0000] "GET /storage/Lifu01EDJ7Ut6TddSYHjp3UIU68at3w1/Iy8Fu9HqgDtnP8UfSfGKaue42thpneF5TND6fB3g.png HTTP/1.1" 200 17678 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:43:33 [error] 29#29: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "GET /client/invoice/JI23G1UgrSYbW04gWKVSApBp8RIOkhQH/download?t=1708512213557&t=1708512213557 HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:43:33 +0000] "GET /client/invoice/JI23G1UgrSYbW04gWKVSApBp8RIOkhQH/download?t=1708512213557&t=1708512213557 HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:43:41 [error] 29#29: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:43:41 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:43:46 [error] 30#30: *27 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "GET /client/quote/aLZDIvYo6W4oFVFZ4LT7IqwtZkiMtDmK/download?t=1708512225942&t=1708512225942 HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:43:46 +0000] "GET /client/quote/aLZDIvYo6W4oFVFZ4LT7IqwtZkiMtDmK/download?t=1708512225942&t=1708512225942 HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:01 [error] 29#29: *29 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:01 +0000] "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:04 [error] 29#29: *31 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:04 +0000] "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:09 [error] 30#30: *33 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:09 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:10 [error] 29#29: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:10 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:12 [error] 29#29: *37 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/invoices/yMYer2weOB?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:12 +0000] "PUT /api/v1/invoices/yMYer2weOB?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:15 [error] 30#30: *39 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:15 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:17 [error] 30#30: *41 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:17 +0000] "PUT /api/v1/quotes/JAPdR8zaGy?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:29 [error] 29#29: *43 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:29 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:46:38 [error] 30#30: *45 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "GET /client/quote/aLZDIvYo6W4oFVFZ4LT7IqwtZkiMtDmK/download?t=1708512397956&t=1708512397956 HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:46:38 +0000] "GET /client/quote/aLZDIvYo6W4oFVFZ4LT7IqwtZkiMtDmK/download?t=1708512397956&t=1708512397956 HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:48:23 [error] 30#30: *47 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:48:23 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:48:24 [error] 29#29: *49 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:48:24 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:48:24 [error] 30#30: *51 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:48:24 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:59:18 [error] 29#29: *54 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:59:18 +0000] "GET /storage/Lifu01EDJ7Ut6TddSYHjp3UIU68at3w1/Iy8Fu9HqgDtnP8UfSfGKaue42thpneF5TND6fB3g.png HTTP/1.1" 200 17678 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
172.19.0.1 - - [21/Feb/2024:10:59:19 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:59:22 [error] 30#30: *56 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "PUT /api/v1/quotes/QJ0dN7meLO?include=activities.history HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:59:22 +0000] "PUT /api/v1/quotes/QJ0dN7meLO?include=activities.history HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"
2024/02/21 10:59:25 [error] 29#29: *58 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: _, request: "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1", upstream: "fastcgi://172.19.0.3:9000", host: "invoiceninja.<<REDACTED>>"
172.19.0.1 - - [21/Feb/2024:10:59:25 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1708501126&first_load=true HTTP/1.1" 502 157 "-" "Dart/3.2 (dart:io)" "10.6.66.1"

hillel · February 21, 2024, 11:14am

Maybe this post will help:

Ph03n1ks · February 21, 2024, 7:56pm

Thanks, I haven’t found anything yet that would solve this issue. This all happened overnight without me touching the containers. I also don’t run this on a VM but on a local server (Synology). I’m pretty sure it’s an nginx issue but I don’t know if the problem is with the container in the latest release or elsewhere. I will pray the next update will fix this.

Ph03n1ks · February 21, 2024, 8:11pm

Short update: previously I used Portainer to inspect my containers but upon checking Synology’s built-in Container Manager, I can see that the invoice ninja app container is constantly stopping and restarting. I can’t seem to find anything relevant or useful in the logs though

Edit: app logs below

2024/02/22 06:13:20	stdout	2024-02-21T20:13:20Z [INFO] [Entrypoint]: Initialising Invoice Ninja...
2024/02/22 06:13:13	stdout	
2024/02/22 06:13:13	stdout	                                                                               
2024/02/22 06:13:13	stdout	  e, report to the developers                                                  
2024/02/22 06:13:13	stdout	  SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Pleas  
2024/02/22 06:13:13	stdout	                                                                               
2024/02/22 06:13:13	stdout	In Connector.php line 65:
2024/02/22 06:13:13	stdout	
2024/02/22 06:13:13	stdout	                                                                               
2024/02/22 06:13:13	stdout	  M VERSIONED') order by table_name)                                           
2024/02/22 06:13:13	stdout	  tables where table_schema = 'ninja' and table_type in ('BASE TABLE', 'SYSTE  
2024/02/22 06:13:13	stdout	  engine as `engine`, table_collation as `collation` from information_schema.  
2024/02/22 06:13:13	stdout	  name`, (data_length + index_length) as `size`, table_comment as `comment`,   
2024/02/22 06:13:13	stdout	  e, report to the developers (Connection: mysql, SQL: select table_name as `  
2024/02/22 06:13:13	stdout	  SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Pleas  
2024/02/22 06:13:13	stdout	                                                                               
2024/02/22 06:13:13	stdout	In Connection.php line 829:
2024/02/22 06:13:13	stdout	
2024/02/22 06:13:11	stdout	
2024/02/22 06:13:11	stdout	  webpatser/laravel-countries ........................................... DONE
2024/02/22 06:13:11	stdout	  turbo124/beacon ....................................................... DONE
2024/02/22 06:13:11	stdout	  spatie/php-structure-discoverer ....................................... DONE
2024/02/22 06:13:11	stdout	  spatie/laravel-data ................................................... DONE
2024/02/22 06:13:11	stdout	  socialiteproviders/manager ............................................ DONE
2024/02/22 06:13:11	stdout	  sentry/sentry-laravel ................................................. DONE
2024/02/22 06:13:11	stdout	  nwidart/laravel-modules ............................................... DONE
2024/02/22 06:13:11	stdout	  nunomaduro/termwind ................................................... DONE
2024/02/22 06:13:11	stdout	  nesbot/carbon ......................................................... DONE
2024/02/22 06:13:11	stdout	  livewire/livewire ..................................................... DONE
2024/02/22 06:13:11	stdout	  laravel/ui ............................................................ DONE
2024/02/22 06:13:11	stdout	  laravel/tinker ........................................................ DONE
2024/02/22 06:13:11	stdout	  laravel/socialite ..................................................... DONE
2024/02/22 06:13:11	stdout	  laravel/slack-notification-channel .................................... DONE
2024/02/22 06:13:11	stdout	  invoiceninja/inspector ................................................ DONE
2024/02/22 06:13:11	stdout	  intervention/image .................................................... DONE
2024/02/22 06:13:11	stdout	  imdhemy/laravel-purchases ............................................. DONE
2024/02/22 06:13:11	stdout	
2024/02/22 06:13:11	stdout	   INFO  Discovering packages.  
2024/02/22 06:13:11	stdout	
2024/02/22 06:13:10	stdout	
2024/02/22 06:13:10	stdout	  routes .......................................................... 201ms DONE
2024/02/22 06:13:10	stdout	  config ........................................................... 67ms DONE
2024/02/22 06:13:10	stdout	
2024/02/22 06:13:10	stdout	   INFO  Caching the framework bootstrap files.  
2024/02/22 06:13:10	stdout	
2024/02/22 06:13:09	stdout	
2024/02/22 06:13:09	stdout	   INFO  Configuration cached successfully.

hillel · February 21, 2024, 9:11pm

Maybe this will help:

Ph03n1ks · February 21, 2024, 9:18pm

LEGEND!! Thank you so much.

hillel · February 21, 2024, 9:19pm

Glad to hear it helped!