Emails not sending - yet another thread (v5.5.55)

charles · January 16, 2023, 11:12pm

Hello,

I don’t regularly send emails directly from IN, since I prefer to download documents and email them myself to my clients. That being said, some core functions of IN rely on being able to send emails, like bulk document downloading. This is how I noticed that emails don’t work anymore on my install - I am runnig on a newer server since the last time it worked, so it may be related.

Here is the way my .env is setup, as previously working:

MAIL_MAILER=smtp
MAIL_HOST=smtp-relay.gmail.com
MAIL_PORT=587
[email protected]
MAIL_PASSWORD=password
MAIL_ENCRYPTION=tls
[email protected]
MAIL_FROM_NAME="name"

I am using Google Workspace SMTP relay with an app password created for Invoice Ninja. I updated my Google Workspace settings to allow my new server IP and I edited my DNS DMARC settings accordingly on my registrar config.

Well, the above doesn’t work anymore, so I wondered if that might be due to some update that changed the syntax in the .env file or something (like the self-host troubleshooting page says I should have a SERVER_NAME= value in .env, but I’m not share what it should be - and apparently that was not necessary for my config either, since I never used that value before. Still, I will happily add that variable if it helps, I would just need to know what to input there.

Here is the link to that page, if anyone is interested:

So, I tried running a clean install in another test subdomain I created to see what result I would get with the “Send Test email”, since there is no such test button in the main app once installed (unfortunately).

Well, it didn’t work, no matter what port I tried using. I even tried turning off two-factor authentification and turning on “less secure apps” for my Google account and using my main email password, but I still got this:

Connection could not be established with host "smtp-relay.gmail.com:587": stream_socket_client(): Unable to connect to smtp-relay.gmail.com:587 (Network is unreachable)

Apparently, this is a common issue when using Laravel to connect to Gmail (if anyone has any idea on how to fix that, I’d very happy), and the “solution” (more like a workaround) I have read the most is to switch the mail driver to Sendmail. The thing is, by doing that, my server now apparently completely bypasses Google, since I can use funky values under USERNAME and HOST and such. The following settings correctly sent me an email (I had “domain.com” switched to my real domain, though) :

Surely, sending emails directly from my server is not great for deliverability (I used to do that and I switched to Google exactly because I often ended up in junk mail despite having proper DNS records and a dedicated IP - the IP reputation is a strong value apparently for agressive spam filters). Still, I decided to rock it for a while to test it out so I went ahead and changed my .env file in my production app to “Sendmail”. According to some Youtube Video ( How to fix Connection could not be established with host smtp.gmail.com✔️ - YouTube) I also had to change SMTP to Sendmail in ./config/mail.php" line 16, so I did that:

'default' => env('MAIL_MAILER', 'sendmail'),

Now, after the obligatory php artisan optimize on my production app, I went to my test company and tried sending myself an invoice. Nothing happened. I tried downloading a zip of several documents, where IN is supposed to email me when the zipping is done: nothing happened.

To make sure my install, besides IN functions, can send emails with the current settings, I tried the following:

php artisan tinker

\Mail::raw('hello world', function($message) {
   $message->subject('Testing email')->to('[email protected]');
});

And it sent me an email, yay! So something is working somewhere.

Back to the console, I tried forcing php artisan queue:work and then… boom, it did a milllion things, including sending me several test emails that were stuck in the queue! Clearly, I have a queue issue on top of the SMTP issue. I’m running the Laravel Supervisor service on my server and I have those variables setup in my .env:

QUEUE_CONNECTION=database
INTERNAL_QUEUE_ENABLED=false

So all in all, I have two main questions:

Any idea how I can I make SMTP work with the Google SMTP Relay to fix the following error without resorting to using Sendmail:
Connection could not be established with host "smtp-relay.gmail.com:587": stream_socket_client(): Unable to connect to smtp-relay.gmail.com:587 (Network is unreachable) ?
Any idea on how I can make sure my Laravel Supervisor service is doing its job (because clearly, it isn’t and I used to have more success by running the queue:work command in a cron job on my previous server)? Here is my laravel-worker.conf, if that helps:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/charleso/public_html/admin/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=charleso
numprocs=2
redirect_stderr=true
stdout_logfile=/home/charleso/public_html/admin/laravel_worker.log
stopwaitsecs=3600

And here is what I get when asking the system for the Supervisor status:

# systemctl status supervisor
● supervisor.service - Supervisor process control system for UNIX
     Loaded: loaded (/lib/systemd/system/supervisor.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-01-15 21:59:14 EST; 20h ago
       Docs: http://supervisord.org
   Main PID: 446 (supervisord)
      Tasks: 3 (limit: 231998)
     Memory: 155.8M
     CGroup: /system.slice/supervisor.service
             ├─   446 /usr/bin/python3 /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
             ├─219909 /opt/cpanel/ea-php81/root/usr/bin/php /home/charleso/public_html/admin/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600
             └─220008 /opt/cpanel/ea-php81/root/usr/bin/php /home/charleso/public_html/admin/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600

Thanks in advance!

hillel · January 17, 2023, 6:59am

Hi,

@david do you have any suggestions?

david · January 17, 2023, 7:24am

Network is unreachable

means your system cannot even connect to the relay service.

try

telnet smtp-relay.gmail.com 587

from the command line to see whether your server can actually get an EHLO from google.

you may also need some additional configuration to make this work with laravel/symfony mailer here is a post which describe the extra details:

https://laracasts.com/discuss/channels/laravel/laravel-incorrectly-handling-ehlo-for-smtp-from-artisancli-issues-with-gmails-smtp-relay-resolved

charles · January 17, 2023, 5:25pm

david:

Network is unreachable
means your system cannot even connect to the relay service.

try
telnet smtp-relay.gmail.com 587
from the command line to see whether your server can actually get an EHLO from google.

Thanks for your reply @david.

It seems to have worked if I read this right - well, it connected, but I’ not sure about the rest:

# telnet smtp-relay.gmail.com 587
Trying 142.251.163.28...
Connected to smtp-relay.gmail.com.
Escape character is '^]'.
220 smtp-relay.gmail.com ESMTP f13-20020a056870c10d00b00143ef5156a9sm2137493oad.46 - gsmtp

I also read the thread you linked to and I added the following line to mail.php:
'local_domain' => env('MAIL_EHLO_DOMAIN',parse_url(env('APP_URL'))['host']),

By the way, according to the post, this seems an important update to do to Laravel installs to make them compliant with Google; maybe this could be added by default in a future update of Invoice Ninja?

Any idea how I can test this, now, besides trying to send an invoice or something like that? I tried running the setup again to hit the email test button, but I think the setup doesn’t read the variables already present in the .env file, so that still gave me the connection error as above.

EDIT:
Editing mail.php with the line pasted above results in emails just failing to get sent, this time , with no other clue being printed anywhere (that I know of).

As soon as I removed that line, emails work again on my test install (with internal queues and Sendmail, to make sure the rest is complying).

EDIT 2:

I fixed my Laravel Supervisor issue: I just had to remove the “sqs” argument in the command line. Apparently “sqs” is for some “sqs driver” and I’m not sure what it does, but it sure did prevent my queue from running more than a few seconds before stopping.

So now, the command line looks like that:
command=php /home/charleso/public_html/admin/artisan queue:work --sleep=3 --tries=3 --max-time=3600

BTW, my server has 3 processors, so I also edited the number of processes to 3, but is that sound? The default Laravel documentation example is showing a whopping 8 processes, but surely that’s overkill when there are not 8 cores available?

Anyway, my emails now get sent immediately with Sendmail, so there’s only the Gmail bug to try to fix now

david · January 17, 2023, 9:08pm

@charles

After you set the smtp configuration to match the laracasts posts with the additional variables you can try the following (after your optimize to update the cache)

php artisan tinker

\Mail::mailer('smtp')->raw('hello world', function($message) {
   $message->subject('Testing email')->to('[email protected]');
});

note you can swap out the mailer variable with smtp/sendmail at this point so you can continually confirm emails are sending with sendmail and/or smtp without having to update .env vars.

charles · January 18, 2023, 4:37pm

Hello @david,

Thanks for the suggestion, but no need for further troubleshooting, my problem is fixed!

I updated my EXIM config to add 142.251.163.28 (Google SMTP-Relay IP) to the trusted Trusted SMTP IP addresses and I added the same IP to my allow list on my server firewall, and then I re-added the new line in mail.php, as discussed above:
'local_domain' => env('MAIL_EHLO_DOMAIN',parse_url(env('APP_URL'))['host']),

It works!!

Thanks again for your help. That mail.php line should really be added to IN by default in a future update, I think

david · January 18, 2023, 8:57pm

@charles

What would be super cool here is if you could paste the end configuration and the other changes you had to make, and i’ll get this into the docs ASAP

charles · January 19, 2023, 3:52am

Sure, I’ll write a step by step soon

Edit @david: for now it seems like every update overwrites mail.php, so I have to reedit it manually every time to add the new line of code. Do you know how I could make the fix more permanent?

charles · January 22, 2023, 7:01pm

@david

OK, so after much testing, I had 3 main things that I had to take care of for emails to send through smtp-relay.gmail.com:

Fix the syntax in the command line in the Laravel Supervisor configuration file (laravel-worker.conf) from the Laravel docs example and remove the sqs argument after queue:work. For some reason, that argument was causing my workers to crash and not recover until a manual restart, thus failing to send my scheduled emails. My command line went from:

command=php /path/to/project/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600

to:

command=php /path/to/project/artisan queue:work --sleep=3 --tries=3 --max-time=3600

Here is the Supervisor guide:

And
Here is my whole working configuration file for reference:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/project/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=user
numprocs=4
redirect_stderr=true
stdout_logfile=/path/to/project/laravel_worker.log
stopwaitsecs=3600

Starting recently, Gmail SMTP Relay requires a proper EHLO hostname domain to be sent during the SMTP handshake: [127.0.0.1] doesn’t cut it anymore. For that, Laravel has to check for a host domain variable and send it along with the handshake request.

Since the mail.php was updated recently in Invoice Ninja with the following line:
'local_domain' => env('MAIL_EHLO_DOMAIN'), ,

it doesn’t need to be manually edited anymore to fix the failed EHLO requirement. All is now needed is a new line in the .env file: MAIL_EHLO_DOMAIN="server.domain.com". Without this, IN will report an error from Gmail that looks like this:

This is what the mail section looks like in my .env:

MAIL_MAILER="smtp"
MAIL_HOST="smtp-relay.gmail.com"
MAIL_PORT="587"
MAIL_USERNAME="[email protected]"
MAIL_PASSWORD="password"
MAIL_ENCRYPTION="tls"
MAIL_FROM_ADDRESS="[email protected]"
MAIL_FROM_NAME="Invoice Ninja"
MAIL_EHLO_DOMAIN="server.domain.com"

Last but not least, Invoice Ninja must be able to communicate with the Gmail servers, and that is not a given. If the connection attempt fails because Invoice Ninja cannot reach Gmail, it gives an error like this one:

image1297×135 9.32 KB

To make sure that doesn’t happen, for starters, IN should be able to bypass the default mail server and connect directly to SMTP to reduce the complexity of the mailing pipeline and avoid possible errors.

On my system running WHM, I have ConfigServer Security & Firewall (CSF) Installed and it has a specific feature for this under the Fix Common Problems button called SMTP_BLOCK:

A very similar setting is also available directly in WHM through Home > Security > SMTP Restrictions that prevents users from connecting to SMTP without going through the mail server:

It should be disabled.

Now, Gmail servers can still be blocked by the firewall. I could not find a list of all of the IPs used by smtp-relay.gmail.com anywhere, so I ran a bunch of manual DNS checks on several DNS checking websites, and depending on the authoritative DNS servers they were connecting to, I got different IPs, but after a few runs I exhausted all possibilities that exist at the moment (or so it seems). Here are all of the North American (USA & Canada) IPs I gathered:

142.251.163.28
172.253.63.28
172.253.118.28
74.125.24.28
142.250.114.28
142.250.4.28
142.251.12.28
172.217.194.28
142.250.31.28
142.251.167.28
172.253.115.28

I added those as allowed IPs in my firewall config.

The IPs can still change eventually, so I also configured the lfd Dynamic DNS setting in CSF (which edits /etc/csf/csf.dyndns):

There, I added smtp-relay.gmail.com and also smtp.gmail.com for good measure.

For those on a more bare bones system, I think running the following script in a cron job should work to update the iptables with the Google SMTP Relay current IP that the server is trying to connect to (I haven’t tested it, though). The script is supposed to be flexible and allow multiple IPs to be added all at once in the DNS checkup results in several IPs:

#!env /bin/bash

ip=`dig +short smtp-relay.gmail.com`

for n in $ip
do
  ipset add allowed $n
done

And finally, Gmail has to be able to accept incoming SMTP connection requests. For Google Workspace users, this is easy to setup in the Google Admin Console:

Step 1: Set up SMTP relay in your Google Admin console

Sign in to your Google Admin console.

Sign in using an administrator account.

In the Admin console, go to Menu Apps Google Workspace Gmail Routing.

You can add, edit, and delete the SMTP relay service setting at the top-level organization only. You can view the setting at the child organizational unit level.
3. Scroll to SMTP relay service and click Configure. If the setting is already configured, click Edit or Add another.
4. Enter a name for the setting and set up the following options:

Setting options What to do

Allowed senders Choose an option:

Only registered Apps users in my domains—Sender must be a Google Workspace user in one of your domains.

Only addresses in my domains—Sender doesn’t have to be a Google Workspace user, but must be a user in one of your registered domains. This option is useful when you use third-party or custom applications to send messages.

Any addresses (not recommended)—Sender address can be any email address, including addresses outside of your domain. This option makes you more vulnerable to abuse, either by malicious software on your users’ devices or by incorrect SMTP settings.If you use the any address option and send messages from a domain that you don’t own or with an empty envelope-from (for example, bounce messages or out-of-office notifications), set up your mail server to use SMTP AUTH to identify the sending domain or to present one of your domain names in the HELO or EHLO command.If the sender is not in one of your domains, the system changes the envelope sender from user@domain_you_don’t_own to postmaster@your_domain, where your_domain is the domain the system receives from SMTP AUTH or from the HELO or EHLO command.|
|Authentication|Check one or both boxes to set an authentication method:

Only accept mail from the specified IP addresses—System accepts only messages sent from IP addresses that you specify.

Require SMTP Authentication—Enforces SMTP authentication to identify the sending domain (connection through TLS required). SMTP authentication verifies the connection by checking the user Google Workspace email address and password.

If you select the specified IP addresses option:

Click Add.

Enter a description and the IP address or range in IPv4 or IPv6 format.
Use your own public IP address. You can specify up to 65,536 IP addresses in one range. For security reasons, we recommend that you keep the IP range as small as possible.

Check or uncheck the Enable box to enable or disable the IP address or range.

Click Save.

To add more IP addresses or ranges, repeat the steps.|
|Encryption|(Optional) To require TLS for connections between your server and Google, check the Require TLS encryption box.

5.Important:* If your email server doesn’t support TLS and you check this box, messages not sent over an encrypted TLS connection are rejected.|

Click Save.

Changes can take up to 24 hours but typically happen more quickly. Learn more

Source: Route outgoing SMTP relay messages through Google - Google Workspace Admin Help

My recommended settings here are:

Only addresses in my domains
Only accept mail from the specified IP addresses > server IP
Require SMTP Authentication > for that I use an app password instead of using the main Google Workspace account password. More info here: Sign in with app passwords - Google Account Help
Require TLS encryption

Source: Route outgoing SMTP relay messages through Google - Google Workspace Admin Help

Then, of course, we have to make sure that all records (MX, DKIM, SPF, DMARC) are properly setup at the DNS host and that the Reverse PTR record is properly pointing to the server IP.

That’s it! Emails should now be properly routed through the Gmail SMTP Relay servers

david · January 22, 2023, 9:04pm

@charles

thanks for this! i am sure this will help the next person who hits this issue!