SMTP stopped working after updating Percona PMM to 2.38.1

SMTP was working before updating the PMM version to 2.38.1.
Now we are not able to sent mail. Tired sending test mails.

Also getting below error
logger=alerting.notifier.email t=2023-08-01T08:37:03.135794852Z level=error msg=“Missing receiver”

logger=alerting.notifier.email t=2023-08-01T08:37:03.135927417Z level=error msg=“Missing group labels”

Configuration

[smtp]
enabled = true
host = smtp.displayme.net:587
user = username@smtp.displayme.net
password = “”““password””"
from_address = domain@smtp.displayme.net
skip_verify = false
startTLS_policy = NoStartTLS

Looks like your DNS servers might be at fault. This worked for me:

Your error message says “lookup … timeout” so this is something on your side.

$ dig smtp.displayme.net +short
3.90.157.212

@matthewb I can’t run this inside PMM docker. PMM is running inside docker. I have tested in host machine. It’s working for me also

Why can’t you run this simple utility inside a docker container? Simply attach to the container and run it. I do it all the time. In any case, you need to do some networking troubleshooting from within the container on why it cannot resolve the DNS.

How did you upgrade to 2.38.1 and what version did you come from? There’s been some pretty significant changes and depending on how you upgrade you may have uncovered something we missed.

If you were previously running PMM < 2.38.0 and did a fresh docker pull to get the 2.38.1 version you not only upgraded PMM but the underlying container OS (prior to 2.38.0 the container OS was based on CentOS7 while 2.38.0 and newer is based on Oracle Linux 9) so that jump may have triggered something on your host system from a compatibility point.

Thinking along the lines of what matthewb was getting at, maybe firewalld or selinux on the host is blocking DNS lookups from the container? I don’t think we ship dig or nslookup by default in the container so you’d have to yum install which ever you prefer to test with. Have you checked inside the docker container to see what /etc/resolv.conf got set to on container init?

May I know how can I do that? I’m attaching screenshot for your reference.

We have done upgrading from home dashboard.

Ok, so based on your answers you’re still on a CentOS7 base container image so if
dig smtp.displayme.net +short works on the host OS here’s the steps for the container:
docker exec -it pmm-server-backup bash this will get you a shell inside the docker container
yum install dig will install the utility
dig smtp.displayme.net +short should give a clue if the container is able to access DNS

assuming the answer is no, take a look at dns resolution with cat /etc/resolv.conf while inside the container. That should match what you have in the hosts /etc/resolv.conf.

I’ve assumed all along that other network functions are working (i.e. clients are able to push metrics to the server, you can still access the home dashboard via a browser, etc) but just to be safe, while in the container see if you can ping the IP address of your DNS server. hope this gets a step closer to the issue.

Still not able to install.

@somename,
The fact that you cannot even run yum install only shows further evidence that your networking/DNS is messed up with that container, and/or the docker host server. You need to perform some basic networking 101 troubleshooting to determine why you cannot access the outside network from within this container. Try starting a simple container like busybox and see if you can reach outside that container. If you cannot, then you have larger docker issues.