I’m trying to setup just a basic test of pmm using the provided docker directions. I see that alert contact points has a grafana-default-email which is not configured. I have tried the example commands to start pmm:
docker run -d
–publish 443:443
-v /apps/pmm:/srv
-e GF_SMTP_ENABLED=true
-e GF_SMTP_HOST=mailrelay:25
-e GF_SMTP_SKIP_VERIFY=true
-e GF_SMTP_FROM_ADDRESS=noreply@test.com
-e GF_SMTP_FROM_NAME=Grafana
–name pmm-server
percona/pmm-server:2.42.0
Which does not seem to configure email, in the /etc/grafana/grafana.ini there is no smtp section and modifying /usr/share/grafana/conf/defaults.ini does not work either it’s just over written.
The examples found on this site for doing the config of email via the website does not work either as the communications section is missing under settings.
Looking for some assistance in how to properly setup smtp.
Thanks.
Hello @Lobo_Lobo20919
Have you tried createing .env file as described here ?
About Percona Alerting - Percona Monitoring and Management
Please follow all steps, including the " Configure an Email contact point" steps, and try sending test email. If it fails, share the error with us.
Here is my .env:
DISABLE_UPDATES=true
DATA_RETENTION=336h
ENABLE_DBAAS=false
PMM_PUBLIC_ADDRESS=10.x.x.x
GF_SMTP_ENABLED=true
GF_SMTP_HOST=mailrelay:25
GF_SMTP_USER=
GF_SMTP_PASSWORD=
GF_SMTP_SKIP_VERIFY=true
GF_SMTP_FROM_ADDRESS=pmm-noreply@xx.com
GF_SMTP_FROM_NAME=Percona Alerting
Our internal relay does not use authentication so I have tried blank fields and just removing GF_SMTP_USER and GF_SMTP_PASSWORD.
Started percona pmm with:
docker run --env-file=.env -p 443:443 -p 80:80 percona/pmm-server:2
The contact point grafana-default-email was updated to have a valid email address in Addresses, all other fields were left as is.
The error from a test message is
Thanks for your assistance
When you open a shell inside the container and run env | grep "GF_" do you see all your values?
I have mine configured exactly like yours and it works just fine. There should be errors in /srv/logs/grafana.log if the SMTP server is rejecting the email. the screenshot feels like a generic error from grafana vs not actually setup. With gmail I had to make sure that the from address and user matched or gmail would silently blackhole the message so as not to give away info on what emails do/don’t exist…maybe your SMTP server does the same?
You can do an SMTP test via telnet pretty easy to make sure the server will accpt mail from your host…worked at many orgs that didn’t allow unauthenticated relaying or restricted sending IP’s so that workstations couldn’t become spam zombies. Just do a yum install telnet inside the PMM container. It looks something like:
telnet smtp.server 25
helo test
mail from: user@domain.com
rcpt to: recipient@domain.com
data
subject: test
this is a test message
(note there’s a double enter after the message to trigger the actual send)
FYI to share the details I have to change any internal host name and IP to something that is not valid, so the mailrealy etc is mocked up for this post, the real values do work.
I do see the GF values in the pmm container
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29d03a8dfe81 percona/pmm-server:2 “/opt/entrypoint.sh” 21 minutes ago Up 21 minutes (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp great_cohen
[root@10.xx.xx.xx ~]# docker exec -it great_choen bash
env | grep GF
GF_SMTP_PASSWORD=
GF_SMTP_HOST=mailrelay:25
GF_SMTP_ENABLED=true
GF_SMTP_FROM_ADDRESS=pmm-noreply@test.com
GF_PLUGIN_DIR=/srv/grafana/plugins
GF_SMTP_FROM_NAME=Percona Alerting
GF_SMTP_SKIP_VERIFY=true
GF_SMTP_USER=
We have other security measures to ensure the relay is not a spambot so it does not require authentication.
I ran the telnet test as requested in the pmm container
[root@29d03a8dfe81 opt] # telnet mailrelay 25
Trying 10.x.x.x…
Connected to mailrelay.
Escape character is ‘^]’.
220 mailrelay ESMTP Postfix
helo test
250 mailrelay
mail from: test@test.com
250 2.1.0 Ok
rcpt to: test@test.com
250 2.1.5 Ok
data
354 End data with .
subject: test from pmm
test message from pmm docker
.
250 2.0.0 Ok: queued as 7341E3803F2C
Mail was received.
Still getting that SMTP not configured error. Looking in the container at what should be the grafana.ini at /etc/grafana/grafana.ini I still don’t see and SMTP section.
I also tried adding a fake user/password, same result.
I’m using the same mail configuration in a promethus alertmanager setup side by side with the percona postgres operator + PMM with out issues. It seems like there is something that is not taking the environment variables and placing them in the grafana.ini when using the docker setup.
From alertmanager
- smarthost: 'mailrelay:25'
from: 'pmm-noreply@test.com'
to: 'test@test.com'
headers:
subject: 'alerts'
Thanks again for taking time to look at this issue.
Ok…that seems to say that grafana isn’t picking up the env vars on container start.
You can try this but it’d be only a test:
get a shell inside the container
Directly edit the grafana.ini values
use supervisorctl restart grafana
test
that should keep the changes long enough to test with and it should work until you update the container (where that edited file would be replaced with the default one). if that works we may have to open a bug with the PMM team (https://jira.percona.com but lets hope it’s not that).
also you can see if grafana is picking up the overrides here:
https:///graph/admin/settings
look in the SMTP section (I think we know it’s not but just an easy place to verify in the future).
Yes that test worked!
We really do need a bug report but I don’t have access to jira so I can only put one in on github.
I have an automated process that builds postgres clusters and adds PMM for various business units. The business units treat it as a black box so our automation needs to be able to fully set up a working pmm, which makes any manual work a problem.
Thanks again for all your help.
Our jira is public so you can register for an account and submit a bug there. I do believe the PMM team monitors github issues and converts to Jira issues so if you’ve already submitted a GH issue, they’ll get it.
