Unable to receive alerts from mail/slack for PMM2 alerting

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi Guys,

I have configured alerting configuration with Slack and Gmail configurations and created the alerts rules for MySQL down and other stuff.
When I stopped the MySQL DB then I am not getting the alerts from Slack or Gmail. But I can see the alerts in the Prometheus Alertmanager URL .

Can you please somebody help on this issue? How do I debug this issue?

Even I don’t see alerts in the below PMM2 GUI screenshot.

image
image1015×489 42.3 KB

2

Hi, can you paste some screens of your configured notification channels? the following post might be helpful: MongoDB Integrated Alerting in Percona Monitoring and Management - Percona Database Performance Blog just ignore the mongodb part.

1 Like
3

What did you mean by “can see the alerts in the prometheus alertmanager url”? if you go to https://<pmm-server>/alertmanager or are you configuring an external alert manager?

when I’m troubleshooting PMM Alerting (not grafana alerts or External Alertmanager) I first hit the https://<pmm-server>/alertmanager url to see if the alert is firing there. if I see nothing in alertmanager grab the expression (click the Alert Rules tab and the down arrow next to the alert rule name) and paste that into the “Explore” feature of Grafana without the threshold criteria to see what is in the victoriametrics database.

If you’re using an external alertmanager integration (an alertmanager running on a separate container/instance that you configure PMM to point to setup at PMM Settings → Alertmanager Integration) then the communications preferences won’t work with that and you’ll need to setup communication channels directly in that alertmanager instance.

1 Like
4

Hi @Ivan_Groenewold

Thanks for the quick response. I have tried the same steps which you have given, but still, I am unable to receive the mail alerts from Mail/Slack. I have attached the screenshots, I am able to see the alerts from an alert manager.

image

image
image529×592 17 KB

image
image
image1296×515 23 KB

image
image1297×542 27 KB

image
image1366×657 34.4 KB

image
image1362×730 47.9 KB
’
image
image1366×735 53.1 KB

Can you help how to debug or fix the issue?

1 Like
5

Interesting…the only thing I can think but can’t tell from the screenshots is do your “filter” criteria for the alert rule match the node that’s down? Looks like the alert rule is filtering for lables of env=test and db=mysql…so if the node that’s down doesn’t meet that filter criteria we’ll not notify any of your communication channels (and nothing will show on the alerts tab). Try just removing the filters for now. Once it’s on the alerts tab we can look beyond to slack/email.

1 Like
6

Hi @steve.hoffman ’

Thanks for the help. It’s working fine for me after removing the filters.
I have a few below queries.

  1. How do I filter my environment and databases.?
  2. For me Gmail notification is not working, I am using my personal Gmail id for sender and receiver in the notification channel.

Once again thanks for the help… Thank you so much for the help.

1 Like
7

I follow the post and add alert for mongodb, but can not see the alerts from an alert manager.

1 Like
8

Hi @sshine

Did you configure the Email/Slack configuration in the communication settings?

1 Like
9

after upgrade to pmm2 2.25，when mongodb was stopped，i can see the alert , the Slack dose not sendout message。 but it can sendout other alert messages，don’t know how to debug it！

1 Like
10

Can you share errors here?

1 Like
11

thanks for your help

1 Like