I have configured alerting configuration with Slack and Gmail configurations and created the alerts rules for MySQL down and other stuff.
When I stopped the MySQL DB then I am not getting the alerts from Slack or Gmail. But I can see the alerts in the Prometheus Alertmanager URL .
Can you please somebody help on this issue? How do I debug this issue?
Even I don’t see alerts in the below PMM2 GUI screenshot.
Hi, can you paste some screens of your configured notification channels? the following post might be helpful: MongoDB Integrated Alerting in Percona Monitoring and Management - Percona Database Performance Blog just ignore the mongodb part.
What did you mean by “can see the alerts in the prometheus alertmanager url”? if you go to https://<pmm-server>/alertmanager or are you configuring an external alert manager?
when I’m troubleshooting PMM Alerting (not grafana alerts or External Alertmanager) I first hit the https://<pmm-server>/alertmanager url to see if the alert is firing there. if I see nothing in alertmanager grab the expression (click the Alert Rules tab and the down arrow next to the alert rule name) and paste that into the “Explore” feature of Grafana without the threshold criteria to see what is in the victoriametrics database.
If you’re using an external alertmanager integration (an alertmanager running on a separate container/instance that you configure PMM to point to setup at PMM Settings → Alertmanager Integration) then the communications preferences won’t work with that and you’ll need to setup communication channels directly in that alertmanager instance.
Thanks for the quick response. I have tried the same steps which you have given, but still, I am unable to receive the mail alerts from Mail/Slack. I have attached the screenshots, I am able to see the alerts from an alert manager.
Can you help how to debug or fix the issue?
Interesting…the only thing I can think but can’t tell from the screenshots is do your “filter” criteria for the alert rule match the node that’s down? Looks like the alert rule is filtering for lables of env=test and db=mysql…so if the node that’s down doesn’t meet that filter criteria we’ll not notify any of your communication channels (and nothing will show on the alerts tab). Try just removing the filters for now. Once it’s on the alerts tab we can look beyond to slack/email.
Hi @steve.hoffman ’
Thanks for the help. It’s working fine for me after removing the filters.
I have a few below queries.
- How do I filter my environment and databases.?
- For me Gmail notification is not working, I am using my personal Gmail id for sender and receiver in the notification channel.
Once again thanks for the help… Thank you so much for the help.
I follow the post and add alert for mongodb, but can not see the alerts from an alert manager.
Did you configure the Email/Slack configuration in the communication settings?
after upgrade to pmm2 2.25，when mongodb was stopped，i can see the alert , the Slack dose not sendout message。 but it can sendout other alert messages，don’t know how to debug it！
Can you share errors here?