How to set new custom Alert and Generate Notification

Hi Expert,
I am very new to Percona, I have configured Percona for MongoDB and getting metrics. Now I want set some custom alerts on MongoDB metrics as per my project requirements and after that want to add SMTP server details for sending email notification; Similar kind of configuration I have done in standalone Prometheus but I am little confuse how configure alert.rules.yml and alertmanager.yml files for Prometheus inside Percona. Here Prometheus is running inside Docker image and I can not update these files directly.
Can you please tell me is it possible ? if possible can you please guide me or provide me documentation link so that I can go through it.
Thank you!!
Atanu
  

Hi Expert,
I am very new to Percona, I have configured Percona for MongoDB and getting metrics. Now I want set some custom alerts on MongoDB metrics as per my project requirements and after that want to add SMTP server details for sending email notification; Similar kind of configuration I have done in standalone Prometheus but I am little confuse how configure alert.rules.yml and alertmanager.yml files for Prometheus inside Percona. Here Prometheus is running inside Docker image and I can not update these files directly.
Can you please tell me is it possible ? if possible can you please guide me or provide me documentation link so that I can go through it.
Thank you!!
Atanu

Hi Atanu,
You may find requested information in the next blogpost
https://www.percona.com/blog/2020/02/21/percona-monitoring-and-management-meet-prometheus-alertmanger/

Thank you adivinho, I have gone through the URL, in that link it is mentioned that we need to configure external Alertmanager.
Here I have three confusions -
1. I have separately install Prometheus and Alertmanager, so can I use that as an external Alertmanager ?
2. How will I determine URL of external Alertmanager ?
3. I think using this interface we can only set “alert.rules.yml” file in external Alert manager buy how will I set “Alertmanager.yml” file ?
Thanks,
Atanu

Yes, you should use an external alertmanager.
Please specify ip and port of host with installed Alertmanager. Alertmanager’s default port is 9093. 
Rules are stored in the folder /srv/prometheus/rules/. It have to be *.yml files.

Thank you adivinho,
I hope I get it correctly :

  1. Metrics will be received into Percona  (Prometheus reside inside Percona)
  2. We need to have an external Alertmanager (separate Prometheus outside Percona)
  3. Need to set an alert rule in Percona web UI with the external Alertmanager URL
  4. SMTP details need to configured in external Alertmanager 

Yes. It’s correct.