# How to set new custom Alert and Generate Notification

**URL:** <https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805>\
**Category:** Percona Server for MongoDB\
**Tags:** community\
**Created:** [July 22, 2020, 10:53am UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805 "2020-07-22T10:53:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Atanu](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/atanu/32/1366_2.png) [@Atanu](https://forums.percona.com/u/Atanu)\
**Post date:** [July 22, 2020, 10:53am UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/1 "2020-07-22T10:53:14Z")

</div>

Hi Expert,  
I am very new to Percona, I have configured Percona for MongoDB and getting metrics. Now I want set some custom alerts on MongoDB metrics as per my project requirements and after that want to add SMTP server details for sending email notification; Similar kind of configuration I have done in standalone Prometheus but I am little confuse how configure&nbsp;alert.rules.yml and alertmanager.yml files for Prometheus inside Percona. Here Prometheus is running inside Docker image and I can not update these files directly.  
Can you please tell me is it possible ? if possible can you please guide me or provide me documentation link so that I can go through it.  
Thank you!!  
Atanu  
&nbsp;&nbsp;

---

<div class="post-metadata">

**Author:** ![Atanu](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/atanu/32/1366_2.png) [@Atanu](https://forums.percona.com/u/Atanu)\
**Post date:** [July 22, 2020, 10:54am UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/2 "2020-07-22T10:54:51Z")

</div>

Hi Expert,  
I am very new to Percona, I have configured Percona for MongoDB and getting metrics. Now I want set some custom alerts on MongoDB metrics as per my project requirements and after that want to add SMTP server details for sending email notification; Similar kind of configuration I have done in standalone Prometheus but I am little confuse how configure&nbsp;alert.rules.yml and alertmanager.yml files for Prometheus inside Percona. Here Prometheus is running inside Docker image and I can not update these files directly.  
Can you please tell me is it possible ? if possible can you please guide me or provide me documentation link so that I can go through it.  
Thank you!!  
Atanu

---

<div class="post-metadata">

**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\
**Post date:** [July 22, 2020, 11:31am UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/3 "2020-07-22T11:31:50Z")

</div>

Hi Atanu,  
You may find requested information in the next blogpost  
[https://www.percona.com/blog/2020/02/21/percona-monitoring-and-management-meet-prometheus-alertmanger/](https://www.percona.com/blog/2020/02/21/percona-monitoring-and-management-meet-prometheus-alertmanger/ "Link: https://www.percona.com/blog/2020/02/21/percona-monitoring-and-management-meet-prometheus-alertmanger/")

---

<div class="post-metadata">

**Author:** ![Atanu](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/atanu/32/1366_2.png) [@Atanu](https://forums.percona.com/u/Atanu)\
**Post date:** [July 22, 2020, 11:23pm UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/4 "2020-07-22T23:23:48Z")

</div>

Thank you&nbsp;[adivinho](https://forums.percona.com/profile/adivinho), I have gone through the URL, in that link it is mentioned that we need to configure external Alertmanager.  
Here I have three confusions -  
1. I have separately install Prometheus and Alertmanager, so can I use that as an external Alertmanager ?  
2. How will I determine URL of external Alertmanager ?  
3. I think using this interface we can only set “alert.rules.yml” file in external Alert manager buy how will I set “Alertmanager.yml” file ?  
Thanks,  
Atanu

---

<div class="post-metadata">

**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\
**Post date:** [July 23, 2020, 3:57am UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/5 "2020-07-23T03:57:11Z")

</div>

Yes, you should use an external alertmanager.   
Please specify ip and port of host with installed Alertmanager.&nbsp;Alertmanager’s default port is 9093.&nbsp;  
Rules are stored in the folder&nbsp;/srv/prometheus/rules/. It have to be&nbsp;\*.yml files.

---

<div class="post-metadata">

**Author:** ![Atanu](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/atanu/32/1366_2.png) [@Atanu](https://forums.percona.com/u/Atanu)\
**Post date:** [July 23, 2020, 2:11pm UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/6 "2020-07-23T14:11:37Z")

</div>

Thank you&nbsp;[adivinho](https://forums.percona.com/profile/adivinho),  
I hope I get it correctly :

1. Metrics will be received into Percona&nbsp; (_Prometheus reside inside Percona_)
2. We need to have an external Alertmanager (_separate Prometheus outside Percona_)
3. Need to set an alert rule in Percona web UI with the external Alertmanager URL
4. SMTP details need to configured in external Alertmanager&nbsp;

---

<div class="post-metadata">

**Author:** ![Vadim\_Yalovets](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadim_yalovets/32/43_2.png) [@Vadim\_Yalovets](https://forums.percona.com/u/Vadim_Yalovets)\
**Post date:** [July 24, 2020, 3:01am UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/7 "2020-07-24T03:01:29Z")

</div>

Yes. It’s correct.

---

<div class="post-metadata">

**Author:** ![mkuna](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@mkuna](https://forums.percona.com/u/mkuna)\
**Post date:** [September 9, 2021, 1:02pm UTC](https://forums.percona.com/t/how-to-set-new-custom-alert-and-generate-notification/7805/8 "2021-09-09T13:02:17Z")

</div>

I had similar approach, but implementing from the server side.

well I did pmm-server and agent installation with kubernetes approach and now planning to setup an integration with external alertmanager.

I defined alert-template, rules and prometheus.base.yml and the

**expectation** is that pmm-server should notify the external alert manager based on the rule that defined.

i dont see entries that got created under /etc/ folder which supposed to be according to this

> **[Integrated Alerting Design in Percona Monitoring and Management - Percona...](https://www.percona.com/blog/2021/04/16/integrated-alerting-design-in-percona-monitoring-and-management/)**
>
> Percona Monitoring and Management 2.13 introduced Integrated Alerting feature as a technical preview, and here we focus on design and implementation details.

alert-template placed /srv/ia/templates/mongos-0.yml

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: pmm-rules-template
  labels:
    app: {{ template "percona.fullname" . }}
    name: {{ template "percona.fullname" . }}
data:
  mongos-0.yml: |-
    ---
    templates:
      - name: mongos-0-ServiceDown
        version: 1
        summary: MongoDB is down
        expr: |-
          up{service_type="mongodb"} == 0
        severity: critical
        annotations:
          summary: MongoDB is down {{`{{$labels.service_name}}`}}
          description: |-
            MongoDB {{`{{$labels.service_name}}`}} on {{`{{$labels.node_name}}`}} is down

```

rules → /srv/prometheus/rules/rules.yml

```auto
#Adding Prometheus rules: vi /srv/prometheus/rules/rules.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: pmm-prometheus-rules
  labels:
    app: {{ template "percona.fullname" . }}
    name: {{ template "percona.fullname" . }}
data:
  rules.yml: |-
    ---
    groups:
        - name: PMM Integrated Alerting
          rules:
            - alert: mongos-0-ServiceDown
              expr: up{service_type="mongos-0"} == 0
              for: 1s
              labels:
                ia: "1"
                severity: critical
                template_name: mongos-0-ServiceDown
              annotations:
                description: MongoDB {{`{{$labels.service_name}}`}} on {{`{{$labels.node_name}}`}} is down
                rule: test-mongos-0
                summary: mongos-0-ServiceDown {{`{{$labels.service_name}}`}}

```

/srv/prometheus/prometheus.base.yml

```auto
#Integrating PMM Prometheus with AlertManager: /srv/prometheus/prometheus.base.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: pmm-alertmanager-config
  labels:
    app: {{ template "percona.fullname" . }}
    name: {{ template "percona.fullname" . }}
data:
  prometheus.base.yml: |-
    ---
    rule_files:
      - /srv/prometheus/rules/rules.yml
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - http://x.x.x.x:9093

```

please let me know if i miss anything