How to set new custom Alert and Generate Notification

MongoDB Percona Server for MongoDB
1

Hi Expert,
I am very new to Percona, I have configured Percona for MongoDB and getting metrics. Now I want set some custom alerts on MongoDB metrics as per my project requirements and after that want to add SMTP server details for sending email notification; Similar kind of configuration I have done in standalone Prometheus but I am little confuse how configure alert.rules.yml and alertmanager.yml files for Prometheus inside Percona. Here Prometheus is running inside Docker image and I can not update these files directly.
Can you please tell me is it possible ? if possible can you please guide me or provide me documentation link so that I can go through it.
Thank you!!
Atanu
  

2

Hi Expert,
I am very new to Percona, I have configured Percona for MongoDB and getting metrics. Now I want set some custom alerts on MongoDB metrics as per my project requirements and after that want to add SMTP server details for sending email notification; Similar kind of configuration I have done in standalone Prometheus but I am little confuse how configure alert.rules.yml and alertmanager.yml files for Prometheus inside Percona. Here Prometheus is running inside Docker image and I can not update these files directly.
Can you please tell me is it possible ? if possible can you please guide me or provide me documentation link so that I can go through it.
Thank you!!
Atanu

1 Like
3

Hi Atanu,
You may find requested information in the next blogpost
https://www.percona.com/blog/2020/02/21/percona-monitoring-and-management-meet-prometheus-alertmanger/

1 Like
4

Thank you adivinho, I have gone through the URL, in that link it is mentioned that we need to configure external Alertmanager.
Here I have three confusions -
1. I have separately install Prometheus and Alertmanager, so can I use that as an external Alertmanager ?
2. How will I determine URL of external Alertmanager ?
3. I think using this interface we can only set “alert.rules.yml” file in external Alert manager buy how will I set “Alertmanager.yml” file ?
Thanks,
Atanu

1 Like
5

Yes, you should use an external alertmanager.
Please specify ip and port of host with installed Alertmanager. Alertmanager’s default port is 9093. 
Rules are stored in the folder /srv/prometheus/rules/. It have to be *.yml files.

1 Like
6

Thank you adivinho,
I hope I get it correctly :

  1. Metrics will be received into Percona  (Prometheus reside inside Percona)
  2. We need to have an external Alertmanager (separate Prometheus outside Percona)
  3. Need to set an alert rule in Percona web UI with the external Alertmanager URL
  4. SMTP details need to configured in external Alertmanager 
1 Like
7

Yes. It’s correct.

1 Like
8

I had similar approach, but implementing from the server side.

well I did pmm-server and agent installation with kubernetes approach and now planning to setup an integration with external alertmanager.

I defined alert-template, rules and prometheus.base.yml and the

expectation is that pmm-server should notify the external alert manager based on the rule that defined.

i dont see entries that got created under /etc/ folder which supposed to be according to this

alert-template placed /srv/ia/templates/mongos-0.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: pmm-rules-template
  labels:
    app: {{ template "percona.fullname" . }}
    name: {{ template "percona.fullname" . }}
data:
  mongos-0.yml: |-
    ---
    templates:
      - name: mongos-0-ServiceDown
        version: 1
        summary: MongoDB is down
        expr: |-
          up{service_type="mongodb"} == 0
        severity: critical
        annotations:
          summary: MongoDB is down {{`{{$labels.service_name}}`}}
          description: |-
            MongoDB {{`{{$labels.service_name}}`}} on {{`{{$labels.node_name}}`}} is down

rules → /srv/prometheus/rules/rules.yml

#Adding Prometheus rules: vi /srv/prometheus/rules/rules.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: pmm-prometheus-rules
  labels:
    app: {{ template "percona.fullname" . }}
    name: {{ template "percona.fullname" . }}
data:
  rules.yml: |-
    ---
    groups:
        - name: PMM Integrated Alerting
          rules:
            - alert: mongos-0-ServiceDown
              expr: up{service_type="mongos-0"} == 0
              for: 1s
              labels:
                ia: "1"
                severity: critical
                template_name: mongos-0-ServiceDown
              annotations:
                description: MongoDB {{`{{$labels.service_name}}`}} on {{`{{$labels.node_name}}`}} is down
                rule: test-mongos-0
                summary: mongos-0-ServiceDown {{`{{$labels.service_name}}`}}

/srv/prometheus/prometheus.base.yml

#Integrating PMM Prometheus with AlertManager: /srv/prometheus/prometheus.base.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: pmm-alertmanager-config
  labels:
    app: {{ template "percona.fullname" . }}
    name: {{ template "percona.fullname" . }}
data:
  prometheus.base.yml: |-
    ---
    rule_files:
      - /srv/prometheus/rules/rules.yml
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - http://x.x.x.x:9093

please let me know if i miss anything

1 Like