Alert manager integration, receiving 3 alerts for mysql and 1 alert for postgres

Percona Monitoring and Management (PMM) PMM 2.x
1

Have integrated pmm2 with

Both Mysql and Postgres have 3 exporters defined when we created a system in PMM

Have created 3 systems and details as follows

  1. postgres-prod1

Exporters:-

postgres_exporter_agent_id_9e585436-28f9-4821-878a-ebb4fe1464ce_hr-5s

postgres_exporter_agent_id_9e585436-28f9-4821-878a-ebb4fe1464ce_lr-1m0s

postgres_exporter_agent_id_9e585436-28f9-4821-878a-ebb4fe1464ce_mr-10s

==============================================================

  1. postgres-test

Exporters:-

postgres_exporter_agent_id_6bcf3f79-96ec-442b-bb53-9556b04686cb_hr-5s

postgres_exporter_agent_id_6bcf3f79-96ec-442b-bb53-9556b04686cb_lr-1m0s

postgres_exporter_agent_id_6bcf3f79-96ec-442b-bb53-9556b04686cb_mr-10s

==========================================================

  1. mysql-test

Exporters:-

mysqld_exporter_agent_id_a46bb1de-bebb-45ca-bde2-e31e82eb0159_hr-5s

mysqld_exporter_agent_id_a46bb1de-bebb-45ca-bde2-e31e82eb0159_lr-1m0s

mysqld_exporter_agent_id_a46bb1de-bebb-45ca-bde2-e31e82eb0159_mr-10s

================================================

When postgres is down we are getting only one alert from alert manager which is having the postgres_exporter_agent_id_9e585436-28f9-4821-878a-ebb4fe1464ce_hr-5s

but where as when mysql is down we are getting 3 alerts

mysqld_exporter_agent_id_a46bb1de-bebb-45ca-bde2-e31e82eb0159_hr-5s

mysqld_exporter_agent_id_a46bb1de-bebb-45ca-bde2-e31e82eb0159_lr-1m0s

mysqld_exporter_agent_id_a46bb1de-bebb-45ca-bde2-e31e82eb0159_mr-10s

For Postgres we are getting only below alert [-5s] but not [-1m0s,-10s]

postgres_exporter_agent_id_9e585436-28f9-4821-878a-ebb4fe1464ce_hr-5s

When both postgres and mysql have 3 jobs each why only postgres we are getting 1 alert only for -5s but for mysql we are getting 3 alerts for the three jobs [-5s,-1m0s,-10s]

alertmanager.yml

prometheus.yml

rules.yml attached

files.zip

files.zip (1.55 KB)

2

Under the hood, mysqld_exporter returns “mysql_up” metric for each job/resolution. So you have to use aggregation in your alerting rule. For example: “min by (service_name, node_name) (mysql_up) == 0”.