Cluster alert rule on pmm

Percona Monitoring and Management (PMM)
, ,
1

Hi Team,

I am using MYSQL xtradb cluster with 5 nodes. I defined an alarm on pmm for Mysql down situations and there is no problem, it works properly.
Recently, nodes were disconnected from the cluster, I was not aware of this situation because the services were working. At the cluster level, I want to receive an alert when any node falls out of the cluster . If there is a ready template or method for this, please let me know.

2

Hello bthnklc,
Are you using Alertmanager as the alerting solution?
If that a case You may want to create a rule something like below:

  • alert: GaleraNodeDown
    expr: wsrep_local_state < 4
    for: 1m
    labels:
    severity: critical
    annotations:
    summary: “Galera Cluster Node Down”
    description: “Node {{ $labels.instance }} is not in the ‘Synced’ state”

Regards,
Denis Subbota.
Managed Services, Percona.
3

Hi Denis ,
Thank you for your response.
We are using PMM but I couldn’t find where to do it. Can you direct me?

4

Hi bthnklc,
What version of pmm server you are using.

Regards,
Denis Subbota.
Managed Services, Percona.

5

Hi @Denis_Subbota ,

Current version: 2.41.2 (March 19, 2024 UTC)

Thanks.

6

for version 2.41.2 you will be able to find Alerting → Alert rules
To create a alert rule you need first to create template for this:

For expression you can use mysql_global_status_wsrep_cluster_status[5m] < [[ .threshold ]]
and as threshold value set 1 so alert will be triggered anytime when your pxc status is not equal 1 (Sync )

and onse it’s created you will be able to add rule:

Regards,
Denis Subbota.
Managed Services, Percona.

7

Hi, if I am using Grafana instead of AlertManager, how can I do this? Thanks.

8

Please try to use grafana documentation for this configuration, and let us know if you facing some issues there.