# Cluster alert rule on pmm

**URL:** <https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519>\
**Category:** Percona Monitoring and Management (PMM)\
**Tags:** pmm, mysql, percona\
**Created:** [May 23, 2024, 12:42pm UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519 "2024-05-23T12:42:33Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![bthnklc](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bthnklc](https://forums.percona.com/u/bthnklc)\
**Post date:** [May 23, 2024, 12:42pm UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/1 "2024-05-23T12:42:33Z")

</div>

Hi Team,

I am using MYSQL xtradb cluster with 5 nodes. I defined an alarm on pmm for Mysql down situations and there is no problem, it works properly.  
Recently, nodes were disconnected from the cluster, I was not aware of this situation because the services were working. At the cluster level, I want to receive an alert when any node falls out of the cluster . If there is a ready template or method for this, please let me know.

---

<div class="post-metadata">

**Author:** ![Denis\_Subbota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/denis_subbota/32/23607_2.png) [@Denis\_Subbota](https://forums.percona.com/u/Denis_Subbota)\
**Post date:** [May 27, 2024, 2:02pm UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/2 "2024-05-27T14:02:01Z")

</div>

Hello [bthnklc](https://forums.percona.com/u/bthnklc),  
Are you using Alertmanager as the alerting solution?  
If that a case You may want to create a rule something like below:

- alert: GaleraNodeDown  
expr: wsrep\_local\_state \< 4  
for: 1m  
labels:  
severity: critical  
annotations:  
summary: “Galera Cluster Node Down”  
description: “Node {{ $labels.instance }} is not in the ‘Synced’ state”

```auto

Regards,
Denis Subbota.
Managed Services, Percona.
```

---

<div class="post-metadata">

**Author:** ![bthnklc](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bthnklc](https://forums.percona.com/u/bthnklc)\
**Post date:** [May 28, 2024, 11:27am UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/3 "2024-05-28T11:27:06Z")

</div>

> [@Denis\_Subbota](#):
>
> Are you using Alertmanager as the alerting solution?  
> If that a case You may want to create a rule something like below:
> 
> - alert: GaleraNodeDown  
> expr: wsrep\_local\_state \< 4  
> for: 1m  
> labels:  
> severity: critical  
> annotations:  
> summary: “Galera Cluster Node Down”  
> description: “Node {{ $labels.instance }} is not in the ‘Synced’ state”

Hi Denis ,  
Thank you for your response.  
We are using PMM but I couldn’t find where to do it. Can you direct me?

---

<div class="post-metadata">

**Author:** ![Denis\_Subbota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/denis_subbota/32/23607_2.png) [@Denis\_Subbota](https://forums.percona.com/u/Denis_Subbota)\
**Post date:** [June 5, 2024, 6:44am UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/4 "2024-06-05T06:44:39Z")

</div>

Hi bthnklc,  
What version of pmm server you are using.

Regards,  
Denis Subbota.  
Managed Services, Percona.

---

<div class="post-metadata">

**Author:** ![bthnklc](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bthnklc](https://forums.percona.com/u/bthnklc)\
**Post date:** [June 5, 2024, 6:51am UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/5 "2024-06-05T06:51:48Z")

</div>

Hi @Denis_Subbota ,

Current version: 2.41.2 (March 19, 2024 UTC)

Thanks.

---

<div class="post-metadata">

**Author:** ![Denis\_Subbota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/denis_subbota/32/23607_2.png) [@Denis\_Subbota](https://forums.percona.com/u/Denis_Subbota)\
**Post date:** [June 12, 2024, 8:18am UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/6 "2024-06-12T08:18:23Z")

</div>

for version 2.41.2 you will be able to find Alerting → Alert rules  
To create a alert rule you need first to create template for this:

> **[Percona Monitoring and Management - Percona Alerting](https://docs.percona.com/percona-monitoring-and-management/get-started/alerting.html#template-example)**
>
> Alerting notifies of important or unusual activity in your database environments so that you can identify and resolve problems quickly. When something needs your attention, Percona Alerting can be configured to automatically send you a notification...

For expression you can use mysql\_global\_status\_wsrep\_cluster\_status[5m] \< [[.threshold]]  
and as threshold value set 1 so alert will be triggered anytime when your pxc status is not equal 1 (Sync )

and onse it’s created you will be able to add rule:

> **[Percona Monitoring and Management - Percona Alerting](https://docs.percona.com/percona-monitoring-and-management/get-started/alerting.html#add-an-alert-rule)**
>
> Alerting notifies of important or unusual activity in your database environments so that you can identify and resolve problems quickly. When something needs your attention, Percona Alerting can be configured to automatically send you a notification...

Regards,  
Denis Subbota.  
Managed Services, Percona.

---

<div class="post-metadata">

**Author:** ![benkim](https://avatars.discourse-cdn.com/v4/letter/b/cdc98d/32.png) [@benkim](https://forums.percona.com/u/benkim)\
**Post date:** [July 25, 2024, 12:58pm UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/7 "2024-07-25T12:58:36Z")

</div>

Hi, if I am using Grafana instead of AlertManager, how can I do this? Thanks.

---

<div class="post-metadata">

**Author:** ![Denis\_Subbota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/denis_subbota/32/23607_2.png) [@Denis\_Subbota](https://forums.percona.com/u/Denis_Subbota)\
**Post date:** [August 16, 2024, 1:34pm UTC](https://forums.percona.com/t/cluster-alert-rule-on-pmm/30519/8 "2024-08-16T13:34:54Z")

</div>

Please try to use grafana documentation for this configuration, and let us know if you facing some issues there.