# Labels are not working if we create an alert rule **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Created:** [June 29, 2023, 10:45am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209 "2023-06-29T10:45:24Z")\ **Posts on this page:** 12\ **Page:** 2
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 1, 2023, 7:09am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/22 "2023-07-01T07:09:28Z")
@Roma_Novikov Could you kindly reaffirm that the Labels are separate from system metrics and service metrics? As shown below, if we add **environment=PROD** for both system metrics and service metrics, there should be no ambiguity. 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1/) --custom-labels="SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA, APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES, **environment=PROD"** --region=“AWS-DC” 192.10.1.1 generic nvmmm101 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [July 1, 2023, 1:48pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/23 "2023-07-01T13:48:30Z")
Hi @Naresh9999, Do you think this [question](https://forums.percona.com/t/custom-labels-and-tags-for-host-metrics/9314/) answers you? ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 1, 2023, 6:23pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/24 "2023-07-01T18:23:58Z")
Hi @kedarpercona Not precisely, since Labels are not mentioned anywhere as being independent of nodes and services. It can cause confusion if we use **environment=PROD** as a shared label for both node-level metrics and DB service-level metrics. This still causes me confusion, in part because it is unclear how notifications will act if they are routed. As you can see below, the environment filters for the Home Dashboard and the Nodes Summary Dashboard both have different conditions. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/e/e0f3e1f67bf49c705a715d12526b5213f0c328a2.png) ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/9/9217017528b71032d1fcea651f48683447fc92dc.png) ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 3, 2023, 12:05pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/25 "2023-07-03T12:05:27Z")
@Roma_Novikov Please, if you could help me with the environmental Label. As we move forward with further alerting configurations, the label confusion has actually struck us. ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [July 4, 2023, 8:12am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/26 "2023-07-04T08:12:27Z")
Hello @Naresh9999, > since Labels are not mentioned anywhere as being independent of nodes and services. It can cause confusion if we use **environment=PROD** as a shared label for both node-level metrics and DB service-level metrics. This still causes me confusion, in part because it is unclear how notifications will act if they are routed. So, I do not understand what is a confusion here. The label “environment=PROD” is indicative of the node and the database on it belongs to “prod environment”. Your alerts should be having the appropriate checks pointing at either of them! > As you can see below, the environment filters for the Home Dashboard and the Nodes Summary Dashboard both have different conditions. Did you mean OS Overview is having additional filters (region/type) while Home Dashboard is not? Can you exactly show us “what alert configuration” has confusion due to “label”? Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 4, 2023, 8:55am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/27 "2023-07-04T08:55:28Z")
@kedarpercona, If we specify **environment=PROD** then it only works for Database service-related alerts (MariaDB replication, down…etc), but not for node-related alerts like CPU, memory…etc. Below are pmm commands that we have configured. 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1/) --custom-labels=“SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA, APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES” --region=“AWS-DC” 192.10.1.1 generic nvmmm101 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/2/2bfc775dbfaed2893165032d8b3b81c8ce041a72.png) ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 4, 2023, 5:38pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/28 "2023-07-04T17:38:25Z")
@kedarpercona I only need a clarification that the commands should work well if I use the **-environment=PROD** label for both the Node command and the DB service command like below. **Node Command:** 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1/) --custom-labels=“SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA, APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES, **environment=PROD** ” --region=“AWS-DC” 192.10.1.1 generic nvmmm101 **DB Service Command:** 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [July 5, 2023, 7:46am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/29 "2023-07-05T07:46:34Z")
Hi Naresh, So the second last comment of yours say: > If we specify **environment=PROD** then it only works for Database service-related alerts (MariaDB replication, down…etc), but not for node-related alerts like CPU, memory…etc. Below are pmm commands that we have configured. Correct, it will not have the label because you have not configured it. As we already saw that once we have them set as custom-label it should work. The best thing you can do is configure it and generate a dummy alert for both OS and Database checks to verify if that works. Then in the last comment you have this comment: > @kedarpercona I only need a clarification that the commands should work well if I use the **-environment=PROD** label for both the Node command and the DB service command like below. About this, we both tested and confirmed that it works. And hence to understand your question, may be to reproduce what you’re seeing, I had a question: > Can you exactly show us “what alert configuration” has confusion due to “label”? So as a next step, I’d suggest you to configure both OS and Database with appropriate “environment=PROD” and then generate an alert. Test out your scenario and let us know. Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 5, 2023, 10:33am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/30 "2023-07-05T10:33:40Z")
@kedarpercona Thanks for the details. For both Node and Database PMM commands, I’ll try to set the common environment=PROD Label, and then I’ll test the alerting. I’ll set up the commands listed below on my database servers and let you know when I’m done. **Node Command:** 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1/) --custom-labels=“SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA,APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES, **environment=PROD** ” --region=“AWS-DC” 192.10.1.1 generic nvmmm101 **DB Service Command:** 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 6, 2023, 8:54am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/31 "2023-07-06T08:54:46Z")
@kedarpercona After the above commands execution, I have configured notification policy with the label **environment=PROD** , and alerts for Node metrics are not functioning, but they are functional for DB metrics. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/a/ad2ce530ccaa7d5a5fb80f8b9dfe6b802bc62a31.png) ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 6, 2023, 12:07pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/32 "2023-07-06T12:07:15Z")
@kedarpercona After changing and creating a notification policy like below, it’s working fine for both the Node and DB services. Commands: 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1/) --custom-labels=“SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA,APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES, **ENV=PROD** ” --region=“AWS-DC” 192.10.1.1 generic nvmmm101 **DB Service Command:** 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/3/3172617553e13eb937f314be56fa3730e18e2a83.png) ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [July 7, 2023, 4:57pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/33 "2023-07-07T16:57:46Z")
@kedarpercona The alerts are currently functioning properly for both DB service-related and Node-related alerts, as shown in the screenshot. Thanks for all the help. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/a/a8147d85be793a9e472b17d8d80d9a6a8a29db9e.png) [Previous page](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209.md?page=1)