# Labels are not working if we create an alert rule **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Created:** [June 29, 2023, 10:45am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209 "2023-06-29T10:45:24Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 10:45am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/1 "2023-06-29T10:45:24Z")
Label’s are not working for alert rules. 1. Without the label filter, the command is working fine. **(1 - avg by(node\_name) (rate(node\_cpu\_seconds\_total{mode=“idle”}[5m]))) \* 100 \> 0** 1. With the label filter, then command is not working. **label\_match((1 - avg by(node\_name) (rate(node\_cpu\_seconds\_total{mode=“idle”}[5m]))) \* 100 \> bool 1, “environment”, “PROD”)** ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 29, 2023, 10:55am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/2 "2023-06-29T10:55:37Z")
Hi @Naresh9999, Can you try without bool? It seems unnecessary: ```auto label_match((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode=“idle”}[5m]))) * 100 > 1, “environment”, “PROD”) ``` Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 11:09am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/3 "2023-06-29T11:09:20Z")
@kedarpercona Thank you for your reply. Even I tried that as well, but no luck. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/7/7d8d91401a4c526b7181096d7d3fc285d99474cd.png) ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 11:11am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/4 "2023-06-29T11:11:15Z")
Without the label, it is working fine. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/0/0638e3d02c53f239a53b36c81111eabee4603905.png) ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 29, 2023, 11:29am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/5 "2023-06-29T11:29:10Z")
Hey, Can you confirm if you really have a label of “environment” and value “PROD”? Can you try with this instead (do replace YOUR\_SERVER\_NAME with correct value) ```auto label_match((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode=“idle”}[5m]))) * 100 > 1, “node_name”, “YOUR_SERVER_NAME”) ``` Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 11:41am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/6 "2023-06-29T11:41:55Z")
Yes @kedarpercona Label is there. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/0/0346685f2ad4816b3d54e91589f7932ceed2ad8f.png) I am using the same environment Label for other metrics is working fine. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/3/3670e6736e67e435bf7354ab2d0a52f937355ed8.png) If I add Label as node\_name then it works fine. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/c/c4021fd0ab2ce7a00154ea22b579b99ec6ca6de0.png) ---
**Author:** ![artem\_gavrilov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/artem_gavrilov/32/1453_2.png) [@artem\_gavrilov](https://forums.percona.com/u/artem_gavrilov)\ **Post date:** [June 29, 2023, 12:17pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/7 "2023-06-29T12:17:31Z")
Hi @Naresh9999 , this happens because of aggregation function `avg by(node_name)`. When it’s applied the only label presented in query result in `node_name` and as you can see filtering applied to the whole query. Proper solution for your case will be apply filter function before aggregation, try this query: `(1 - avg by(node_name) (rate(label_match(node_cpu_seconds_total{}[5m], "environment", "PROD")))) * 100 > 0` ---
**Author:** ![artem\_gavrilov](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/artem_gavrilov/32/1453_2.png) [@artem\_gavrilov](https://forums.percona.com/u/artem_gavrilov)\ **Post date:** [June 29, 2023, 12:24pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/8 "2023-06-29T12:24:35Z")
Or this one: `(1 - avg by(node_name) (label_match(rate(node_cpu_seconds_total{}[5m]), "environment", "PROD"))) * 100 > 0` ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 12:31pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/9 "2023-06-29T12:31:35Z")
@artem_gavrilov Thank you for your reply. I tried both ways, but no luck. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/6/6496fa2ebba668d74b9dee90d4d46d1241fc4303.png) ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/0/02b641d9409c00f5a5d74151875200d77a02cc35.png) ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 29, 2023, 1:14pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/10 "2023-06-29T13:14:32Z")
Hello @Naresh, For your last chart query, when you type `{mode="idle",environment="` do you get “PROD” as one of the suggestion? Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 1:48pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/11 "2023-06-29T13:48:57Z")
No @kedarpercona Even, I could not fetch the environment label when I typed after mode, I am able to see only Linux-related Labels while I am typing or start typing; they are automatically displayed. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/8/88dd77c90069a77f64f57a4bb377bbc47d66c284.png) ## Below, Linux node related labels are working fine: pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1) --custom-labels=“SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA, APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES” --region=“AWS-DC” 192.10.1.1 generic nvmmm101 ## Below, DB service-related labels are not working: pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none --environment=PROD --replication-set=ECOM-REP As per my observations, node related Labels, including custom labels, I am able to see while I am typing, but service-related labels I cannot see when I am typing or start typing. It seems there is an issue with service related Labels. ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 29, 2023, 2:00pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/12 "2023-06-29T14:00:34Z")
Right, Exactly my doubt. And now you know that since they’re not present, filter will not work. Some how your labels are not applied to this system metric, now the question becomes why it is not applied! Because it is not added while adding the system agent… You might want to add the environment=PROD in custom-labels and try again. Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 2:06pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/13 "2023-06-29T14:06:10Z")
@kedarpercona But already, environment label exists, so we can not add in the custom labels, right? pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none --environment=PROD --replication-set=ECOM-REP We can say it’s a bug. Why? Because an environment filter must be applied to system, right? ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 29, 2023, 2:15pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/14 "2023-06-29T14:15:24Z")
@Naresh9999, If you do `pmm-admin list` you will not different agents. When you’re mentioning here `pmm-admin add mysql`, it is going to add an agent for MySQL with said properties and it will only apply to the data collected specific to “mysql”. The one we 're concerned about is `node_exporter`. Thus when you configure the node, consider adding a customer label. It appears that --environment is not available for node\_exporter. Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 3:55pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/15 "2023-06-29T15:55:16Z")
@kedarpercona I think system metrics should be required to have a unique environment name, such as env or something comparable. OR Will it accept an environment Label for both system metrics and service metrics? Like below. 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1) --custom-labels="SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA, APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES, **environment=PROD"** --region=“AWS-DC” 192.10.1.1 generic nvmmm101 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 29, 2023, 5:15pm UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/17 "2023-06-29T17:15:17Z")
@artem_gavrilov @kedarpercona Will it accept an environment Label for both system metrics and service metrics? Like below. 1. pmm-admin config --server-insecure-tls --server-url=[http://admin:XXXXXXXXX@192.168.1.1](http://admin:XXXXXXXXX@192.168.1.1) --custom-labels="SERVICE\_RUNNING=MariaDB, DB\_SERVICE\_TYPE=REPLICA, APP\_TYPE=ECOM,BACKUP\_JOB=YES,IsSlave=YES, **environment=PROD"** --region=“AWS-DC” 192.10.1.1 generic nvmmm101 2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP [/quote] ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 30, 2023, 9:16am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/18 "2023-06-30T09:16:52Z")
Can one of you please help me with the above issue? ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 30, 2023, 10:03am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/19 "2023-06-30T10:03:29Z")
Hi Naresh, To answer your question ` "Will it accept an environment Label"` I’d actually have to try… If you have a lab ready, can you try yourself please? Thanks, K ---
**Author:** ![kedarpercona](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kedarpercona/32/809_2.png) [@kedarpercona](https://forums.percona.com/u/kedarpercona)\ **Post date:** [June 30, 2023, 10:40am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/20 "2023-06-30T10:40:55Z")
Okay @Naresh9999, for me this works → **–custom-labels=“environment=PROD”** ```auto [root@ip]# pmm-admin config --server-insecure-tls --server-url=http://admin:admin@IP --custom-labels="environment=PROD" --force Warning: PMM Server requires TLS communications with client. Checking local pmm-agent status... pmm-agent is running. Registering pmm-agent on PMM Server... Registered. Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated. Reloading pmm-agent configuration... Configuration reloaded. Checking local pmm-agent status... pmm-agent is running. ``` ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/a/a62acecda634dcd411c2692540e7c93df9bb2eaa.png) Thanks, K ---
**Author:** ![Naresh9999](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/naresh9999/32/5568_2.png) [@Naresh9999](https://forums.percona.com/u/Naresh9999)\ **Post date:** [June 30, 2023, 11:04am UTC](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209/21 "2023-06-30T11:04:09Z")
@kedarpercona Can you please add DB service with the below command? why because DB service also has environment=PROD Label. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none **–environment=PROD** --replication-set=ECOM-REP And, also confirm that Labels are independent to system metrics and service metrics? [Next page](https://forums.percona.com/t/labels-are-not-working-if-we-create-an-alert-rule/23209.md?page=2)