Labels are not working if we create an alert rule

Percona Monitoring and Management (PMM)
1

Label’s are not working for alert rules.

  1. Without the label filter, the command is working fine.

(1 - avg by(node_name) (rate(node_cpu_seconds_total{mode=“idle”}[5m]))) * 100 > 0

  1. With the label filter, then command is not working.

label_match((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode=“idle”}[5m]))) * 100 > bool 1, “environment”, “PROD”)

2

Hi @Naresh9999,

Can you try without bool? It seems unnecessary:

label_match((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode=“idle”}[5m]))) * 100 > 1, “environment”, “PROD”)

Thanks,
K

3

@kedarpercona Thank you for your reply.

Even I tried that as well, but no luck.

image
image1827×428 23.8 KB

4

Without the label, it is working fine.

image
image1818×397 42.7 KB

5

Hey,

Can you confirm if you really have a label of “environment” and value “PROD”?
Can you try with this instead (do replace YOUR_SERVER_NAME with correct value)

label_match((1 - avg by(node_name) (rate(node_cpu_seconds_total{mode=“idle”}[5m]))) * 100 > 1, “node_name”, “YOUR_SERVER_NAME”)

Thanks,
K

6

Yes @kedarpercona Label is there.

image
image1701×337 19.6 KB

I am using the same environment Label for other metrics is working fine.

image
image1840×789 52.3 KB

If I add Label as node_name then it works fine.

image
image1855×561 38.4 KB

7

Hi @Naresh9999 , this happens because of aggregation function avg by(node_name). When it’s applied the only label presented in query result in node_name and as you can see filtering applied to the whole query. Proper solution for your case will be apply filter function before aggregation, try this query:
(1 - avg by(node_name) (rate(label_match(node_cpu_seconds_total{}[5m], "environment", "PROD")))) * 100 > 0

8

Or this one: (1 - avg by(node_name) (label_match(rate(node_cpu_seconds_total{}[5m]), "environment", "PROD"))) * 100 > 0

9

@artem_gavrilov Thank you for your reply. I tried both ways, but no luck.

image
image1875×675 40.2 KB

image
image1920×783 56.3 KB

10

Hello @Naresh,

For your last chart query, when you type {mode="idle",environment=" do you get “PROD” as one of the suggestion?

Thanks,
K

11

No @kedarpercona

Even, I could not fetch the environment label when I typed after mode, I am able to see only Linux-related Labels while I am typing or start typing; they are automatically displayed.

image
image1851×904 89.2 KB

Below, Linux node related labels are working fine:

pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels=“SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA, APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES” --region=“AWS-DC” 192.10.1.1 generic nvmmm101

Below, DB service-related labels are not working:

pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none --environment=PROD --replication-set=ECOM-REP

As per my observations, node related Labels, including custom labels, I am able to see while I am typing, but service-related labels I cannot see when I am typing or start typing. It seems there is an issue with service related Labels.

12

Right, Exactly my doubt.
And now you know that since they’re not present, filter will not work. Some how your labels are not applied to this system metric, now the question becomes why it is not applied! Because it is not added while adding the system agent… You might want to add the environment=PROD in custom-labels and try again.

Thanks,
K

13

@kedarpercona But already, environment label exists, so we can not add in the custom labels, right?

pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none --environment=PROD --replication-set=ECOM-REP

We can say it’s a bug. Why? Because an environment filter must be applied to system, right?

14

@Naresh9999,

If you do pmm-admin list you will not different agents.
When you’re mentioning here pmm-admin add mysql, it is going to add an agent for MySQL with said properties and it will only apply to the data collected specific to “mysql”.
The one we 're concerned about is node_exporter. Thus when you configure the node, consider adding a customer label. It appears that --environment is not available for node_exporter.

Thanks,
K

15

@kedarpercona

I think system metrics should be required to have a unique environment name, such as env or something comparable.

OR

Will it accept an environment Label for both system metrics and service metrics? Like below.

  1. pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels="SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA, APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES,environment=PROD" --region=“AWS-DC” 192.10.1.1 generic nvmmm101

  2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP

17

@artem_gavrilov @kedarpercona

Will it accept an environment Label for both system metrics and service metrics? Like below.

  1. pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels="SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA, APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES,environment=PROD" --region=“AWS-DC” 192.10.1.1 generic nvmmm101
  2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP
    [/quote]
18

Can one of you please help me with the above issue?

19

Hi Naresh,

To answer your question "Will it accept an environment Label" I’d actually have to try… If you have a lab ready, can you try yourself please?

Thanks,
K

20

Okay @Naresh9999,

for me this works → –custom-labels=“environment=PROD”

[root@ip]# pmm-admin config --server-insecure-tls --server-url=http://admin:admin@IP --custom-labels="environment=PROD" --force
Warning: PMM Server requires TLS communications with client.
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.

image
image1924×296 30.1 KB

Thanks,
K

21

@kedarpercona Can you please add DB service with the below command? why because DB service also has environment=PROD Label.

pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP

And, also confirm that Labels are independent to system metrics and service metrics?