@Roma_Novikov
Could you kindly reaffirm that the Labels are separate from system metrics and service metrics?
As shown below, if we add environment=PROD for both system metrics and service metrics, there should be no ambiguity.
- pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels="SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA, APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES,environment=PROD" --region=“AWS-DC” 192.10.1.1 generic nvmmm101
- pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP
Hi @Naresh9999,
Do you think this question answers you?
Hi @kedarpercona
Not precisely, since Labels are not mentioned anywhere as being independent of nodes and services. It can cause confusion if we use environment=PROD as a shared label for both node-level metrics and DB service-level metrics. This still causes me confusion, in part because it is unclear how notifications will act if they are routed.
As you can see below, the environment filters for the Home Dashboard and the Nodes Summary Dashboard both have different conditions.
@Roma_Novikov
Please, if you could help me with the environmental Label. As we move forward with further alerting configurations, the label confusion has actually struck us.
Hello @Naresh9999,
since Labels are not mentioned anywhere as being independent of nodes and services. It can cause confusion if we use environment=PROD as a shared label for both node-level metrics and DB service-level metrics. This still causes me confusion, in part because it is unclear how notifications will act if they are routed.
So, I do not understand what is a confusion here. The label “environment=PROD” is indicative of the node and the database on it belongs to “prod environment”. Your alerts should be having the appropriate checks pointing at either of them!
As you can see below, the environment filters for the Home Dashboard and the Nodes Summary Dashboard both have different conditions.
Did you mean OS Overview is having additional filters (region/type) while Home Dashboard is not?
Can you exactly show us “what alert configuration” has confusion due to “label”?
Thanks,
K
@kedarpercona, If we specify environment=PROD then it only works for Database service-related alerts (MariaDB replication, down…etc), but not for node-related alerts like CPU, memory…etc. Below are pmm commands that we have configured.
- pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels=“SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA, APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES” --region=“AWS-DC” 192.10.1.1 generic nvmmm101
- pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP
@kedarpercona I only need a clarification that the commands should work well if I use the -environment=PROD label for both the Node command and the DB service command like below.
Node Command:
- pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels=“SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA, APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES,environment=PROD” --region=“AWS-DC” 192.10.1.1 generic nvmmm101
DB Service Command:
2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP
Hi Naresh,
So the second last comment of yours say:
If we specify environment=PROD then it only works for Database service-related alerts (MariaDB replication, down…etc), but not for node-related alerts like CPU, memory…etc. Below are pmm commands that we have configured.
Correct, it will not have the label because you have not configured it. As we already saw that once we have them set as custom-label it should work.
The best thing you can do is configure it and generate a dummy alert for both OS and Database checks to verify if that works.
Then in the last comment you have this comment:
@kedarpercona I only need a clarification that the commands should work well if I use the -environment=PROD label for both the Node command and the DB service command like below.
About this, we both tested and confirmed that it works.
And hence to understand your question, may be to reproduce what you’re seeing, I had a question:
Can you exactly show us “what alert configuration” has confusion due to “label”?
So as a next step, I’d suggest you to configure both OS and Database with appropriate “environment=PROD” and then generate an alert. Test out your scenario and let us know.
Thanks,
K
@kedarpercona Thanks for the details.
For both Node and Database PMM commands, I’ll try to set the common environment=PROD Label, and then I’ll test the alerting. I’ll set up the commands listed below on my database servers and let you know when I’m done.
Node Command:
- pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels=“SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA,APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES,environment=PROD” --region=“AWS-DC” 192.10.1.1 generic nvmmm101
DB Service Command:
2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP
@kedarpercona After the above commands execution, I have configured notification policy with the label environment=PROD, and alerts for Node metrics are not functioning, but they are functional for DB metrics.
@kedarpercona After changing and creating a notification policy like below, it’s working fine for both the Node and DB services.
Commands:
- pmm-admin config --server-insecure-tls --server-url=http://admin:XXXXXXXXX@192.168.1.1 --custom-labels=“SERVICE_RUNNING=MariaDB, DB_SERVICE_TYPE=REPLICA,APP_TYPE=ECOM,BACKUP_JOB=YES,IsSlave=YES,ENV=PROD” --region=“AWS-DC” 192.10.1.1 generic nvmmm101
DB Service Command:
2. pmm-admin add mysql --server-insecure-tls --username=‘pmm’ --password=‘XXXXXXXXX’ --host=‘192.10.1.1’ --port=3306 --service-name=ECOM --disable-tablestats-limit=5000 --query-source=none –environment=PROD --replication-set=ECOM-REP
@kedarpercona The alerts are currently functioning properly for both DB service-related and Node-related alerts, as shown in the screenshot.
Thanks for all the help.
1 Like
