Now that Alertmanager integration is proceeding a question about custom labels:
We originally connected our agents to the pmm-server with custom labels for environment and database name. These labels don’t appear on host metrics so the Alertmanager (either integrated or open source) is unable to match if you want to, for instance, only alert on something with env=prod. (at least for say, low memory.)
Is it now possible or in the works, since Alertmanager is becoming integrated, that custom labels and tags will work for both Database metrics as well as Host level metrics?
That’s the issue we’re running into, I believe they are correctly specified but they don’t show up on host metrics. So we ran these commands to setup the agents several months ago: sudo yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm sudo yum install pmm2-client pmm-admin config --server-insecure-tls --server-url=https://myPMMserver pmm-admin add mysql --username=user --password=pw --custom-labels=“environment=DEV,database=pricing”
However, now when I check the metrics explorer, only MySQL metrics have the environment and database custom labels. Host metric labels (like I’m looking at “node_filesystem_free_bytes” right now) do not have custom labels attached to them. I can provide screenshots if you need. I asked a similar question a few months ago and I was told to open a JIRA ticket: Question about PMM setup and Prometheus labels
I got it as you managed somehow to add custom labels for nodes.
You are right pmm-admin config has no flag custom-labels and there are no easy way to add custom labels while the ticket hasn’t done.
Here is a workaround by adding labels directly to PMM DB. But you have to be careful by using next commands.
docker exec -it <PMM Docker Name> psql -U postgres -d pmm-managed -c "update nodes set custom_labels='{\"environment\":\"DEV\",\"database\":\"pricing\"}' where node_name='<Your Node Name>'"
