Strange behavior of visualization in Grafana.

[Stateros]

Hello folks. Merry Christmas and Happy New year.
I have a strange problem which I faced few days ago.
I have latest pmm-server 2.2.0 (docker) and pmm-client 2.2.0

On the screen you can see lot of duplicate metrics of “lag” metric. The same situation with all other mysql and linux metrics.
Data looks correct, but very confused, specially, when you need to make some investigation.

And one more thing: for each host I have 3 mysql_exporter like:
mysqld_exporter_agent_id_2d43b81d-6dea-4aad-8628-eab969a87056_hr-5s (1/1 up)[/h] [h=2]mysqld_exporter_agent_id_2d43b81d-6dea-4aad-8628-eab969a87056_lr-1m0s (1/1 up)[/h] [h=2]mysqld_exporter_agent_id_2d43b81d-6dea-4aad-8628-eab969a87056_mr-5s (1/1 up) is it valid situation?

What can I check, change for investigate this strange issue?

[Stateros]

roma.novikov alexey.palazhchenko
Hello, I checked deeply this issue, but stacked.
So the main problem is that at some point current agent is dead, and there is a new agent with different id. How can I understand the reason of failing PMM Client?
Here is a screenshot from the prometheus. Node_name, cluster and other labels are the same. They are controlled by automation:

[Stateros]

Unfortunately I can’t upload good quality screenshot, so here it in high resolution: [url]Dropbox - File Deleted