Lost data when deploying PMM into vmware machines

Percona Monitoring and Management (PMM) PMM 1.x
1

This is what my grafana dashboard looks like:

I deployed it as docker image first, the uncontinuously lines came out and I thought it may be the heavy load.
Then I tried to deploy it to ESXi virtual machine and increased the CPUs&Mem, the same happened.

At the end, I run in VirtualBox, everything is perfect, the graph seems continuously.

Then I realized that my docker host is a ESXi vm. So I get to this conclusion: PMM-Server does not work well with vmware machines.

Could someone help me what have I miss ?

photoid=47710
photoid=477101824×894 97.7 KB

2

Hi seanlook,

Prometheus fetches data every 1s from all mysql hosts, so it requires very good network performance and very fast VM reaction.
if ESXi has any network/CPU/memory delays Prometheus can ignores monitoring results.

please check state of Targets - [URL]http://PMM-SERVER-IP/prometheus/targets[/URL], all of them should be “UP”.
please try to refresh this page at least 3-5 times with several seconds delay between refresh.

3

Thanks for your reply. I see lots of UNKNOW state.
I can tell pmm-server and client network traffic status is not the proplem. MySQL instance node load is not heavy.

I tried to visit client port:
https://pmm-client-ip:42000/metrics – fast
https://pmm-client-ip:42002/metrics-mr – very slow

It seems say that all mysqld_exporter response slowly at the same time.

4

/metrics-mr fetch time should be less than 1 seconds

can you show output of the following command?


wget https://PMM-CLIENT-IP:42002/metrics-mr --no-check-certificate

I am wondering response size and time consumed to fetch.

5

As attached.

Linux metrics should be independent with MySQL metrics. But both OS panel and MySQL panel data lost.

I’ve changed the scrape_interval=10s, scrape_timeout=5s, still uncontinuously. Within virtualbox, nothing lost.

photoid=47720
photoid=477201014×284 18 KB

photoid=47721
photoid=477211000×463 17.4 KB

6

can you share /var/log/prometheus.log ?