Not the answer you need?
Register and ask your own question!

Lost data when deploying PMM into vmware machines

seanlookseanlook EntrantCurrent User Role Beginner
This is what my grafana dashboard looks like:
fetch?filedataid=637&type=thumb

I deployed it as docker image first, the uncontinuously lines came out and I thought it may be the heavy load.
Then I tried to deploy it to ESXi virtual machine and increased the CPUs&Mem, the same happened.

At the end, I run in VirtualBox, everything is perfect, the graph seems continuously.

Then I realized that my docker host is a ESXi vm. So I get to this conclusion: PMM-Server does not work well with vmware machines.

Could someone help me what have I miss ?

Comments

  • MykolaMykola Percona Percona Staff Role
    Hi seanlook,

    Prometheus fetches data every 1s from all mysql hosts, so it requires very good network performance and very fast VM reaction.
    if ESXi has any network/CPU/memory delays Prometheus can ignores monitoring results.

    please check state of Targets - http://PMM-SERVER-IP/prometheus/targets, all of them should be "UP".
    please try to refresh this page at least 3-5 times with several seconds delay between refresh.
  • seanlookseanlook Entrant Current User Role Beginner
    Mykola wrote: »
    Hi seanlook,

    Prometheus fetches data every 1s from all mysql hosts, so it requires very good network performance and very fast VM reaction.
    if ESXi has any network/CPU/memory delays Prometheus can ignores monitoring results.

    please check state of Targets - http://PMM-SERVER-IP/prometheus/targets, all of them should be "UP".
    please try to refresh this page at least 3-5 times with several seconds delay between refresh.


    Thanks for your reply. I see lots of `UNKNOW` state.
    I can tell pmm-server and client network traffic status is not the proplem. MySQL instance node load is not heavy.

    I tried to visit client port:
    https://pmm-client-ip:42000/metrics -- fast
    https://pmm-client-ip:42002/metrics-mr -- very slow

    It seems say that all mysqld_exporter response slowly at the same time.
  • MykolaMykola Percona Percona Staff Role
    /metrics-mr fetch time should be less than 1 seconds

    can you show output of the following command?
      wget https://PMM-CLIENT-IP:42002/metrics-mr --no-check-certificate
    
    I am wondering response size and time consumed to fetch.
  • seanlookseanlook Entrant Current User Role Beginner
    As attached.

    Linux metrics should be independent with MySQL metrics. But both OS panel and MySQL panel data lost.

    I've changed the scrape_interval=10s, scrape_timeout=5s, still uncontinuously. Within virtualbox, nothing lost.
  • MykolaMykola Percona Percona Staff Role
    can you share /var/log/prometheus.log ?
This discussion has been closed.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.