Not the answer you need?
Register and ask your own question!

PMM Gravana does not show all data

2

Comments

  • MykolaMykola Percona Percona Staff Role
    Looks like you have high load.
    can you change config?
    try set section
    docker exec -it pmm-server vi /opt/prometheus/prometheus.yml
    docker restart pmm-server
    

    after that please monitor prometheus targets page http://pmm-server-ip/prometheus/targets
    /metrics-mr and /metrics-lr targets should became green
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    I changed it & restart the pmm-server.
    After that:
    It did not work abour 30 mins. In 30 mins:

    And after that i restart pmm again. It is not work for now:
    1.png 55.8K
    2.png 138K
    3.png 96.8K
    4.png 128.6K
    6.png 177.3K
  • MykolaMykola Percona Percona Staff Role
    looks like prometheus become broken due to some typo in the config
    you can just recreate container (without data lose)
    please don't remove pmm-data container, delete only pmm-server
    docker stop pmm-server
    docker rm pmm-server
    docker run -d \
       -p 80:80 \
       --volumes-from pmm-data \
       --name pmm-server \
       --restart always \
       percona/pmm-server:1.1.1
    
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Done. And we have a progress - now all servers have the same problem :)
    7.png 131K
  • MykolaMykola Percona Percona Staff Role
    lets try again to modify for mysql-lr section, but now with config backup and without vim :))

    please run the following commands, wait for several minutest and check targets again.
    docker exec -it pmm-server cp /opt/prometheus/prometheus.yml /opt/prometheus/prometheus.yml_BAK
    docker exec -it pmm-server sed -i -e '/mysql-lr/{n;;; s/5s/14s/}' /opt/prometheus/prometheus.yml
    docker restart pmm-server
    
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Same result
  • MykolaMykola Percona Percona Staff Role
    /metrics-lr targets still in "unknown" state?
    am I understand correctly?
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    When i wrote #38, result was like in #36, it is about 1h after pmm restart, i.e. all servers in UNKNOWN state.
    Now:
    8.png 134.6K
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    And there is still gaps exists on graphics(gaps on pmm-server own statistics):
    9.png 94.4K
  • MykolaMykola Percona Percona Staff Role
    lets revert config from backup and restart
     docker exec -it pmm-server cp /opt/prometheus/prometheus.yml_BAK /opt/prometheus/prometheus.yml docker restart pmm-server
    
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Good: all good servers have all metrix
    Bad: We have gaps like on pics before, bad server do not have metrics we talk about
    Question: is it OK that PMM after restart docker container start to show data after 1h? Is start can take so lot of time?...
  • MykolaMykola Percona Percona Staff Role
    Question: is it OK that PMM after restart docker container start to show data after 1h? Is start can take so lot of time?

    no, it is something strange, can you synchronize time on your docker server?
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Hm. Time on pmm-server container delay from db hosts for 3 sec, from my host for 2 sec. Docker server and docker container have the same time.
  • MykolaMykola Percona Percona Staff Role
    Is Grafana only affected or you don't see graphs also in prometheus?

    http://PMM-SERVER-IP/prometheus/graph?g0.range_input=1h&g0.expr=node_load1&g0.tab=0
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    :

    Screens from grafana and prometheus
    12.png 52.2K
    11.png 105.4K
    12.png 52.2K
    11.png 105.4K
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    But it is Linux metrics, it always exist for bad and good servers.
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Some other news: we have 3 graphs, all different - bad server have no some metrics, other have all metrics, other have gaps in some metrics:
    1.png 173.8K
    2.png 195.4K
    3.png 202.4K
    2.png 195.4K
    1.png 173.8K
  • MykolaMykola Percona Percona Staff Role
    But it is Linux metrics, it always exist for bad and good servers.
    oh, can you check prometheus graph for mysql_global_variables_max_connections ?
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Bad server is missing. And i have feeling like we walk by the circle.
  • MykolaMykola Percona Percona Staff Role
    How many tables do you have?

    I think we have very long or wrong response from mysqld_exporter.

    can you measure response time with the following command?
    wget https://192.168.200.206:42002/metrics-lr --no-check-certificate
    
    wget output should show start time in the first line and end time in the last
  • MykolaMykola Percona Percona Staff Role
    After some internal discussion, my colleague created feature request for mysqld_exporter - https://github.com/prometheus/mysqld...ter/issues/188
    feel free to +1 it

    also we created two internal tickets
    https://jira.percona.com/browse/PMM-663
    https://jira.percona.com/browse/PMM-664
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Done :)

    And ones more. I checked memory usage by mysqld_exporter with top.
    Bad server:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    18853 root 20 0 2441360 2,054g 4680 S 280,4 13,2 19:02.85 mysqld_exporter

    Good servers:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    26945 root 20 0 766688 117948 3560 S 7,0 0,7 2550:20 mysqld_exporter

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    15257 root 20 0 316880 144584 3168 S 23,6 7,7 2103:36 mysqld_exporter

    On bad server RES=2,054g (10 min after pmm-admin restart)
    On good servers RES=117948(144584) (few days after restart)
  • MykolaMykola Percona Percona Staff Role
    How many tables do you have?

    can you show output of the following command?
     wget https://192.168.200.206:42002/metrics-lr --no-check-certificate
    
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    Tables count in bad db = 3639, good db`s = 1736 each
    About output - command downloaded file 250M size. If you want i can share it somewhere.

    OUTPUT FROM BAD SERVER:

    --2017-03-16 17:04:10-- https://bad_ip:42002/metrics-lr
    Подключение к bad_ip:42002... соединение установлено.
    ПРЕДУПРЕЖДЕНИЕ: невозможно проверить сертификат bad_ip, выпущенный «O=PMM Client»:
    Невозможно локально проверить подлинность запрашивающего.
    HTTP-запрос отправлен. Ожидание ответа… 200 OK
    Длина: 261433316 (249M) [text/plain]
    Сохранение в: «metrics-lr»

    metrics-lr 100%[================================================== ==========================================>] 249,32M 5,15MB/s in 57s

    2017-03-16 17:06:04 (4,34 MB/s) - «metrics-lr» сохранён [261433316/261433316]

    OUTPUT FROM GOOD SERVER:

    --2017-03-16 17:48:43-- https://good_ip:42002/metrics-lr
    Подключение к good_ip:42002... соединение установлено.
    ПРЕДУПРЕЖДЕНИЕ: невозможно проверить сертификат good_ip, выпущенный «O=PMM Client»:
    Невозможно локально проверить подлинность запрашивающего.
    HTTP-запрос отправлен. Ожидание ответа… 200 OK
    Длина: 8876739 (8,5M) [text/plain]
    Сохранение в: «metrics-lr.1»

    metrics-lr.1 100%[============================================================================================>] 8,46M 3,30MB/s in 2,6s

    2017-03-16 17:48:50 (3,30 MB/s) - «metrics-lr.1» сохранён [8876739/8876739]
  • MykolaMykola Percona Percona Staff Role
    wow!
    can you share this archive??
  • MykolaMykola Percona Percona Staff Role
    I found that table_statistics is the biggest part of all output (235Mb of 249M).
    grep mysql_info_schema_table_statistics_rows_ metrics-lr | wc -c
    246603314
    

    can you disable it?
    pmm-admin remove mysql:metrics
    pmm-admin add mysql:metrics --disable-tablestats
    
    unfortunately, two graphs on "MySQL Table Statistics" dashboard became empty for bad host - "Top Tables by Rows Read" "Top Tables by Rows Read"

    finally, after this change everything should work fine :)
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    MySQL Table Statistics show no data on all graphics with --disable-tablestats
  • aleksey.filippovaleksey.filippov Contributor Current User Role Beginner
    May be i can use --disable-tablestats-limit with some count for bad server to collect as more stats as possible?
This discussion has been closed.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.