PMM Gravana does not show all data

News:
All servers: good server

Gaps looks like network issues:
prometheus very depends on network delays.
if communication between exporter<->prometheus near 1s you can receive such issues.

Yep, network. Yesterday reinstall pmm server on new host and newer docker, and gaps has gone.

But problem with bad server stayed. Funny, but bad server shows good statistics for some time before reinstalling, but not long, less then hour, all metrics, all values. If you look on last picture, you can see max connection metric not exist, after that exist.
Really, i try to use Developer Tool to “open graphs for good and bad hosts and check that “mysql-overview” dashboard template always the same for all hosts (and has mysql_global_status_connections line in description)”. May be i don’t understand something, but i don’t see differences. If you can show some screen shot about how did you do that, it would help me to check that i do it in right way.

I had check Prometheus again. Looks like we watch wrong parameter(mysql_global_status_connections). Look on screen shot(third server were added, and its work fine):

can you check exporter output?

curl https://login:password&#64;pmm-client:42002/metrics-hr --insecure | grep mysql_global_variables_max_connections

root@2f744998d466:/opt# curl https://root:somepass@good_server:42002/metrics-hr --insecure | grep mysql_global_variables_max_connections
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 102k 100 102k 0 0 1436k 0 --:–:-- --:–:-- --:–:-- 1446k

root@2f744998d466:/opt# curl https://root:somepass@bad_server:42002/metrics-hr --insecure | grep mysql_global_variables_max_connections
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 103k 100 103k 0 0 1205k 0 --:–:-- --:–:-- --:–:-- 1215k

Oh, Aleksey, sorry, right command

curl https://login:password&#64;pmm-client:42002/metrics-lr --insecure | grep mysql_global_variables_max_connections

root@2f744998d466:/opt# curl https://root:somepass@good_server:42002/metrics-lr --insecure | grep mysql_global_variables_max_connections
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0# HELP mysql_global_variables_max_connections Generic gauge metric from SHOW GLOBAL VARIABLES.

TYPE mysql_global_variables_max_connections gauge

mysql_global_variables_max_connections 151
100 5742k 100 5742k 0 0 2845k 0 0:00:02 0:00:02 --:–:-- 2846k

root@2f744998d466:/opt# curl https://root:somepass@bad_server::42002/metrics-lr --insecure | grep mysql_global_variables_max_connections
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 31.4M 0 1056 0 0 130 0 70:24:32 0:00:08 70:24:24 215# HELP mysql_global_variables_max_connections Generic gauge metric from SHOW GLOBAL VARIABLES.

TYPE mysql_global_variables_max_connections gauge

mysql_global_variables_max_connections 800
100 31.4M 100 31.4M 0 0 3487k 0 0:00:09 0:00:09 --:–:-- 8014k

So, am I understand correctly,
values exists in exporter output and in prometheus database?

May be i confuse you with previous picture, because there is 3 servers but we talk about 2. mysql_global_variables_max_connections does not showed on picture for bad server, there is only two good servers.
I.e. mysql_global_variables_max_connections values exists in exporter output for bad and good server, values exists in prometheus ONLY for good server(s).
I just cant exclude second good server from picture. But if you compare mysql_global_status_connections and

Hi,

can you open targrets page? [url]http://PMM-SERVER-IP/prometheus/targets[/url]
and check status (UP or DOWN) of target.
also can you copy-paste “Last Scrape” value, wait a few seconds, refresh page and copy new “Last Scrape” values again (3-5 values needed)

instance “mysql.db” is a bad server. Others good.

Last Scrape:
1m4.203s ago
11.794s ago
19.312s ago
28.762s ago
37.713s ago
44.95s ago
51.95s ago
58.661s ago

do you see any errors in /var/log/prometheus.log ?
can you share it?

[URL]DepositFiles

OR

[url]http://rg.to/file/78fa59c303b761b7832ab38facd92001/prometheus.log.html[/url]

Looks like you have high load.
can you change config?
try set section

docker exec -it pmm-server vi /opt/prometheus/prometheus.yml
docker restart pmm-server

after that please monitor prometheus targets page [url]http://pmm-server-ip/prometheus/targets[/url]
/metrics-mr and /metrics-lr targets should became green

I changed it & restart the pmm-server.
After that:
It did not work abour 30 mins. In 30 mins:

And after that i restart pmm again. It is not work for now:

looks like prometheus become broken due to some typo in the config
you can just recreate container (without data lose)
please don’t remove pmm-data container, delete only pmm-server

docker stop pmm-server
docker rm pmm-server
docker run -d \
-p 80:80 \
--volumes-from pmm-data \
--name pmm-server \
--restart always \
percona/pmm-server:1.1.1 

Done. And we have a progress - now all servers have the same problem :slight_smile:

lets try again to modify for mysql-lr section, but now with config backup and without vim :))

please run the following commands, wait for several minutest and check targets again.

docker exec -it pmm-server cp /opt/prometheus/prometheus.yml /opt/prometheus/prometheus.yml_BAK
docker exec -it pmm-server sed -i -e '/mysql-lr/{n;;; s/5s/14s/}' /opt/prometheus/prometheus.yml
docker restart pmm-server