PMM Gravana does not show all data

Oh, Aleksey, sorry, right command

curl https://login:password@pmm-client:42002/metrics-lr --insecure | grep mysql_global_variables_max_connections

root@2f744998d466:/opt# curl https://root:somepass@good_server:42002/metrics-lr --insecure | grep mysql_global_variables_max_connections
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0# HELP mysql_global_variables_max_connections Generic gauge metric from SHOW GLOBAL VARIABLES.

TYPE mysql_global_variables_max_connections gauge

mysql_global_variables_max_connections 151
100 5742k 100 5742k 0 0 2845k 0 0:00:02 0:00:02 --:–:-- 2846k

root@2f744998d466:/opt# curl https://root:somepass@bad_server::42002/metrics-lr --insecure | grep mysql_global_variables_max_connections
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 31.4M 0 1056 0 0 130 0 70:24:32 0:00:08 70:24:24 215# HELP mysql_global_variables_max_connections Generic gauge metric from SHOW GLOBAL VARIABLES.

TYPE mysql_global_variables_max_connections gauge

mysql_global_variables_max_connections 800
100 31.4M 100 31.4M 0 0 3487k 0 0:00:09 0:00:09 --:–:-- 8014k

So, am I understand correctly,
values exists in exporter output and in prometheus database?

May be i confuse you with previous picture, because there is 3 servers but we talk about 2. mysql_global_variables_max_connections does not showed on picture for bad server, there is only two good servers.
I.e. mysql_global_variables_max_connections values exists in exporter output for bad and good server, values exists in prometheus ONLY for good server(s).
I just cant exclude second good server from picture. But if you compare mysql_global_status_connections and

Hi,

can you open targrets page? [url]http://PMM-SERVER-IP/prometheus/targets[/url]
and check status (UP or DOWN) of target.
also can you copy-paste “Last Scrape” value, wait a few seconds, refresh page and copy new “Last Scrape” values again (3-5 values needed)

instance “mysql.db” is a bad server. Others good.

Last Scrape:
1m4.203s ago
11.794s ago
19.312s ago
28.762s ago
37.713s ago
44.95s ago
51.95s ago
58.661s ago

do you see any errors in /var/log/prometheus.log ?
can you share it?

[URL]DepositFiles

OR

[url]http://rg.to/file/78fa59c303b761b7832ab38facd92001/prometheus.log.html[/url]

Looks like you have high load.
can you change config?
try set section

docker exec -it pmm-server vi /opt/prometheus/prometheus.yml
docker restart pmm-server

after that please monitor prometheus targets page [url]http://pmm-server-ip/prometheus/targets[/url]
/metrics-mr and /metrics-lr targets should became green

I changed it & restart the pmm-server.
After that:
It did not work abour 30 mins. In 30 mins:

And after that i restart pmm again. It is not work for now:

looks like prometheus become broken due to some typo in the config
you can just recreate container (without data lose)
please don’t remove pmm-data container, delete only pmm-server

docker stop pmm-server
docker rm pmm-server
docker run -d \
-p 80:80 \
--volumes-from pmm-data \
--name pmm-server \
--restart always \
percona/pmm-server:1.1.1 

Done. And we have a progress - now all servers have the same problem :slight_smile:

lets try again to modify for mysql-lr section, but now with config backup and without vim :))

please run the following commands, wait for several minutest and check targets again.

docker exec -it pmm-server cp /opt/prometheus/prometheus.yml /opt/prometheus/prometheus.yml_BAK
docker exec -it pmm-server sed -i -e '/mysql-lr/{n;;; s/5s/14s/}' /opt/prometheus/prometheus.yml
docker restart pmm-server

Same result

/metrics-lr targets still in “unknown” state?
am I understand correctly?

When i wrote #38, result was like in #36, it is about 1h after pmm restart, i.e. all servers in UNKNOWN state.
Now:

And there is still gaps exists on graphics(gaps on pmm-server own statistics):

lets revert config from backup and restart


docker exec -it pmm-server cp /opt/prometheus/prometheus.yml_BAK /opt/prometheus/prometheus.yml docker restart pmm-server 

Good: all good servers have all metrix
Bad: We have gaps like on pics before, bad server do not have metrics we talk about
Question: is it OK that PMM after restart docker container start to show data after 1h? Is start can take so lot of time?..