System Time
NTP Server (0.pool.ntp.org) | unable to get ntp time: %!s()
PMM Server | 2018-12-18 14:38:14 +0000 GMT
PMM Client | 2018-12-18 09:43:08 -0500 EST
PMM Client to PMM Server Time Drift | 294s
Time is out of sync. Please make sure the server time is correct to see the metrics.
This can be related to some problems. Pls check ntp on your Client and Server
[baasp@server ~]$ sudo pmm-admin check-network –-no-emoji
PMM Network Status
Server Address | XXX.XXX.XXX.XXX
Client Address | XXX.XXX.XXX.XXX
System Time
NTP Server (0.pool.ntp.org) | unable to get ntp time: %!s()
PMM Server | 2019-01-08 18:28:09 +0000 GMT
PMM Client | 2019-01-08 13:28:21 -0500 EST
PMM Client to PMM Server Time Drift | OK
Connection: Client → Server
SERVER SERVICE STATUS
Consul API OK
Prometheus API OK
Query Analytics API OK
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
linux:metrics mysql80_lcormysqlp01 XXX.XXX.XXX.XXX:42000 OK YES YES
mysql:metrics mysql80_lcormysqlp01 XXX.XXX.XXX.XXX:42002 DOWN YES YES
When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify).
If it’s running, check out the logs /var/log/pmm-*.log
When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.
I have been able to see that it is the High Resolution MySQL data that is not being gathered.
I need to get a port opened before I can see the data stream directly in a browser now.
In addition the client is on 1.16.0 and the server is running 1.17.0. This too is being addressed. But I don’t believe it is a factor as the issue has persisted for months.
HELP mysql_info_schema_innodb_metrics_adaptive_hash_index_adaptive_hash_searches_btree_total Number of searches using B-tree on an index search
.
.
.
HELP mysql_info_schema_innodb_metrics_buffer_buffer_pool_read_ahead_evicted_total Read-ahead pages evicted without being accessed (innodb_buffer_pool_read_ahead_evicted)
It would appear to be gathering the data just not processing it:(
Thanks in advance for reviewing and advising on this issue.
[me@lcormysqld01 ~]$ sudo pmm-admin check-network –-no-emoji
PMM Network Status
Server Address | SERVER_IP
Client Address | CLIENT_IP
System Time
NTP Server (0.pool.ntp.org) | unable to get ntp time: %!s()
PMM Server | 2019-01-22 13:24:48 +0000 GMT
PMM Client | 2019-01-22 08:26:25 -0500 EST
PMM Client to PMM Server Time Drift | 97s
Time is out of sync. Please make sure the server time is correct to see the metrics.
Connection: Client → Server
SERVER SERVICE STATUS
Consul API OK
Prometheus API OK
Query Analytics API OK
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
linux:metrics mysql80_lcormysqld01 CLIENT_IP:42000 OK YES YES
mysql:metrics mysql80_lcormysqld01 CLIENT_IP:42002 DOWN YES YES
When an endpoint is down it may indicate that the corresponding service is stopped (run ‘pmm-admin list’ to verify).
If it’s running, check out the logs /var/log/pmm-*.log
When all endpoints are down but ‘pmm-admin list’ shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.
This problem apparently caused to pluggin:
mysql> SHOW VARIABLES LIKE ‘%dictionary%’;
±----------------------------------±------+
| Variable_name | Value |
±----------------------------------±------+
| validate_password.dictionary_file | |
±----------------------------------±------+
1 row in set (0.00 sec)
To resolve this problem, I disabled in the mysql:
mysql> SELECT @@plugin_dir;
±-------------------------+
| @@plugin_dir |
±-------------------------+
| /usr/lib64/mysql/plugin/ |
±-------------------------+
1 row in set (0.00 sec)