Bug in ProxySQL dashboard

[mosi0815]

I think the bugfix included in 1.1.4 was not 100%. I still have some graphs with malfunctions.
At first the server status is always showing the same state. On my test system it alwas says 1, 1, 1 and on production 1, 1 ,2. Have a look at the screenshots. I deliberately shut down one server.
The hostgroup size is also wrong. But I think those graphs use the same datasource.
The second issue is the latency graph. It used to work but isn’t anymore. -> See screenshot

[Mykola]

Hi mosi0815,

can you share versions of pmm-client, PMM Server, ProxySQL and OS?

[mosi0815]

Hi!

OS: Ubuntu 16.04
ProxySQL: 1.3.6-1.1.xenial
PMM Server: 1.1.4 on Docker
PMM Client: 1.1.4-1.xenial

[Mykola]

can you share the output of the following command?

curl -s http://pmm-client-ip:42004/metrics | grep latency

[mosi0815]

I had to change your command a little bit:

curl -s --insecure https://ip:42004/metrics | grep latency

Here’s the output:

# HELP proxysql_connection_pool_latency_us Connection pool usage statistic.
# TYPE proxysql_connection_pool_latency_us gauge
proxysql_connection_pool_latency_us{endpoint="xtradb1:3306",hostgroup="10"} 578
proxysql_connection_pool_latency_us{endpoint="xtradb1:3306",hostgroup="20"} 578
proxysql_connection_pool_latency_us{endpoint="xtradb2:3306",hostgroup="10"} 692
proxysql_connection_pool_latency_us{endpoint="xtradb2:3306",hostgroup="20"} 692
proxysql_connection_pool_latency_us{endpoint="xtradb3:3306",hostgroup="10"} 584
proxysql_connection_pool_latency_us{endpoint="xtradb3:3306",hostgroup="20"} 584

[mosi0815]

I just upgraded to 1.1.5.
The latency graph is back but the node status is still not right. (See screenshots)

[Michael_Coburn]

Hi mosi0815, what specifically isn’t right about your node status?
Could you share:

curl -s --insecure https://ip:42004/metrics | grep "proxysql_connection_pool_status"

output of ProxySQL admin information:

select * from mysql_servers;

[mosi0815]

Please look at the screenshots in my post before. Grafana says 1,1,3 but it’s 1,1,1.

Here is what you wanted:

+--------------+----------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | xtradb1 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | |
| 10 | xtradb2 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | |
| 10 | xtradb3 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | xtradb3 | 3306 | OFFLINE_SOFT | 1000 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | xtradb2 | 3306 | OFFLINE_SOFT | 1000 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | xtradb1 | 3306 | ONLINE | 1000 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+

# HELP proxysql_connection_pool_status Connection pool usage statistic.
# TYPE proxysql_connection_pool_status gauge
proxysql_connection_pool_status{endpoint="xtradb1:3306",hostgroup="10"} 1
proxysql_connection_pool_status{endpoint="xtradb1:3306",hostgroup="20"} 1
proxysql_connection_pool_status{endpoint="xtradb2:3306",hostgroup="10"} 3
proxysql_connection_pool_status{endpoint="xtradb2:3306",hostgroup="20"} 3
proxysql_connection_pool_status{endpoint="xtradb3:3306",hostgroup="10"} 1
proxysql_connection_pool_status{endpoint="xtradb3:3306",hostgroup="20"} 3

As you can see the outputs don’t match.

[Michael_Coburn]

mosi0815 , thanks for sharing the output. Agreed this is suspicious, and we (PMM Dev team) are discussing this internally and attempting to reproduce. We’ll update this thread shortly.

[Michael_Coburn]

Hi mosi0815 , one of our Engineers has requested the full /metrics output. You may want to DM or email it to me if you prefer michael.coburn@percona.com

[mosi0815]

I emailed it to you because the PM feature somehow doesn’t work.

[Michael_Coburn]

Hi mosi0815 , my apologies for the delay on this topic. Our Developers want to confirm that you are collecting data at the same point in time, that there isn’t any potential for drift between what ProxySQL vs proxysql_exporter reports. Can you please run the commands below, and replace IP with the address of the instance where proxysql_exporter is running:

date
curl -s --insecure https://IP:42004/metrics | grep proxysql_connection_pool_status
date
mysql -uadmin -padmin -S/tmp/proxysql_admin.sock -e "select * from stats_mysql_connection_pool ORDER BY hostgroup, srv_host, srv_port;"
date
mysql -uadmin -padmin -S/tmp/proxysql_admin.sock -e "select * from runtime_mysql_servers"

Thanks mosi0815 ,

[mosi0815]

Hi,

on Sunday I did my usual reboot cycle and the monitor worked correctly. Now I tried to reproduce the issue on our test system to collect the logs you requested. No chance. Everything works fine now. The question is why. The only updated which got in was a proxysql update.

[Michael_Coburn]

mosi0815 , well on one hand I am delighted that your environment is again working! But on the other hand I fear that your experience may be the first of several to occur related to discrepancies between status in ProxySQL vs exporter. In my own exploratory testing I was using latest releases:
[LIST]
[]ProxySQL 1.3.7 (Percona build)
[]proxysql_exporter from pmm-client 1.1.5
[/LIST] I’m going to close this thread, but if you happen to see this issue reoccur please start a new thread and we’ll take action. Thanks again for your cooperation while we diagnosed this!