Not the answer you need?
Register and ask your own question!

Bug in ProxySQL dashboard

mosi0815mosi0815 ContributorInactive User Role Beginner
I think the bugfix included in 1.1.4 was not 100%. I still have some graphs with malfunctions.
At first the server status is always showing the same state. On my test system it alwas says 1, 1, 1 and on production 1, 1 ,2. Have a look at the screenshots. I deliberately shut down one server.
The hostgroup size is also wrong. But I think those graphs use the same datasource.
The second issue is the latency graph. It used to work but isn't anymore. -> See screenshot

Comments

  • MykolaMykola Percona Percona Staff Role
    Hi mosi0815,

    can you share versions of pmm-client, PMM Server, ProxySQL and OS?
  • mosi0815mosi0815 Contributor Inactive User Role Beginner
    Hi!

    OS: Ubuntu 16.04
    ProxySQL: 1.3.6-1.1.xenial
    PMM Server: 1.1.4 on Docker
    PMM Client: 1.1.4-1.xenial
  • MykolaMykola Percona Percona Staff Role
    can you share the output of the following command?
    curl -s http://pmm-client-ip:42004/metrics | grep latency
    
  • mosi0815mosi0815 Contributor Inactive User Role Beginner
    I had to change your command a little bit:
    curl -s --insecure https://ip:42004/metrics | grep latency
    

    Here's the output:
    # HELP proxysql_connection_pool_latency_us Connection pool usage statistic.
    # TYPE proxysql_connection_pool_latency_us gauge
    proxysql_connection_pool_latency_us{endpoint="xtradb1:3306",hostgroup="10"} 578
    proxysql_connection_pool_latency_us{endpoint="xtradb1:3306",hostgroup="20"} 578
    proxysql_connection_pool_latency_us{endpoint="xtradb2:3306",hostgroup="10"} 692
    proxysql_connection_pool_latency_us{endpoint="xtradb2:3306",hostgroup="20"} 692
    proxysql_connection_pool_latency_us{endpoint="xtradb3:3306",hostgroup="10"} 584
    proxysql_connection_pool_latency_us{endpoint="xtradb3:3306",hostgroup="20"} 584
    
  • mosi0815mosi0815 Contributor Inactive User Role Beginner
    I just upgraded to 1.1.5.
    The latency graph is back but the node status is still not right. (See screenshots)
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi mosi0815, what specifically isn't right about your node status?
    Could you share:
    curl -s --insecure https://ip:42004/metrics | grep "proxysql_connection_pool_status"
    

    output of ProxySQL admin information:
    select * from mysql_servers;
    
  • mosi0815mosi0815 Contributor Inactive User Role Beginner
    Please look at the screenshots in my post before. Grafana says 1,1,3 but it's 1,1,1.

    Here is what you wanted:
    +--------------+----------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
    | hostgroup_id | hostname | port | status       | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
    +--------------+----------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
    | 10           | xtradb1  | 3306 | ONLINE       | 1000   | 0           | 1000            | 0                   | 0       | 0              |         |
    | 10           | xtradb2  | 3306 | ONLINE       | 1000   | 0           | 1000            | 0                   | 0       | 0              |         |
    | 10           | xtradb3  | 3306 | ONLINE       | 1000   | 0           | 1000            | 0                   | 0       | 0              |         |
    | 20           | xtradb3  | 3306 | OFFLINE_SOFT | 1000   | 0           | 1000            | 0                   | 0       | 0              |         |
    | 20           | xtradb2  | 3306 | OFFLINE_SOFT | 1000   | 0           | 1000            | 0                   | 0       | 0              |         |
    | 20           | xtradb1  | 3306 | ONLINE       | 1000   | 0           | 1000            | 0                   | 0       | 0              |         |
    +--------------+----------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
    
    # HELP proxysql_connection_pool_status Connection pool usage statistic.
    # TYPE proxysql_connection_pool_status gauge
    proxysql_connection_pool_status{endpoint="xtradb1:3306",hostgroup="10"} 1
    proxysql_connection_pool_status{endpoint="xtradb1:3306",hostgroup="20"} 1
    proxysql_connection_pool_status{endpoint="xtradb2:3306",hostgroup="10"} 3
    proxysql_connection_pool_status{endpoint="xtradb2:3306",hostgroup="20"} 3
    proxysql_connection_pool_status{endpoint="xtradb3:3306",hostgroup="10"} 1
    proxysql_connection_pool_status{endpoint="xtradb3:3306",hostgroup="20"} 3
    
    

    As you can see the outputs don't match.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    mosi0815 , thanks for sharing the output. Agreed this is suspicious, and we (PMM Dev team) are discussing this internally and attempting to reproduce. We'll update this thread shortly.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi mosi0815 , one of our Engineers has requested the full /metrics output. You may want to DM or email it to me if you prefer [EMAIL="michael.coburn@percona.com"][email protected][/EMAIL]
    [code]
  • mosi0815mosi0815 Contributor Inactive User Role Beginner
    I emailed it to you because the PM feature somehow doesn't work.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi mosi0815 , my apologies for the delay on this topic. Our Developers want to confirm that you are collecting data at the same point in time, that there isn't any potential for drift between what ProxySQL vs proxysql_exporter reports. Can you please run the commands below, and replace IP with the address of the instance where proxysql_exporter is running:
    date
    curl -s --insecure https://IP:42004/metrics | grep proxysql_connection_pool_status
    date
    mysql -uadmin -padmin -S/tmp/proxysql_admin.sock -e "select * from stats_mysql_connection_pool ORDER BY hostgroup, srv_host, srv_port;"
    date
    mysql -uadmin -padmin -S/tmp/proxysql_admin.sock -e "select * from runtime_mysql_servers"
    
    Thanks mosi0815 ,
  • mosi0815mosi0815 Contributor Inactive User Role Beginner
    Hi,

    on Sunday I did my usual reboot cycle and the monitor worked correctly. Now I tried to reproduce the issue on our test system to collect the logs you requested. No chance. Everything works fine now. The question is why. The only updated which got in was a proxysql update.
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    mosi0815 , well on one hand I am delighted that your environment is again working! But on the other hand I fear that your experience may be the first of several to occur related to discrepancies between status in ProxySQL vs exporter. In my own exploratory testing I was using latest releases:
    • ProxySQL 1.3.7 (Percona build)
    • proxysql_exporter from pmm-client 1.1.5
    I'm going to close this thread, but if you happen to see this issue reoccur please start a new thread and we'll take action. Thanks again for your cooperation while we diagnosed this!
This discussion has been closed.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.