pmm mysql:queries not connecting to server v1.0.4

Percona Monitoring and Management (PMM) PMM 1.x
1

I have setup and deployed the pmm-server as a Marathon app within Mesos and have successfully setup the linux:metrics and mysql:metrics monitoring on the mysql nodes I am interested in monitoring. I have not been able to successfully get mysql:queries to return anything to the server. Whenever I attempt to view any of the registered nodes I get an error message saying:

“QAN API error: “MySQL Query Analytics configuration not found”.
Check the /var/log/qan-api.log file in docker container for more information.”

Clicking into the server info tab also reports that no agents are connected.

I setup the pmm-client on the mysql nodes with the following steps:

  1. sudo pmm-client config --server pmm.dcos.us.monitoring.net --client-address 10.190.110.180
  2. sudo pmm-admin add mysql --user abc --password 123 --query-source=perfschema

NOTE: I have had to manually specify the client-address as without it the pmm-client was assigning the client-address to something other then the nodes ip address and resulted in errors attempting to listen on a local ports (42000, 42001, etc).

After the above the output of the version, config, list and check-network commands are as follows:

sudo pmm-admin --version
1.0.4

sudo pmm-admin config
OK, PMM server is alive.

PMM Server | pmm.dcos.us.monitoring.net
Client Name | node1.northamerica.company.net
Client Address | 10.190.110.180

sudo pmm-admin list
pmm-admin 1.0.4

PMM Server | pmm.dcos.us.monitoring.net
Client Name | node1.northamerica.company.net
Client Address | 10.190.110.180
Service manager | unix-systemv

SERVICE TYPE NAME CLIENT PORT RUNNING DATA SOURCE OPTIONS

linux:metrics node1.northamerica.company.net 42000 YES -
mysql:queries node1.northamerica.company.net 42001 YES root:@unix(/var/lib/mysql/mysql.sock) query_source=perfschema
mysql:metrics node1.northamerica.company.net 42002 YES root:@unix(/var/lib/mysql/mysql.sock)

PMM Network Status

Server | pmm.dcos.us.monitoring.net
Client | 10.190.110.180

  • Client → Server

SERVER SERVICE CONNECTIVITY

Consul API

2

I get the following error when attempting to hit the agents endpoint of the qan-api:

curl pmm.dcos.us.monitoring.net/qan-api/agents
{
“Error”: “Agent.List: ah.GetAll: sql: Scan error on column index 5: unsupported Scan, storing driver.Value type into type *time.Time”
}

3

There was a bug on 1.0.4 when adding mysql:queries went through ok despite not enabling query analytics correctly.

You can try:
sudo pmm-admin remove mysql:queries
sudo pmm-admin add mysql:queries --user abc --password 123 --query-source=perfschema

Tomorrow, we will have 1.0.5 release which fixes such incorrect behaviour.

4

I have tried multiple times to add/remove mysql:queries but no matter what I do the log-ws and agent-ws always log out the “bad status” message. My guess is there is something preventing the websockets connecting to the server. Is port 80 the only port the server and agent communicate over? If this is not the case then I will need to add additional port mapping within Marathon LB to handle that. I can’t think of anything else that would prevent the websockets from connecting to the server. None of the listed ports (9000, 9001, 9003, 9004) are successfully listening on ports on the mysql node.

5

The above did not do anything. I have tried that multiple times and it doesn’t change anything. I am assuming the update marks the agent as not started correctly if the websockets are not setup properly? Was there a bug around them getting setup in the first place? I have been trying to test the connection but considering this requires me looking at the source control and trying to figure out what I need to test this has been rather hard. What additional steps should I take to help debug or help resolve this issue?