Query Analytics hosts list

Percona Monitoring and Management (PMM) PMM 1.x
1

Hello, I installed PMM Server 1.13.0 in docker and trying monitor MySQL. Everything is ok but on “Query Analitics” tab I have empty hosts list.

When I open any other metric and then go back to “Query Analitics” I see all queryes etc but hosts list still empty.

Can you help with this issue?

photoid=52273
photoid=522731913×591 40.3 KB

photoid=52274
photoid=522741635×633 104 KB

photoid=52275
photoid=522751835×768 186 KB

2

Hi Tarrok

Can you please provide the output from:

pmm-admin list

Also can you provide the contents of the error log file from a host, it is stored in:

/var/log/pmm-mysql-queries*.log

Lastly, you can fetch the logs from PMM Server in one click by navigating to the Prometheus Dashboard, and under Panel PMM Server Logs you will see a link to download logs.zip to your machine. It would be helpful if you could either DM me, or mail me michael.coburn@percona.com and I can review them.

I’m surprised to see rows in the Query Analytics output yet no hosts in the drop down - this is the first time I’ve seen this issue! Share with me the details as requested above and I’ll dig further into this!

pmmdemo-server-logs.PNG
pmmdemo-server-logs.PNG738×637 36.5 KB

3

Hi Michael Coburn and thank you for replay!

My pmm-admin command output:

pmm-admin 1.11.0

PMM Server | pmm.luxnet.ua (SSL, password-protected)
Client Name | db.lux.fm
Client Address | 192.168.31.34 
Service Manager | linux-systemd

-------------- ---------- ----------- -------- ------------------------------------------ --------------------------------------------------------------------------------------
SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS 
-------------- ---------- ----------- -------- ------------------------------------------ --------------------------------------------------------------------------------------
mysql:queries db.lux.fm - YES root:***@unix(/mnt/mysql/data/mysql.sock) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1 
linux:metrics db.lux.fm 42000 YES - 
mysql:metrics db.lux.fm 42002 YES root:***@unix(/mnt/mysql/data/mysql.sock)

pmm-mysql-queries-0.log file contains only Warnings when I have problems with Internet connections on client side:

# Version: percona-qan-agent 1.0.5
# Basedir: /usr/local/percona/qan-agent
# PID: 4261
# API: pmm.luxnet.ua/qan-api
# UUID: ed7e4565d73b478757c61d0e35c5b078
2018/08/31 16:51:23.025098 main.go:153: Starting agent...
2018/08/31 16:51:23.025904 main.go:321: Agent is ready
2018/08/31 16:51:23.684859 main.go:194: API is ready
{UUID: CollectFrom: Interval:60 ExampleQueries:0xc4204910a0 MaxSlowLogSize:1073741824 SlowLogRotation:0xc4204910a1 RetainSlowLogs:0xc4204910a8 Start:[] Stop:[] ReportLimit:200}
{UUID:a2f83a990ac44d7755ae637c86ee75a7 CollectFrom:slowlog Interval:60 ExampleQueries:0xc4204910a0 MaxSlowLogSize:1073741824 SlowLogRotation:0xc4204910a1 RetainSlowLogs:0xc4204910a8 Start:[] Stop:[] ReportLimit:200}
2018/09/03 12:18:18.882978 WARNING agent ws error: EOF
2018/09/03 12:18:18.883163 WARNING agent Lost connection to API, reconnecting
2018/09/03 12:18:21.901516 WARNING agent-ws websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/cmd: dial tcp 194.44.175.8:443: getsockopt: connection refused
2018/09/03 12:18:21.901718 WARNING log Lost connection to API
2018/09/03 12:18:25.915836 WARNING log-ws websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/log: dial tcp 194.44.175.8:443: getsockopt: connection refused
2018/09/03 12:18:28.911215 WARNING agent-ws websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/cmd: dial tcp 194.44.175.8:443: getsockopt: connection refused
2018/09/03 12:18:32.925623 WARNING log-ws websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/log: dial tcp 194.44.175.8:443: getsockopt: connection refused
2018/09/06 14:37:36.026398 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:37:49.026906 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:38:02.031524 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:38:39.032367 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:38:52.032887 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:39:05.034407 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:39:42.026397 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:39:55.026880 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout
2018/09/06 14:40:08.028351 WARNING data-sender Cannot connect to API: websocket.Dial wss://pmm.luxnet.ua:443/qan-api/agents/ed7e4565d73b478757c61d0e35c5b078/data: dial tcp 194.44.175.8:443: i/o timeout

Server logs I sent you to email.

4

Hi Tarrok

It looks to me that you have some level of network problems - take a look at the dashboard Network Overview, for example:
http://18.223.109.241/graph/d/000000193/network-overview?refresh=1m&orgId=1&var-interval=$__auto_interval_interval&var-host=pmm-server&var-device=All

This should show you if you have a higher than normal level of errors.

I’m a​​​​​​Adding relevant log file data here (no site-specific data exists so it is safe).
qan-api.log:

[0m2018/09/06 11:57:55 server.go:2923: http: response.WriteHeader on hijacked connection
2018/09/06 11:57:55 server.go:2923: http: response.Write on hijacked connection
[0;31mERROR 2018/09/06 11:57:55 results.go:336: Response write failed: http: connection has been hijacked
[0m2018/09/06 11:57:55.372 127.0.0.1 404 2.68584ms WS /agents/46a88e4f1a4d4e507c16359e19dda957/cmd
[0m2018/09/06 11:57:58.575 127.0.0.1 0 129.346539ms WS /agents/58daa0cbc0ea47494f7774c30d8d39d7/data
[0m2018/09/06 11:57:59.056 127.0.0.1 0 73.210028ms WS /agents/ba8a6c9da09c4f737b4113cbb4f65d92/data
[0m2018/09/06 11:58:00.146 127.0.0.1 0 39.104597ms WS /agents/1b2f8460a6f1440f4eaa1000cdfb8124/data
[0m2018/09/06 11:58:00.410 127.0.0.1 0 70.530365ms WS /agents/60f08e980138453042cf14284b4fd659/data
[0m2018/09/06 11:58:02.413 127.0.0.1 0 73.09395ms WS /agents/021ff46a585942597f64d2e7d90cf03e/data
[0m2018/09/06 11:58:04 server.go:2923: http: response.WriteHeader on hijacked connection
2018/09/06 11:58:04 server.go:2923: http: response.Write on hijacked connection
[0;31mERROR 2018/09/06 11:58:04 results.go:336: Response write failed: http: connection has been hijacked
[0m2018/09/06 11:58:04.129 127.0.0.1 404 2.97105ms WS /agents/10bf777b28474cdb702d22815db10f11/log
[0m2018/09/06 11:58:06.337 127.0.0.1 0 8.143179ms WS /agents/2aa04aadcaa54c765cd8d059d9f53b1a/data
[0m2018/09/06 11:58:06 server.go:2923: http: response.WriteHeader on hijacked connection
2018/09/06 11:58:06 server.go:2923: http: response.Write on hijacked connection
[0;31mERROR 2018/09/06 11:58:06 results.go:336: Response write failed: http: connection has been hijacked
[0m2018/09/06 11:58:06.585 127.0.0.1 404 2.853672ms WS /agents/e7ea27cb7a0c48956d386b9a452cd2c4/cmd
[0m2018/09/06 11:58:06.403 127.0.0.1 0 207.473388ms WS /agents/0a2dbe8fdb94414d7a85afb03935f7ba/data

error.log

2018/09/06 11:57:06 [error] 167#0: *569318 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 185.41.250.16, server: _, request: "GET /qan-api/agents/f67247e16851468b450e8da017a780cc/log HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/f67247e16851468b450e8da017a780cc/log", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:16 [error] 166#0: *569344 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/46a88e4f1a4d4e507c16359e19dda957/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/46a88e4f1a4d4e507c16359e19dda957/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:20 [error] 164#0: *569356 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.106, server: _, request: "GET /qan-api/agents/1a50ba9e8bba476d76fedb6ac240d6a6/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/1a50ba9e8bba476d76fedb6ac240d6a6/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:23 [error] 166#0: *569360 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.106, server: _, request: "GET /qan-api/agents/1a50ba9e8bba476d76fedb6ac240d6a6/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/1a50ba9e8bba476d76fedb6ac240d6a6/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:23 [error] 162#0: *569362 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.106, server: _, request: "GET /qan-api/agents/1a50ba9e8bba476d76fedb6ac240d6a6/log HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/1a50ba9e8bba476d76fedb6ac240d6a6/log", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:26 [error] 167#0: *569364 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.106, server: _, request: "GET /qan-api/agents/1a50ba9e8bba476d76fedb6ac240d6a6/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/1a50ba9e8bba476d76fedb6ac240d6a6/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:33 [error] 163#0: *569374 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/6614e3adec544af85cfe09e3b1e33aa6/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/6614e3adec544af85cfe09e3b1e33aa6/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:36 [error] 167#0: *569384 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/b73f88e5f9a84f455d748d190886d925/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/b73f88e5f9a84f455d748d190886d925/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:39 [error] 167#0: *569392 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/b73f88e5f9a84f455d748d190886d925/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/b73f88e5f9a84f455d748d190886d925/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:40 [error] 166#0: *569399 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/6614e3adec544af85cfe09e3b1e33aa6/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/6614e3adec544af85cfe09e3b1e33aa6/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:44 [error] 167#0: *569415 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/797f8ff755cd408b55f2f9118c652076/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/797f8ff755cd408b55f2f9118c652076/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:48 [error] 164#0: *569421 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/44ca625cfb1f4c3a4499e7e756b58791/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/44ca625cfb1f4c3a4499e7e756b58791/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:50 [error] 164#0: *569428 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/10bf777b28474cdb702d22815db10f11/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/10bf777b28474cdb702d22815db10f11/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:51 [error] 161#0: *569437 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/44ca625cfb1f4c3a4499e7e756b58791/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/44ca625cfb1f4c3a4499e7e756b58791/data", host: "pmm.luxnet.ua:443"
2018/09/06 11:57:54 [error] 164#0: *569447 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 194.44.175.2, server: _, request: "GET /qan-api/agents/44ca625cfb1f4c3a4499e7e756b58791/data HTTP/1.1", upstream: "http://127.0.0.1:9001/agents/44ca625cfb1f4c3a4499e7e756b58791/data", host: "pmm.luxnet.ua:443"