Can't add mysql:queries to remote pmm server

Percona Monitoring and Management (PMM) PMM 1.x
#1

Hi !
I was try to add mysql:queries to my remote pmm server, but with no luck.

[root@hotel ~]# pmm-admin list
pmm-admin 1.1.3

PMM Server | 46.xx.xx.xx (password-protected)
Client Name | client-dns
Client Address | 88.xx.xx.xx
Service Manager | unix-systemv

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

linux:metrics xxxxx 42000 YES -
mysql:metrics xxxxx 42002 YES root:***@tcp(127.0.0.1:3306) tablestats=OFF

Both servers - mysql and pmm set with public IP and placed in different DC, linux and mysql metric working fine, but when i try add queries i get next error:

pmm-admin add mysql:queries
Error adding MySQL queries: timeout 10s waiting on agent to connect to API.

if check log

Version: percona-qan-agent 1.1.3

Basedir: /usr/local/percona/qan-agent

PID: 45814

API: 46.xx.xx.xx/qan-api

UUID: xxx

2017/05/26 11:22:48.517407 main.go:165: Starting agent…
2017/05/26 11:22:48.517892 main.go:346: Agent is ready
2017/05/26 11:22:58.548048 main.go:201: Cannot connect to API: GET http://admin:admin@192.168.122.68/qan-api/agents/xxx error: client.Do: Get http://admin:admin@192.168.122.68/qan-api/agents/xxx: dial tcp 192.168.122.68:80: i/o timeout. Verify that the agent UUID and API hostname printed above are correct and that no network issues prevent this host from accessing the API. Connection attempts to API will continue
until successful, but additional errors will not be logged, and agent will not send data until connected to API.

looks like qan agent try to use default private address for server connection.

if check qan-agent config they contain right server ip

“ApiHostname”:“46.xx.xx.xx”,

So what’s wrong with my installation and why qan-agent still try connect to private server ip except public?

#2

Hi Bolsvet,

you can review architecture document if you want - https://www.percona.com/doc/percona-monitoring-and-management/architecture.html
pay attention that PMM Client connects to PMM Server and PMM Server should have a possibility to connect to client.

try the following steps
remove all services


reconfigure pmm-admin with the following command

pmm-admin config --server pmm-SERVER-PUBLIC-ip --bind-address pmm-CLIENT-PRIVATE-ip --client-address pmm-CLIENT-PUBLIC-ip


add services again
#3

Ok, i do what you say,

pmm-admin remove --all
OK, 2 services were removed.

pmm-admin config --server 46.xx.xx.xx --bind-address 88.xx.xx.xx --client-address 88.xx.xx.xx --server-user admin --server-password admin
OK, PMM server is alive.

PMM Server | 46.xx.xx.xx (password-protected)
Client Name | client-dns
Client Address | 88.xx.xx.xx

pmm-admin add mysql
[linux:metrics] OK, now monitoring this system.
[mysql:metrics] OK, now monitoring MySQL metrics using DSN root:***@tcp(127.0.0.1:3306)
[mysql:queries] Error adding MySQL queries: timeout 10s waiting on agent to connect to API.

but qan agent still try connect to private ip of server

2017/05/26 14:16:30.183509 main.go:201: Cannot connect to API: GET http://admin:admin@192.168.122.68/qan-api/agents/xxx error: client.Do: Get http://admin:admin@192.168.122.68/qan-api/agents/xxx: dial tcp 192.168.122.68:80: i/o timeout. Verify that the agent UUID and API hostname printed above are correct and that no network issues prevent this host from accessing the API. Connection attempts to API will continue
until successful, but additional errors will not be logged, and agent will not send data until connected to API.

#4

is 88.xx.xx.xx address local (and visible in ifconfig output)?
can you share file content?

#5

Yes it visible, and it local for this server

ifconfig

em1 Link encap:Ethernet inet addr:88.xx.xx.xx
em2 Link encap:Ethernet inet addr:10.x.x.x

content of pmm.yml file

server_address: 46.xx.xx.xx
client_address: 88.xx.xx.xx
bind_address: 88.xx.xx.xx
client_name: client-dns
server_user: admin
server_password: admin

#6

can PMM Server reach itself?
can you run the following command on pmm server side and share output?

docker exec -it pmm-server curl http://admin:admin@192.168.122.68:80/qan-api/

Do you have http proxy between pmm client and pmm server?
can you run the following command on pmm clietn side and share output?

curl http://admin:admin@46.xx.xx.xx:80/qan-api/
#7

Ok, part of problem was solved. It was problem in nginx - it not send remote ip header to container.

Now when i try to connect

[root@hotel log]# pmm-admin repair
OK, removed 1 orphaned services.
[root@hotel log]# pmm-admin add mysql
[linux:metrics] OK, already monitoring this system.
[mysql:metrics] OK, already monitoring MySQL metrics.
[mysql:queries] Error adding MySQL queries: timeout 10s waiting on agent to connect to API.


# Version: percona-qan-agent 1.1.3
# Basedir: /usr/local/percona/qan-agent
# PID: 48867
# API: 46.xx.xx.xx/qan-api
# UUID: 60fb0cbdb96648d446f92ba267c392a7
2017/05/26 15:19:03.082891 main.go:165: Starting agent...
2017/05/26 15:19:03.083386 main.go:346: Agent is ready
2017/05/26 15:19:03.118075 main.go:207: API is ready
2017/05/26 15:19:04.110111 WARNING agent-ws bad status
2017/05/26 15:19:04.111023 WARNING log-ws bad status
2017/05/26 15:19:07.135117 WARNING log-ws bad status
2017/05/26 15:19:07.137429 WARNING agent-ws bad status
2017/05/26 15:19:14.161181 WARNING agent-ws bad status
2017/05/26 15:19:14.161585 WARNING log-ws bad status
2017/05/26 15:19:29.183139 WARNING agent-ws bad status
2017/05/26 15:19:29.187081 WARNING log-ws bad status
2017/05/26 15:20:00.211216 WARNING agent-ws bad status
2017/05/26 15:20:00.216240 WARNING log-ws bad status
2017/05/26 15:20:06.104079 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:20:09.124754 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:20:12.145883 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:21:00.233388 WARNING agent-ws bad status
2017/05/26 15:21:00.238123 WARNING log-ws bad status
2017/05/26 15:21:09.104098 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:21:12.126775 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:21:15.146306 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:22:00.254524 WARNING agent-ws bad status
2017/05/26 15:22:00.260663 WARNING log-ws bad status
2017/05/26 15:22:12.104978 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:22:15.124612 WARNING data-sender Cannot connect to API: bad status
2017/05/26 15:22:18.146449 WARNING data-sender Cannot connect to API: bad status
#8

qan-agent use websockets for some API calls (like /agent-ws /log-ws) so you shouldn’t use a proxy or configure nginx to proxying websockets.
anyway correct work via proxy is not guaranteed.

#9

I move monitoring node to same private network with monitored mysql hosts and now all work as expected. Thank you !

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2021 Percona LLC. All rights reserved.