Not the answer you need?
Register and ask your own question!

Can't add mysql:queries to remote pmm server

BolsvetBolsvet EntrantCurrent User Role Beginner
Hi !
I was try to add mysql:queries to my remote pmm server, but with no luck.

[[email protected] ~]# pmm-admin list
pmm-admin 1.1.3

PMM Server | 46.xx.xx.xx (password-protected)
Client Name | client-dns
Client Address | 88.xx.xx.xx
Service Manager | unix-systemv






SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





linux:metrics xxxxx 42000 YES -
mysql:metrics xxxxx 42002 YES root:***@tcp(127.0.0.1:3306) tablestats=OFF

Both servers - mysql and pmm set with public IP and placed in different DC, linux and mysql metric working fine, but when i try add queries i get next error:

pmm-admin add mysql:queries
Error adding MySQL queries: timeout 10s waiting on agent to connect to API.

if check log

# Version: percona-qan-agent 1.1.3
# Basedir: /usr/local/percona/qan-agent
# PID: 45814
# API: 46.xx.xx.xx/qan-api
# UUID: xxx
2017/05/26 11:22:48.517407 main.go:165: Starting agent...
2017/05/26 11:22:48.517892 main.go:346: Agent is ready
2017/05/26 11:22:58.548048 main.go:201: Cannot connect to API: GET http://admin:[email protected]/qan-api/agents/xxx error: client.Do: Get http://admin:[email protected]/qan-api/agents/xxx: dial tcp 192.168.122.68:80: i/o timeout. Verify that the agent UUID and API hostname printed above are correct and that no network issues prevent this host from accessing the API. Connection attempts to API will continue
until successful, but additional errors will not be logged, and agent will not send data until connected to API.

looks like qan agent try to use default private address for server connection.

if check qan-agent config they contain right server ip

"ApiHostname":"46.xx.xx.xx",

So what's wrong with my installation and why qan-agent still try connect to private server ip except public?

Comments

  • MykolaMykola Percona Percona Staff Role
    Hi Bolsvet,

    you can review architecture document if you want - https://www.percona.com/doc/percona-monitoring-and-management/architecture.html
    pay attention that PMM Client connects to PMM Server and PMM Server should have a possibility to connect to client.

    try the following steps
    remove all services
    reconfigure pmm-admin with the following command
    [code]pmm-admin config  --server  pmm-SERVER-PUBLIC-ip --bind-address pmm-CLIENT-PRIVATE-ip  --client-address pmm-CLIENT-PUBLIC-ip
    
    add services again
    [code]
  • BolsvetBolsvet Entrant Current User Role Beginner
    Ok, i do what you say,

    pmm-admin remove --all
    OK, 2 services were removed.

    pmm-admin config --server 46.xx.xx.xx --bind-address 88.xx.xx.xx --client-address 88.xx.xx.xx --server-user admin --server-password admin
    OK, PMM server is alive.

    PMM Server | 46.xx.xx.xx (password-protected)
    Client Name | client-dns
    Client Address | 88.xx.xx.xx

    pmm-admin add mysql
    [linux:metrics] OK, now monitoring this system.
    [mysql:metrics] OK, now monitoring MySQL metrics using DSN root:***@tcp(127.0.0.1:3306)
    [mysql:queries] Error adding MySQL queries: timeout 10s waiting on agent to connect to API.

    but qan agent still try connect to private ip of server

    2017/05/26 14:16:30.183509 main.go:201: Cannot connect to API: GET http://admin:[email protected]/qan-api/agents/xxx error: client.Do: Get http://admin:[email protected]/qan-api/agents/xxx: dial tcp 192.168.122.68:80: i/o timeout. Verify that the agent UUID and API hostname printed above are correct and that no network issues prevent this host from accessing the API. Connection attempts to API will continue
    until successful, but additional errors will not be logged, and agent will not send data until connected to API.
  • MykolaMykola Percona Percona Staff Role
    Bolsvet wrote: »
    pmm-admin config --server 46.xx.xx.xx --bind-address 88.xx.xx.xx --client-address 88.xx.xx.xx --server-user admin --server-password admin
    is 88.xx.xx.xx address local (and visible in ifconfig output)?
    can you share file content?
  • BolsvetBolsvet Entrant Current User Role Beginner
    Yes it visible, and it local for this server

    ifconfig

    em1 Link encap:Ethernet inet addr:88.xx.xx.xx
    em2 Link encap:Ethernet inet addr:10.x.x.x

    content of pmm.yml file

    server_address: 46.xx.xx.xx
    client_address: 88.xx.xx.xx
    bind_address: 88.xx.xx.xx
    client_name: client-dns
    server_user: admin
    server_password: admin
  • MykolaMykola Percona Percona Staff Role
    can PMM Server reach itself?
    can you run the following command on pmm server side and share output?
    docker exec -it pmm-server curl http://admin:admin@192.168.122.68:80/qan-api/
    

    Do you have http proxy between pmm client and pmm server?
    can you run the following command on pmm clietn side and share output?
    curl http://admin:admin@46.xx.xx.xx:80/qan-api/
    
  • BolsvetBolsvet Entrant Current User Role Beginner
    Mykola wrote: »

    Do you have http proxy between pmm client and pmm server?

    Ok, part of problem was solved. It was problem in nginx - it not send remote ip header to container.

    Now when i try to connect
    [root@hotel log]# pmm-admin repair
    OK, removed 1 orphaned services.
    [root@hotel log]# pmm-admin add mysql
    [linux:metrics] OK, already monitoring this system.
    [mysql:metrics] OK, already monitoring MySQL metrics.
    [mysql:queries] Error adding MySQL queries: timeout 10s waiting on agent to connect to API.
    
    
    # Version: percona-qan-agent 1.1.3
    # Basedir: /usr/local/percona/qan-agent
    # PID:     48867
    # API:     46.xx.xx.xx/qan-api
    # UUID:    60fb0cbdb96648d446f92ba267c392a7
    2017/05/26 15:19:03.082891 main.go:165: Starting agent...
    2017/05/26 15:19:03.083386 main.go:346: Agent is ready
    2017/05/26 15:19:03.118075 main.go:207: API is ready
    2017/05/26 15:19:04.110111 WARNING agent-ws bad status
    2017/05/26 15:19:04.111023 WARNING log-ws bad status
    2017/05/26 15:19:07.135117 WARNING log-ws bad status
    2017/05/26 15:19:07.137429 WARNING agent-ws bad status
    2017/05/26 15:19:14.161181 WARNING agent-ws bad status
    2017/05/26 15:19:14.161585 WARNING log-ws bad status
    2017/05/26 15:19:29.183139 WARNING agent-ws bad status
    2017/05/26 15:19:29.187081 WARNING log-ws bad status
    2017/05/26 15:20:00.211216 WARNING agent-ws bad status
    2017/05/26 15:20:00.216240 WARNING log-ws bad status
    2017/05/26 15:20:06.104079 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:20:09.124754 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:20:12.145883 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:21:00.233388 WARNING agent-ws bad status
    2017/05/26 15:21:00.238123 WARNING log-ws bad status
    2017/05/26 15:21:09.104098 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:21:12.126775 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:21:15.146306 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:22:00.254524 WARNING agent-ws bad status
    2017/05/26 15:22:00.260663 WARNING log-ws bad status
    2017/05/26 15:22:12.104978 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:22:15.124612 WARNING data-sender Cannot connect to API:  bad status
    2017/05/26 15:22:18.146449 WARNING data-sender Cannot connect to API:  bad status
    
  • MykolaMykola Percona Percona Staff Role
    qan-agent use websockets for some API calls (like /agent-ws /log-ws) so you shouldn't use a proxy or configure nginx to proxying websockets.
    anyway correct work via proxy is not guaranteed.
  • BolsvetBolsvet Entrant Current User Role Beginner
    I move monitoring node to same private network with monitored mysql hosts and now all work as expected. Thank you !
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.