Not the answer you need?
Register and ask your own question!

pmm2-client - Failed to establish two-way communication channel

We are having issues connecting pmm2-client with pmm-server ( pmm2 ). Everything was working working from March 2020 till 29Th September 5PM but then all of sudden all 

the pmm clients metrics stopped and on client logs are showing failed to connect to pmm server with below error 

Error Log

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[36mINFO#033[0m[2020-10-08T04:51:07.085+00:00] Connected to pmm.xxxxxxxx.com:443.    #033[36mcomponent#033[0m=client

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[36mINFO#033[0m[2020-10-08T04:51:07.085+00:00] Establishing two-way communication channel ... #033[36mcomponent#033[0m=client

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[37mDEBU#033[0m[2020-10-08T04:51:07.085+00:00] Sending message (4 bytes): id:1 ping:<> .   #033[37mcomponent#033[0m=channel

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[37mDEBU#033[0m[2020-10-08T04:51:07.090+00:00] Closing with error: rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field

Oct 8 04:51:07 n01-nsy1 pmm-agent: failed to receive message

Oct 8 04:51:07 n01-nsy1 pmm-agent: github.com/percona/pmm-agent/client/channel.(*Channel).runReceiver

Oct 8 04:51:07 n01-nsy1 pmm-agent: /tmp/go/src/github.com/percona/pmm-agent/client/channel/channel.go:199

Oct 8 04:51:07 n01-nsy1 pmm-agent: runtime.goexit

Oct 8 04:51:07 n01-nsy1 pmm-agent: /usr/local/go/src/runtime/asm_amd64.s:1373 #033[37mcomponent#033[0m=channel

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[37mDEBU#033[0m[2020-10-08T04:51:07.090+00:00] Exiting receiver goroutine.          #033[37mcomponent#033[0m=channel

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[31mERRO#033[0m[2020-10-08T04:51:07.090+00:00] Failed to establish two-way communication channel: : HTTP status code 464; transport: missing content-type field. #033[31mcomponent#033[0m=client

Oct 8 04:51:07 n01-nsy1 pmm-agent: #033[37mDEBU#033[0m[2020-10-08T04:51:07.090+00:00] Connection closed.

********************************************************************


pmm-admin config --server-insecure-tls --server-url='https://admin:[email protected]:443' --force --debug


DEBUG 2020-10-08 04:50:32.690101797Z: Running: pmm-agent --server-address=pmm.xxxxx:443 --server-username=admin --server-password=xxxxxxx --server-insecure-tls --debug setup --force 58.x.x.x generic dev-iboss01-sysdb01-n01-nsy1DEBUG 2020-10-08 04:50:34.816381163Z: Result: &commands.configResult{Warning:"", Output:"Checking local pmm-agent status...\npmm-agent is running.\nRegistering pmm-agent on PMM Server...\nRegistered.\nConfiguration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.\nReloading pmm-agent configuration...\nConfiguration reloaded.\nChecking local pmm-agent status...\npmm-agent is running."}

DEBUG 2020-10-08 04:50:34.816405248Z: Error: <nil>

Checking local pmm-agent status...

pmm-agent is running.

Registering pmm-agent on PMM Server...

Registered.

Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.

Reloading pmm-agent configuration...

Configuration reloaded.

Checking local pmm-agent status...

pmm-agent is running.


[[email protected] ~]#

[[email protected] ~]# pmm-admin status

Failed to get PMM Server parameters from local pmm-agent: pmm-agent is running, but not set up.

Please run `pmm-admin config` with --server-url flag.

******************************************************************************

Client is able to telnet to pmm server via port 443. 

pmm server is able to telnet client via ports 4200x


Even tried upgrading the pmm server and client to the latest version 2.10.1 but still the agent is failing to connect to the server.

Can somebody please help me on this?

Tagged:

Answers

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.