PPM2 client cannot comunicate with pmm-server

Percona Monitoring and Management (PMM)
1

Hi, I am trying to run PMM2 platform on Kubernetes.
For pmm-server was used official Helm Chart, for pmm-client was written own.

pmm-server works correctly and even could collect metrics from my MySQL DB if initialize it manually: pmm-admin add MySQL ...

But when I try to run pmm-client it always fails with the same error, here is a detailed log:

Starting pmm-agent setup ...
INFO[2021-04-05T12:06:55.815+00:00] Loading configuration file /usr/local/percona/pmm2/pmm-agent.yml.  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
INFO[2021-04-05T12:06:55.815+00:00] Updating PMM Server address from "<PMM_SERVER_IP>" to "<PMM_SERVER_IP>:443".  component=setup
Checking local pmm-agent status...
DEBU[2021-04-05T12:06:55.816+00:00] POST /local/Status HTTP/1.1
Host: 0.0.0.0:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}
  component=setup
pmm-agent is not running.
Registering pmm-agent on PMM Server...
DEBU[2021-04-05T12:06:55.816+00:00] Status error: &url.Error{Op:"Post", URL:"http://0.0.0.0:7777/local/Status", Err:(*net.OpError)(0xc0005160f0)}  component=setup
DEBU[2021-04-05T12:06:55.816+00:00] POST /v1/management/Node/Register HTTP/1.1
Host: <PMM_SERVER_IP>:443
User-Agent: Go-http-client/1.1
Content-Length: 197
Accept: application/json
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
Accept-Encoding: gzip

{"node_type":"GENERIC_NODE","node_name":"monit-mysql-client-agent-5x9wx","address":"10.0.5.152","machine_id":"5","distro":"linux","reregister":true,"metrics_mode":"AUTO","disable_collectors":null}
  component=setup
DEBU[2021-04-05T12:06:55.860+00:00] HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-store
Connection: keep-alive
Content-Type: application/json
Date: Mon, 05 Apr 2021 12:06:55 GMT
Grpc-Metadata-Content-Type: application/grpc
Pragma: no-cache
Strict-Transport-Security: max-age=15724800; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block

177
{
  "generic_node": {
    "node_id": "/node_id/dd25c611-1179-4cd5-a221-9c440862f658",
    "node_name": "monit-mysql-client-agent-5x9wx",
    "address": "10.0.5.152",
    "machine_id": "5",
    "distro": "linux"
  },
  "pmm_agent": {
    "agent_id": "/agent_id/d058bbdd-4d10-4503-b4b3-5f31d5784cb6",
    "runs_on_node_id": "/node_id/dd25c611-1179-4cd5-a221-9c440862f658"
  }
}
0

  component=setup
Registered.
DEBU[2021-04-05T12:06:55.860+00:00] Register error: <nil>                         component=setup
Configuration file /usr/local/percona/pmm2/pmm-agent.yml updated.
Please start pmm-agent: `pmm-agent --config-file=/usr/local/percona/pmm2/pmm-agent.yml`.
pmm-agent setup exited with 0.
Starting pmm-agent ...
INFO[2021-04-05T12:06:55.895+00:00] Loading configuration file /usr/local/percona/pmm2/pmm-agent.yml.  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=main
INFO[2021-04-05T12:06:55.895+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=main
INFO[2021-04-05T12:06:55.895+00:00] Updating PMM Server address from "<PMM_SERVER_IP>" to "<PMM_SERVER_IP>:443".  component=main
DEBU[2021-04-05T12:06:55.895+00:00] Loaded configuration: &{ID:/agent_id/n5.ams3 ListenAddress:0.0.0.0 ListenPort:7777 Server:{Address:<PMM_SERVER_IP>:443 Username:admin Password:admin InsecureTLS:true WithoutTLS:false} Paths:{ExportersBase:/usr/local/percona/pmm2/exporters NodeExporter:/usr/local/percona/pmm2/exporters/node_exporter MySQLdExporter:/usr/local/percona/pmm2/exporters/mysqld_exporter MongoDBExporter:/usr/local/percona/pmm2/exporters/mongodb_exporter PostgresExporter:/usr/local/percona/pmm2/exporters/postgres_exporter ProxySQLExporter:/usr/local/percona/pmm2/exporters/proxysql_exporter RDSExporter:/usr/local/percona/pmm2/exporters/rds_exporter VMAgent:/usr/local/percona/pmm2/exporters/vmagent TempDir:/tmp PTSummary:/usr/local/percona/pmm2/tools/pt-summary PTPgSummary:/usr/local/percona/pmm2/tools/pt-pg-summary PTMySqlSummary:/usr/local/percona/pmm2/tools/pt-mysql-summary PTMongoDBSummary:/usr/local/percona/pmm2/tools/pt-mongodb-summary SlowLogFilePrefix:} Ports:{Min:51997 Max:51999} Debug:true Trace:false Setup:{NodeType: NodeName: MachineID: Distro: ContainerID: ContainerName: NodeModel: Region: Az: Address: MetricsMode: DisableCollectors: Force:false SkipRegistration:false}}  component=main
INFO[2021-04-05T12:06:55.896+00:00] Starting...                                   component=client
INFO[2021-04-05T12:06:55.896+00:00] Connecting to https://admin:***@<PMM_SERVER_IP>:443/ ...  component=client
INFO[2021-04-05T12:06:55.896+00:00] Starting local API server on http://0.0.0.0:7777/ ...  component=local-server/JSON
DEBU[2021-04-05T12:06:55.896+00:00] Debug handlers:
	http://0.0.0.0:7777/debug/metrics
	http://0.0.0.0:7777/debug/vars
	http://0.0.0.0:7777/debug/requests
	http://0.0.0.0:7777/debug/events
	http://0.0.0.0:7777/debug/pprof  component=local-server/JSON
DEBU[2021-04-05T12:06:55.896+00:00] Starting gRPC server on http://127.0.0.1:38653/ ...  component=local-server/gRPC
DEBU[2021-04-05T12:06:55.896+00:00] Reflection and channelz are enabled.          component=local-server/gRPC
INFO[2021-04-05T12:06:55.897+00:00] Started.                                      component=local-server/JSON
INFO[2021-04-05T12:06:55.914+00:00] Connected to <PMM_SERVER_IP>:443.              component=client
INFO[2021-04-05T12:06:55.914+00:00] Establishing two-way communication channel ...  component=client
DEBU[2021-04-05T12:06:55.914+00:00] Sending message (4 bytes): id:1 ping:{}.      component=channel
DEBU[2021-04-05T12:07:00.896+00:00] Closing with error: rpc error: code = Canceled desc = context canceled
failed to receive message
github.com/percona/pmm-agent/client/channel.(*Channel).runReceiver
	/tmp/go/src/github.com/percona/pmm-agent/client/channel/channel.go:199
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1374  component=channel
DEBU[2021-04-05T12:07:06.950+00:00] Exiting receiver goroutine.                   component=channel
ERRO[2021-04-05T12:07:06.950+00:00] Failed to establish two-way communication channel: context canceled.  component=client
DEBU[2021-04-05T12:07:06.950+00:00] Connection closed.

What I am doing wrong?

2

Hi Andrey,

when you say official charts, which ones do you mean ?
GitHub - Percona-Lab/percona-openshift: Set of scripts to run Percona software in OpenShift / Kubernetes ?

Where do you try to run pmm-client, from you logs it looks like it couldn’t connect?
Could you please share how and where do you run client? How do you set IP, or host:port?

Thanks,
Denys

1 Like