Failed to get connection service info: timeout (context deadline exceeded) when trying to add MySQL with pmm-admin

Percona Monitoring and Management (PMM) PMM 2.x
1

Description:

Removed a mysql server from pmm2 as we wanted to disable tablestats but can’t reconnect it due to “failed to get connection service info: timeout (context deadline exceeded).”

Command: pmm-admin add mysql --query-source=slowlog --size-slow-logs=1GiB --disable-tablestats=true --username=pmm --password=<pass> mysql-1 10.0.10.11:3306

Version:

PMM-Client 2.42
8.0.32-24 Percona Server

Logs:

DEBUG 2024-06-19 09:04:33.221272939Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.28.0/client/runtime.go:456 github.com/go-openapi/runtime/client.(*Runtime).Submit() POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}

DEBUG 2024-06-19 09:04:33.222577733Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.28.0/client/runtime.go:512 github.com/go-openapi/runtime/client.(*Runtime).Submit() HTTP/1.1 200 OK
Content-Length: 1139
Content-Type: application/json
Date: Wed, 19 Jun 2024 09:04:33 GMT
Grpc-Metadata-Content-Type: application/grpc

{
  "agent_id":  "/agent_id/177596f0-c8b6-4201-a60f-02bd75295426",
  "runs_on_node_id":  "/node_id/f3e0aca7-d43b-41b5-aba5-d5a70a2ae9e3",
  "server_info":  {
    "url":  "https://<user>:<token>@10.0.10.9:443/",
    "insecure_tls":  true,
    "connected":  true,
    "version":  "2.42.0",
    "latency":  null,
    "clock_drift":  null
  },
  "agents_info":  [
    {
      "agent_id":  "/agent_id/1e03499c-02a2-416d-8017-9fd1a3d02647",
      "agent_type":  "VM_AGENT",
      "status":  "RUNNING",
      "listen_port":  42000,
      "process_exec_path":  "/usr/local/percona/pmm2/exporters/vmagent"
    },
    {
      "agent_id":  "/agent_id/94502783-8c9b-4084-ac5b-4f49b06d0340",
      "agent_type":  "NODE_EXPORTER",
      "status":  "RUNNING",
      "listen_port":  42001,
      "process_exec_path":  "/usr/local/percona/pmm2/exporters/node_exporter"
    }
  ],
  "config_filepath":  "/usr/local/percona/pmm2/config/pmm-agent.yaml",
  "agent_version":  "2.42.0",
  "node_name":  "mysql-1",
  "connection_uptime":  99.7
}
DEBUG 2024-06-19 09:04:33.222964633Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.28.0/client/runtime.go:456 github.com/go-openapi/runtime/client.(*Runtime).Submit() POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}

DEBUG 2024-06-19 09:04:33.223540972Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.28.0/client/runtime.go:512 github.com/go-openapi/runtime/client.(*Runtime).Submit() HTTP/1.1 200 OK
Content-Length: 1139
Content-Type: application/json
Date: Wed, 19 Jun 2024 09:04:33 GMT
Grpc-Metadata-Content-Type: application/grpc

{
  "agent_id":  "/agent_id/177596f0-c8b6-4201-a60f-02bd75295426",
  "runs_on_node_id":  "/node_id/f3e0aca7-d43b-41b5-aba5-d5a70a2ae9e3",
  "server_info":  {
    "url":  "https://<user>:<token>@10.0.10.9:443/",
    "insecure_tls":  true,
    "connected":  true,
    "version":  "2.42.0",
    "latency":  null,
    "clock_drift":  null
  },
  "agents_info":  [
    {
      "agent_id":  "/agent_id/1e03499c-02a2-416d-8017-9fd1a3d02647",
      "agent_type":  "VM_AGENT",
      "status":  "RUNNING",
      "listen_port":  42000,
      "process_exec_path":  "/usr/local/percona/pmm2/exporters/vmagent"
    },
    {
      "agent_id":  "/agent_id/94502783-8c9b-4084-ac5b-4f49b06d0340",
      "agent_type":  "NODE_EXPORTER",
      "status":  "RUNNING",
      "listen_port":  42001,
      "process_exec_path":  "/usr/local/percona/pmm2/exporters/node_exporter"
    }
  ],
  "config_filepath":  "/usr/local/percona/pmm2/config/pmm-agent.yaml",
  "agent_version":  "2.42.0",
  "node_name":  "mysql-1",
  "connection_uptime":  99.7
}
DEBUG 2024-06-19 09:04:33.223820526Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.28.0/client/runtime.go:456 github.com/go-openapi/runtime/client.(*Runtime).Submit() POST /v1/management/MySQL/Add HTTP/1.1
Host: 10.0.10.9:443
User-Agent: Go-http-client/1.1
Content-Length: 430
Accept: application/json
Authorization: Basic <token>
Content-Type: application/json
Accept-Encoding: gzip

{"node_id":"/node_id/f3e0aca7-d43b-41b5-aba5-d5a70a2ae9e3","service_name":"mysql-1","address":"10.0.10.11","port":3306,"pmm_agent_id":"/agent_id/177596f0-c8b6-4201-a60f-02bd75295426","username":"pmm","password":"<pass>","qan_mysql_slowlog":true,"disable_comments_parsing":true,"max_slowlog_file_size":"1073741824","tablestats_group_table_limit":-1,"metrics_mode":"AUTO","disable_collectors":null,"log_level":"warn"}

DEBUG 2024-06-19 09:04:36.2455206Z: /tmp/go/pkg/mod/github.com/go-openapi/runtime@v0.28.0/client/runtime.go:512 github.com/go-openapi/runtime/client.(*Runtime).Submit() HTTP/1.1 400 Bad Request
Content-Length: 200
Connection: keep-alive
Content-Type: application/json
Date: Wed, 19 Jun 2024 09:04:36 GMT
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;

{
  "error":  "failed to get connection service info: timeout (context deadline exceeded).",
  "code":  9,
  "message":  "failed to get connection service info: timeout (context deadline exceeded)."
}
DEBUG 2024-06-19 09:04:36.245688875Z: /tmp/go/src/github.com/percona/pmm/admin/cli/cli.go:132 github.com/percona/pmm/admin/cli.printResponse() Result: <nil>
DEBUG 2024-06-19 09:04:36.245742808Z: /tmp/go/src/github.com/percona/pmm/admin/cli/cli.go:133 github.com/percona/pmm/admin/cli.printResponse() Error: &my_sql.AddMySQLDefault{_statusCode:400, Payload:(*my_sql.AddMySQLDefaultBody)(0xc00045e780)}
failed to get connection service info: timeout (context deadline exceeded).

Expected Result:

As I understood using --disable-tablestats should solve this?

2

Found a hidden flag --skip-connection-check when looking through the code for pmm-admin and could reconnect the server again using that