Pmm-agent is not connected to PMM Server

Percona Monitoring and Management (PMM) PMM 2.x
1

Hi Team, Currently I’m unable to add node into my PMM2 server it’s saying agent can’t connect to pmm server. Please note that currently my pmm server is working for previously added node. But I can not add new node.

> # pmm-admin config --debug --server-insecure-tls --server-url=https://sadmin:PassWord4@PMM-Server-IP:443 MySQL-Node-IP generic Haalkhata-DB-02 --force
Output of this Command:

DEBUG 2022-06-15 06:23:29.960834528Z: Running: pmm-agent --server-address=PMM-Server-IP:443 --server-username=sadmin --server-password=PassWord4 --server-insecure-tls --debug setup --force --metrics-mode=auto MySQL-Node-IP generic Haalkhata-DB-02
DEBUG 2022-06-15 06:23:31.04559521Z: Result: &commands.configResult{Warning:"", Output:"Checking local pmm-agent status...\npmm-agent is running.\nRegistering pmm-agent on PMM Server...\nRegistered.\nConfiguration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.\nReloading pmm-agent configuration...\nConfiguration reloaded.\nChecking local pmm-agent status...\npmm-agent is running."}
DEBUG 2022-06-15 06:23:31.045668742Z: Error: <nil>
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.

Adding MySQL Server with PMM:

# pmm-admin add mysql --server-url=https://sadmin:PassWord4@PMM-Server-IP:443 --query-source=perfschema --disable-tablestats --username=mysql_user --password='mysql_user_password' --service-name=Haalkhata-DB-02 --host=MySQL-Node-IP --port=3366 --environment=Prod-MA
pmm-agent is not connected to PMM Server

It’s saying pmm-agent isnot connected to PMM Server, then I run debug command bellow is the output.


# pmm-agent --debug --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
INFO[2022-06-15T12:26:05.432+06:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=main
INFO[2022-06-15T12:26:05.433+06:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=main
INFO[2022-06-15T12:26:05.434+06:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=main
DEBU[2022-06-15T12:26:05.434+06:00] Loaded configuration: &{ID:/agent_id/42f3e698-95f3-4a92-bb17-414ba50fa9ba ListenAddress:127.0.0.1 ListenPort:7777 Server:{Address:PMM-Server-IP:443 Username:sadmin Password:PassWord4 InsecureTLS:true WithoutTLS:false} Paths:{PathsBase:/usr/local/percona/pmm2 ExportersBase:/usr/local/percona/pmm2/exporters NodeExporter:/usr/local/percona/pmm2/exporters/node_exporter MySQLdExporter:/usr/local/percona/pmm2/exporters/mysqld_exporter MongoDBExporter:/usr/local/percona/pmm2/exporters/mongodb_exporter PostgresExporter:/usr/local/percona/pmm2/exporters/postgres_exporter ProxySQLExporter:/usr/local/percona/pmm2/exporters/proxysql_exporter RDSExporter:/usr/local/percona/pmm2/exporters/rds_exporter AzureExporter:/usr/local/percona/pmm2/exporters/azure_exporter VMAgent:/usr/local/percona/pmm2/exporters/vmagent TempDir:/tmp PTSummary:/usr/local/percona/pmm2/tools/pt-summary PTPgSummary:/usr/local/percona/pmm2/tools/pt-pg-summary PTMySqlSummary:/usr/local/percona/pmm2/tools/pt-mysql-summary PTMongoDBSummary:/usr/local/percona/pmm2/tools/pt-mongodb-summary SlowLogFilePrefix:} Ports:{Min:42000 Max:51999} Debug:true Trace:false Setup:{NodeType: NodeName: MachineID: Distro: ContainerID: ContainerName: NodeModel: Region: Az: Address: MetricsMode: DisableCollectors: CustomLabels: Force:false SkipRegistration:false}}  component=main
INFO[2022-06-15T12:26:05.434+06:00] Starting...                                   component=client
INFO[2022-06-15T12:26:05.434+06:00] Connecting to https://sadmin:***@PMM-Server-IP:443/ ...  component=client
INFO[2022-06-15T12:26:05.435+06:00] Starting local API server on http://127.0.0.1:7777/ ...  component=local-server/JSON
DEBU[2022-06-15T12:26:05.435+06:00] Debug handlers:
        http://127.0.0.1:7777/debug/metrics
        http://127.0.0.1:7777/debug/vars
        http://127.0.0.1:7777/debug/requests
        http://127.0.0.1:7777/debug/events
        http://127.0.0.1:7777/debug/pprof  component=local-server/JSON
DEBU[2022-06-15T12:26:05.435+06:00] Starting gRPC server on http://127.0.0.1:40807/ ...  component=local-server/gRPC
DEBU[2022-06-15T12:26:05.435+06:00] Reflection and channelz are enabled.          component=local-server/gRPC
INFO[2022-06-15T12:26:05.437+06:00] Started.                                      component=local-server/JSON
PANI[2022-06-15T12:26:05.438+06:00] listen tcp 127.0.0.1:7777: bind: address already in use  component=local-server/JSON
panic: (*logrus.Entry) 0xc000246000

goroutine 83 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000becb0, 0xc000720720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /tmp/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc000244930, 0x0, 0xc00082df90, 0x1, 0x1)
        /tmp/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc000244930, 0xc00082df90, 0x1, 0x1)
        /tmp/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/percona/pmm-agent/agentlocal.(*Server).runJSONServer.func2(0xc000244930, 0xc000822000)
        /tmp/go/src/github.com/percona/pmm-agent/agentlocal/agent_local.go:297 +0xf2
created by github.com/percona/pmm-agent/agentlocal.(*Server).runJSONServer
        /tmp/go/src/github.com/percona/pmm-agent/agentlocal/agent_local.go:294 +0xcf3

# pmm-admin status
Failed to get PMM Agent status from local pmm-agent: pmm-agent is not connected to PMM Server.
1 Like
2

When you add mysql, you do not need to provide the server-url and don’t provide config paths unless you have a custom setup. Keep things simple.

Before you add mysql, check that PMM is running pmm-admin status pmm-admin list

listen tcp 127.0.0.1:7777: bind: address already in use

From your log, looks like something is already running on port 7777. Use ss to find out what.

2 Likes
4

This is what I get in /var/log/message in agent Server:

pmm-agent[396812]: #033[36mINFO#033[0m[2022-06-16T12:49:06.462+06:00] Connecting to https://sadmin:***@PMM-Server-IP:443/ ...  #033[36mcomponent#033[0m=client
pmm-agent[396812]: #033[36mINFO#033[0m[2022-06-16T12:49:06.467+06:00] Connected to PMM-Server-IP:443.               #033[36mcomponent#033[0m=client
pmm-agent[396812]: #033[36mINFO#033[0m[2022-06-16T12:49:06.467+06:00] Establishing two-way communication channel ...  #033[36mcomponent#033[0m=client
pmm-agent[396812]: #033[31mERRO#033[0m[2022-06-16T12:49:11.462+06:00] Failed to establish two-way communication channel: context canceled.  #033[31mcomponent#033[0m=client
1 Like
5

Issue is Solved now !!
we were facing this issue while adding new remote agent into PMM-Servers, existing running node was working fine… after updating both PMM-Server & agent version it’s successfully conneceted & communicate with PMM-Server.

Thanks for your time :heart_eyes: :smiling_face_with_three_hearts:

1 Like