Failed to get PMM Server parameters from local pmm-agent: pmm-agent is running, but not set up

Percona Monitoring and Management (PMM)
1

Dear All,
i am configuring the agent on linux node and having below error while doing so, can you please have a look on urgent basis:

[root@stg01 rroot]# rpm -qa | grep pmm
pmm2-client-2.37.0-6.el7.x86_64
[root@stg01 rroot]#
[root@stg01 rroot]# pmm-admin config --force --server-insecure-tls --server-url=https://admin:password@pmm-server-IP
Checking local pmm-agent status…
pmm-agent is running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Internal server error…
[root@stg01 rroot]# pmm-admin list
Failed to get PMM Server parameters from local pmm-agent: pmm-agent is running, but not set up.
Please run pmm-admin config with --server-url flag.

2

Hello @farhan.ext,
Since the config failed, you need to investigate this. Is PMM server running? Can you reach PMM server from this host? (ping, curl, etc). If not, that’s the issue you need to resolve.

3

Hello @Matthew ,
please note that I can telnet the host I want to monitor using PMM server.

4

Telnet over which port? 22? 80? 443? 8080?

stg01 # ping pmm-server.local
stg01 # telnet pmm-server.local 443
stg01 # curl https://pmm-server.local/
5

Hello @matthewb,
below is the telnet and curl log on the suggested ports:

[root@stg01 ~]# telnet pmm-server-IP 443
Trying pmm-server-IP…
Connected to pmm-server-IP.
Escape character is ‘^]’.
Connection closed by foreign host.
[root@stg01 ~]# telnet pmm-server-IP 22
Trying pmm-server-IP…
^C
[root@stg01 ~]# telnet pmm-server-IP 80
Trying pmm-server-IP…
^C
[root@stg01 ~]# telnet pmm-server-IP 8080
Trying pmm-server-IP…
^C
[root@stg01 ~]# curl https://pmm-server-IP/
curl: (60) Peer’s certificate issuer has been marked as not trusted by the user.
More details here: curl - SSL CA Certificates

curl performs SSL certificate verification by default, using a “bundle”
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn’t adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you’d like to turn off curl’s verification of the certificate, use
the -k (or --insecure) option.
[root@stg01 ~]# curl -k https://pmm-server-IP/

302 Found

302 Found

nginx [root@stg01 ~]#
6

Hello @matthewb,
just to update, I have performed the below steps and none is working so far:

1 - we got a space issue on the host where we set-up PMM-SERVER (as volumn created over ‘/’ )
Please note that at this point version was 2.37.0 for both and we added all of our clients around 15 in PMM monitoring.

2 - we removed the container for PMM-SERVER and pmm-data to free up the space

3 - Now when we are trying to load 2.38 image we are always having “unhealthy” in PMM-SERVER cluster health status. And change Admin password command is having issue.

4 - While loading 2.37.0 image PMM-SERVER container became healthy in status and then we are having the issue while configuring the client/s.
Please note that as all of the clients we want to add in monitoring were already added before we ran out of space; none of the client host/s is configuring with the latest PMM-SERVER (2.37.0)

Error:
[root@stg01 ~]# pmm-admin config --force --server-insecure-tls --server-url=https://admin:admin@pmm-server IP
Checking local pmm-agent status…
pmm-agent is not running. Please re-run pmm-agent setup with --config-file flag.

7

Make sure all PMM Clients have also upgraded to same 2.37.0 as PMM Server.

How did you install the clients? Have tried restarting the pmm-agent?

8

I installed the clients using the RPM, below are my installation and configuration commands:

rpm -Uvh pmm2-client-2.37.0-6.el7.x86_64.rpm

and then

pmm-admin config --force --server-insecure-tls --server-url=https://admin:password@pmm-server-IP

9

Hi @matthewb, below is the output if we execute it with --DEBUG flag:
[root@stg01 ~]# pmm-admin config --server-insecure-tls --server-url=http://admin:admin@PMM-SERVER-IP --force --debug
DEBUG 2023-08-03 05:05:29.738642946Z: Running: pmm-agent --server-address=pmm-server-IP:443 --server-username=admin --server-password=PMM@2023 --listen-port=7777 --server-insecure-tls --log-level=warn --debug --log-lines-count=1024 setup --force --metrics-mode=auto 10.40.132.92 generic stg01
DEBUG 2023-08-03 05:05:29.836881596Z: Result: &commands.configResult{Warning:“Warning: PMM Server requires TLS communications with client.”, Output:“Checking local pmm-agent status…\npmm-agent is running.\nRegistering pmm-agent on PMM Server…\nFailed to register pmm-agent on PMM Server: Internal server error…”}
DEBUG 2023-08-03 05:05:29.838997562Z: Error: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0009a8000), Stderr:uint8{0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6d, 0x79, 0x73, 0x71, 0x6c, 0x64, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6d, 0x6f, 0x6e, 0x67, 0x6f, 0x64, 0x62, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x70, 0x6f, 0x73, 0x74, 0x67, 0x72, 0x65, 0x73, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x70, 0x72, 0x6f, 0x78, 0x79, 0x73, 0x71, 0x6c, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x72, 0x64, 0x73, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x61, 0x7a, 0x75, 0x72, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x76, 0x6d, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x35, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x50, 0x4f, 0x53, 0x54, 0x20, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x48, 0x6f, 0x73, 0x74, 0x3a, 0x20, 0x31, 0x32, 0x37, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x31, 0x3a, 0x37, 0x37, 0x37, 0x37, 0xd, 0xa, 0x55, 0x73, 0x65, 0x72, 0x2d, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x3a, 0x20, 0x47, 0x6f, 0x2d, 0x68, 0x74, 0x74, 0x70, 0x2d, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x33, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x67, 0x7a, 0x69, 0x70, 0xd, 0xa, 0xd, 0xa, 0x7b, 0x7d, 0xa, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x37, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4f, 0x4b, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x32, 0x33, 0x33, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x54, 0x68, 0x75, 0x2c, 0x20, 0x30, 0x33, 0x20, 0x41, 0x75, 0x67, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x30, 0x35, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x20, 0x47, 0x4d, 0x54, 0xd, 0xa, 0x47, 0x72, 0x70, 0x63, 0x2d, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2d, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x67, 0x72, 0x70, 0x63, 0xd, 0xa, 0xd, 0xa, 0x7b, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x72, 0x75, 0x6e, 0x73, 0x5f, 0x6f, 0x6e, 0x5f, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x3a, 0x20, 0x6e, 0x75, 0x6c, 0x6c, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x3a, 0x20, 0x5b, 0x5d, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x70, 0x61, 0x74, 0x68, 0x22, 0x3a, 0x20, 0x22, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x2f, 0x70, 0x6d, 0x6d, 0x2d, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x79, 0x61, 0x6d, 0x6c, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x3a, 0x20, 0x22, 0x32, 0x2e, 0x33, 0x38, 0x2e, 0x31, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x75, 0x70, 0x74, 0x69, 0x6d, 0x65, 0x22, 0x3a, 0x20, 0x30, 0xa, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x37, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x20, 0x3c, 0x6e, 0x69, 0x6c, 0x3e, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x37, 0x38, 0x38, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x50, 0x4f, 0x53, 0x54, 0x20, 0x2f, 0x76, 0x31, 0x2f, 0x6d, 0x61, 0x6e, 0x61, 0x67, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x2f, 0x4e, 0x6f, 0x64, 0x65, 0x2f, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x48, 0x6f, 0x73, 0x74, 0x3a, 0x20, 0x31, 0x30, 0x2e, 0x34, 0x30, 0x2e, 0x31, 0x33, 0x31, 0x2e, 0x32, 0x31, 0x36, 0x3a, 0x34, 0x34, 0x33, 0xd, 0xa, 0x55, 0x73, 0x65, 0x72, 0x2d, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x3a, 0x20, 0x47, 0x6f, 0x2d, 0x68, 0x74, 0x74, 0x70, 0x2d, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x32, 0x32, 0x32, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x42, 0x61, 0x73, 0x69, 0x63, 0x20, 0x59, 0x57, 0x52, 0x74, 0x61, 0x57, 0x34, 0x36, 0x55, 0x45, 0x31, 0x4e, 0x51, 0x44, 0x49, 0x77, 0x4d, 0x6a, 0x4d, 0x3d, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x67, 0x7a, 0x69, 0x70, 0xd, 0xa, 0xd, 0xa, 0x7b, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3a, 0x22, 0x47, 0x45, 0x4e, 0x45, 0x52, 0x49, 0x43, 0x5f, 0x4e, 0x4f, 0x44, 0x45, 0x22, 0x2c, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x66, 0x74, 0x73, 0x68, 0x61, 0x74, 0x73, 0x74, 0x30, 0x33, 0x22, 0x2c, 0x22, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x22, 0x3a, 0x22, 0x31, 0x30, 0x2e, 0x34, 0x30, 0x2e, 0x31, 0x33, 0x32, 0x2e, 0x39, 0x32, 0x22, 0x2c, 0x22, 0x6d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x22, 0x2f, 0x6d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x5f, 0x69, 0x64, 0x2f, 0x38, 0x39, 0x65, 0x37, 0x66, 0x63, 0x36, 0x34, 0x31, 0x37, 0x37, 0x34, 0x34, 0x32, 0x37, 0x36, 0x62, 0x66, 0x37, 0x32, 0x65, 0x35, 0x37, 0x31, 0x39, 0x66, 0x35, 0x34, 0x39, 0x36, 0x63, 0x64, 0x22, 0x2c, 0x22, 0x64, 0x69, 0x73, 0x74, 0x72, 0x6f, 0x22, 0x3a, 0x22, 0x6c, 0x69, 0x6e, 0x75, 0x78, 0x22, 0x2c, 0x22, 0x72, 0x65, 0x72, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x22, 0x3a, 0x74, 0x72, 0x75, 0x65, 0x2c, 0x22, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x5f, 0x6d, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x22, 0x41, 0x55, 0x54, 0x4f, 0x22, 0x2c, 0x22, 0x64, 0x69, 0x73, 0x61, 0x62, 0x6c, 0x65, 0x5f, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73, 0x22, 0x3a, 0x6e, 0x75, 0x6c, 0x6c, 0x7d, 0xa, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x38, 0x33, 0x32, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x35, 0x30, 0x30, 0x20, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x20, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x45, 0x72, 0x72, 0x6f, 0x72, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x39, 0x32, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x6b, 0x65, 0x65, 0x70, 0x2d, 0x61, 0x6c, 0x69, 0x76, 0x65, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x54, 0x68, 0x75, 0x2c, 0x20, 0x30, 0x33, 0x20, 0x41, 0x75, 0x67, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x30, 0x35, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x20, 0x47, 0x4d, 0x54, 0xd, 0xa, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x3a, 0x20, 0x6e, 0x67, 0x69, 0x6e, 0x78, 0xd, 0xa, 0x53, 0x74, 0x72, 0x69, 0x63, 0x74, 0x2d, 0x54, 0x72, 0x61, 0x6e, 0x73, 0x70, 0x6f, 0x72, 0x74, 0x2d, 0x53, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x3a, 0x20, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x36, 0x33, 0x30, 0x37, 0x32, 0x30, 0x30, 0x30, 0x3b, 0x20, 0x69, 0x6e, 0x63, 0x6c, 0x75, 0x64, 0x65, 0x53, 0x75, 0x62, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x73, 0x3b, 0xd, 0xa, 0xd, 0xa, 0x7b, 0xa, 0x20, 0x20, 0x22, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x22, 0x3a, 0x20, 0x22, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x2e, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x31, 0x33, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x2e, 0x22, 0xa, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x33, 0x54, 0x30, 0x39, 0x3a, 0x30, 0x35, 0x3a, 0x32, 0x39, 0x2e, 0x38, 0x33, 0x32, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x20, 0x26, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x4e, 0x6f, 0x64, 0x65, 0x44, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x7b, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x43, 0x6f, 0x64, 0x65, 0x3a, 0x35, 0x30, 0x30, 0x2c, 0x20, 0x50, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x3a, 0x28, 0x2a, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x4e, 0x6f, 0x64, 0x65, 0x44, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x42, 0x6f, 0x64, 0x79, 0x29, 0x28, 0x30, 0x78, 0x63, 0x30, 0x30, 0x30, 0x31, 0x33, 0x37, 0x39, 0x38, 0x30, 0x29, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa}}
Warning: PMM Server requires TLS communications with client.
Checking local pmm-agent status…
pmm-agent is running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Internal server error…
DEBUG 2023-08-03 05:05:29.839414084Z: exit status 1, stderr:
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=setup
INFO[2023-08-03T09:05:29.784+04:00] Using /usr/local/percona/pmm2/exporters/vmagent component=setup
DEBU[2023-08-03T09:05:29.785+04:00] POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}
component=setup
DEBU[2023-08-03T09:05:29.787+04:00] HTTP/1.1 200 OK
Content-Length: 233
Content-Type: application/json
Date: Thu, 03 Aug 2023 05:05:29 GMT
Grpc-Metadata-Content-Type: application/grpc

{
“agent_id”: “”,
“runs_on_node_id”: “”,
“server_info”: null,
“agents_info”: ,
“config_filepath”: “/usr/local/percona/pmm2/config/pmm-agent.yaml”,
“agent_version”: “2.38.1”,
“node_name”: “”,
“connection_uptime”: 0
} component=setup
DEBU[2023-08-03T09:05:29.787+04:00] Status error: component=setup
DEBU[2023-08-03T09:05:29.788+04:00] POST /v1/management/Node/Register HTTP/1.1
Host: pmm-server-IP:443
User-Agent: Go-http-client/1.1
Content-Length: 222
Accept: application/json
Authorization: Basic YWRtaW46UE1NQDIwMjM=
Content-Type: application/json
Accept-Encoding: gzip

{“node_type”:“GENERIC_NODE”,“node_name”:“stg01”,“address”:“STG-IP”,“machine_id”:“/machine_id/89e7fc6417744276bf72e5719f5496cd”,“distro”:“linux”,“reregister”:true,“metrics_mode”:“AUTO”,“disable_collectors”:null}
component=setup
DEBU[2023-08-03T09:05:29.832+04:00] HTTP/1.1 500 Internal Server Error
Content-Length: 92
Connection: keep-alive
Content-Type: application/json
Date: Thu, 03 Aug 2023 05:05:29 GMT
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;

{
“error”: “Internal server error.”,
“code”: 13,
“message”: “Internal server error.”
} component=setup
DEBU[2023-08-03T09:05:29.832+04:00] Register error: &node.RegisterNodeDefault{_statusCode:500, Payload:(*node.RegisterNodeDefaultBody)(0xc000137980)} component=setup
[root@stg01 ~]#

10

@matthewb and this is what I am having on client host:

[root@stg01 ~]# pmm-admin status
Agent ID : /agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63
Node ID :
Node name:

PMM Server:
URL : https://pmm-server-IP:443/
Version:

PMM Client:
Connected : false
Connection uptime: 0
pmm-admin version: 2.37.0
pmm-agent version: 2.37.0
Agents:

[root@stg01 ~]# service pmm-agent status
Redirecting to /bin/systemctl status pmm-agent.service
● pmm-agent.service - pmm-agent
Loaded: loaded (/usr/lib/systemd/system/pmm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-31 11:53:30 +04; 2 days ago
Main PID: 4672 (pmm-agent)
Tasks: 26
Memory: 59.2M
CGroup: /system.slice/pmm-agent.service
└─4672 /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml

Aug 03 09:16:27 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:16:27.343+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:16:44 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:16:44.241+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:16:57 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:16:57.444+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:17:12 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:17:12.753+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:17:26 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:17:26.077+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:17:38 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:17:38.194+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:17:54 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:17:54.646+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:18:06 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:18:06.131+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:18:20 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:18:20.235+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
Aug 03 09:18:38 stg01.cbuae.gov pmm-agent[4672]: ERRO[2023-08-03T09:18:38.032+04:00] Failed to establish two-way communication channel: No Agent with ID “/agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63”. component=client
[root@stg01 ~]# cat /usr/local/percona/pmm2/config/pmm-agent.yaml

Updated by pmm-agent setup.

id: /agent_id/ac28d699-bdb5-4808-927e-e614e71f1a63
listen-address: 127.0.0.1
listen-port: 7777
server:
address: pmm-server-IP:443
username: api_key
password: eyJrIjoiQ1lHbWFObkJBWUQycG83ZlkybFRSNW5aTlFJUHNDeGoiLCJuIjoicG1tLWFnZW50LXdzaWRicHJkMDEuY2J1YWUuZ292LTUwNDczOTI4NTgxODEwOTA1NjAiLCJpZCI6MX0=
insecure-tls: true
paths:
paths_base: /usr/local/percona/pmm2
exporters_base: /usr/local/percona/pmm2/exporters
node_exporter: /usr/local/percona/pmm2/exporters/node_exporter
mysqld_exporter: /usr/local/percona/pmm2/exporters/mysqld_exporter
mongodb_exporter: /usr/local/percona/pmm2/exporters/mongodb_exporter
postgres_exporter: /usr/local/percona/pmm2/exporters/postgres_exporter
proxysql_exporter: /usr/local/percona/pmm2/exporters/proxysql_exporter
rds_exporter: /usr/local/percona/pmm2/exporters/rds_exporter
azure_exporter: /usr/local/percona/pmm2/exporters/azure_exporter
vmagent: /usr/local/percona/pmm2/exporters/vmagent
tempdir: /tmp
pt_summary: /usr/local/percona/pmm2/tools/pt-summary
pt_pg_summary: /usr/local/percona/pmm2/tools/pt-pg-summary
pt_mysql_summary: /usr/local/percona/pmm2/tools/pt-mysql-summary
pt_mongodb_summary: /usr/local/percona/pmm2/tools/pt-mongodb-summary
ports:
min: 42000
max: 51999
log-level: warn
debug: false
trace: false
loglinescount: 1024
window-connected-time: 1h0m0s
[root@stg01 ~]#

11

--server-url= should be https, not http as shown in your output above. It even says so in the output:

DEBUG 2023-08-03 05:05:29.836881596Z: Result: &commands.configResult{
Warning:“Warning: PMM Server requires TLS communications with client.
12

Dear @matthewb tried with https as well and below is the log:

[root@stg01 ~]# pmm-admin config --force --server-insecure-tls --server-url=https://admin:admin@pmm-server-IP --debug
DEBUG 2023-08-04 06:33:21.341321391Z: Running: pmm-agent --server-address=pmm-server-IP:443 --server-username=admin --server-password=PMM@2023 --listen-port=7777 --server-insecure-tls --log-level=warn --debug --log-lines-count=1024 setup --force --metrics-mode=auto 10.40.132.90 generic stg01
DEBUG 2023-08-04 06:33:21.419013821Z: Result: &commands.configResult{Warning:“”, Output:“Checking local pmm-agent status…\npmm-agent is running.\nRegistering pmm-agent on PMM Server…\nFailed to register pmm-agent on PMM Server: Internal server error…”}
DEBUG 2023-08-04 06:33:21.419533227Z: Error: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc000160000), Stderr:uint8{0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6d, 0x79, 0x73, 0x71, 0x6c, 0x64, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6d, 0x6f, 0x6e, 0x67, 0x6f, 0x64, 0x62, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x70, 0x6f, 0x73, 0x74, 0x67, 0x72, 0x65, 0x73, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x70, 0x72, 0x6f, 0x78, 0x79, 0x73, 0x71, 0x6c, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x72, 0x64, 0x73, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x61, 0x7a, 0x75, 0x72, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x76, 0x6d, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x36, 0x39, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x50, 0x4f, 0x53, 0x54, 0x20, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x48, 0x6f, 0x73, 0x74, 0x3a, 0x20, 0x31, 0x32, 0x37, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x31, 0x3a, 0x37, 0x37, 0x37, 0x37, 0xd, 0xa, 0x55, 0x73, 0x65, 0x72, 0x2d, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x3a, 0x20, 0x47, 0x6f, 0x2d, 0x68, 0x74, 0x74, 0x70, 0x2d, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x33, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x67, 0x7a, 0x69, 0x70, 0xd, 0xa, 0xd, 0xa, 0x7b, 0x7d, 0xa, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x37, 0x31, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4f, 0x4b, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x32, 0x33, 0x33, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x46, 0x72, 0x69, 0x2c, 0x20, 0x30, 0x34, 0x20, 0x41, 0x75, 0x67, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x30, 0x36, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x20, 0x47, 0x4d, 0x54, 0xd, 0xa, 0x47, 0x72, 0x70, 0x63, 0x2d, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2d, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x67, 0x72, 0x70, 0x63, 0xd, 0xa, 0xd, 0xa, 0x7b, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x72, 0x75, 0x6e, 0x73, 0x5f, 0x6f, 0x6e, 0x5f, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x3a, 0x20, 0x6e, 0x75, 0x6c, 0x6c, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x3a, 0x20, 0x5b, 0x5d, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x70, 0x61, 0x74, 0x68, 0x22, 0x3a, 0x20, 0x22, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x2f, 0x70, 0x6d, 0x6d, 0x2d, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x79, 0x61, 0x6d, 0x6c, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x3a, 0x20, 0x22, 0x32, 0x2e, 0x33, 0x37, 0x2e, 0x30, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x75, 0x70, 0x74, 0x69, 0x6d, 0x65, 0x22, 0x3a, 0x20, 0x30, 0xa, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x37, 0x31, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x20, 0x3c, 0x6e, 0x69, 0x6c, 0x3e, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x33, 0x37, 0x31, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x50, 0x4f, 0x53, 0x54, 0x20, 0x2f, 0x76, 0x31, 0x2f, 0x6d, 0x61, 0x6e, 0x61, 0x67, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x2f, 0x4e, 0x6f, 0x64, 0x65, 0x2f, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x48, 0x6f, 0x73, 0x74, 0x3a, 0x20, 0x31, 0x30, 0x2e, 0x34, 0x30, 0x2e, 0x31, 0x33, 0x31, 0x2e, 0x32, 0x31, 0x36, 0x3a, 0x34, 0x34, 0x33, 0xd, 0xa, 0x55, 0x73, 0x65, 0x72, 0x2d, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x3a, 0x20, 0x47, 0x6f, 0x2d, 0x68, 0x74, 0x74, 0x70, 0x2d, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x32, 0x32, 0x32, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x42, 0x61, 0x73, 0x69, 0x63, 0x20, 0x59, 0x57, 0x52, 0x74, 0x61, 0x57, 0x34, 0x36, 0x55, 0x45, 0x31, 0x4e, 0x51, 0x44, 0x49, 0x77, 0x4d, 0x6a, 0x4d, 0x3d, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x67, 0x7a, 0x69, 0x70, 0xd, 0xa, 0xd, 0xa, 0x7b, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3a, 0x22, 0x47, 0x45, 0x4e, 0x45, 0x52, 0x49, 0x43, 0x5f, 0x4e, 0x4f, 0x44, 0x45, 0x22, 0x2c, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x66, 0x74, 0x73, 0x68, 0x61, 0x74, 0x73, 0x74, 0x30, 0x31, 0x22, 0x2c, 0x22, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x22, 0x3a, 0x22, 0x31, 0x30, 0x2e, 0x34, 0x30, 0x2e, 0x31, 0x33, 0x32, 0x2e, 0x39, 0x30, 0x22, 0x2c, 0x22, 0x6d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x22, 0x2f, 0x6d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x5f, 0x69, 0x64, 0x2f, 0x38, 0x39, 0x65, 0x37, 0x66, 0x63, 0x36, 0x34, 0x31, 0x37, 0x37, 0x34, 0x34, 0x32, 0x37, 0x36, 0x62, 0x66, 0x37, 0x32, 0x65, 0x35, 0x37, 0x31, 0x39, 0x66, 0x35, 0x34, 0x39, 0x36, 0x63, 0x64, 0x22, 0x2c, 0x22, 0x64, 0x69, 0x73, 0x74, 0x72, 0x6f, 0x22, 0x3a, 0x22, 0x6c, 0x69, 0x6e, 0x75, 0x78, 0x22, 0x2c, 0x22, 0x72, 0x65, 0x72, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x22, 0x3a, 0x74, 0x72, 0x75, 0x65, 0x2c, 0x22, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x5f, 0x6d, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x22, 0x41, 0x55, 0x54, 0x4f, 0x22, 0x2c, 0x22, 0x64, 0x69, 0x73, 0x61, 0x62, 0x6c, 0x65, 0x5f, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73, 0x22, 0x3a, 0x6e, 0x75, 0x6c, 0x6c, 0x7d, 0xa, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x34, 0x31, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x35, 0x30, 0x30, 0x20, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x20, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x45, 0x72, 0x72, 0x6f, 0x72, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x39, 0x32, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x6b, 0x65, 0x65, 0x70, 0x2d, 0x61, 0x6c, 0x69, 0x76, 0x65, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x46, 0x72, 0x69, 0x2c, 0x20, 0x30, 0x34, 0x20, 0x41, 0x75, 0x67, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x30, 0x36, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x20, 0x47, 0x4d, 0x54, 0xd, 0xa, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x3a, 0x20, 0x6e, 0x67, 0x69, 0x6e, 0x78, 0xd, 0xa, 0x53, 0x74, 0x72, 0x69, 0x63, 0x74, 0x2d, 0x54, 0x72, 0x61, 0x6e, 0x73, 0x70, 0x6f, 0x72, 0x74, 0x2d, 0x53, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x3a, 0x20, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x36, 0x33, 0x30, 0x37, 0x32, 0x30, 0x30, 0x30, 0x3b, 0x20, 0x69, 0x6e, 0x63, 0x6c, 0x75, 0x64, 0x65, 0x53, 0x75, 0x62, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x73, 0x3b, 0xd, 0xa, 0xd, 0xa, 0x7b, 0xa, 0x20, 0x20, 0x22, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x22, 0x3a, 0x20, 0x22, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x2e, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x20, 0x31, 0x33, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x22, 0x3a, 0x20, 0x22, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x2e, 0x22, 0xa, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x33, 0x2d, 0x30, 0x38, 0x2d, 0x30, 0x34, 0x54, 0x31, 0x30, 0x3a, 0x33, 0x33, 0x3a, 0x32, 0x31, 0x2e, 0x34, 0x31, 0x34, 0x2b, 0x30, 0x34, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x20, 0x26, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x4e, 0x6f, 0x64, 0x65, 0x44, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x7b, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x43, 0x6f, 0x64, 0x65, 0x3a, 0x35, 0x30, 0x30, 0x2c, 0x20, 0x50, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x3a, 0x28, 0x2a, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x4e, 0x6f, 0x64, 0x65, 0x44, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x42, 0x6f, 0x64, 0x79, 0x29, 0x28, 0x30, 0x78, 0x63, 0x30, 0x30, 0x30, 0x32, 0x35, 0x64, 0x61, 0x37, 0x30, 0x29, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa}}
Checking local pmm-agent status…
pmm-agent is running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Internal server error…
DEBUG 2023-08-04 06:33:21.419866866Z: exit status 1, stderr:
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=setup
INFO[2023-08-04T10:33:21.369+04:00] Using /usr/local/percona/pmm2/exporters/vmagent component=setup
DEBU[2023-08-04T10:33:21.369+04:00] POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}
component=setup
DEBU[2023-08-04T10:33:21.371+04:00] HTTP/1.1 200 OK
Content-Length: 233
Content-Type: application/json
Date: Fri, 04 Aug 2023 06:33:21 GMT
Grpc-Metadata-Content-Type: application/grpc

{
“agent_id”: “”,
“runs_on_node_id”: “”,
“server_info”: null,
“agents_info”: ,
“config_filepath”: “/usr/local/percona/pmm2/config/pmm-agent.yaml”,
“agent_version”: “2.37.0”,
“node_name”: “”,
“connection_uptime”: 0
} component=setup
DEBU[2023-08-04T10:33:21.371+04:00] Status error: component=setup
DEBU[2023-08-04T10:33:21.371+04:00] POST /v1/management/Node/Register HTTP/1.1
Host: pmm-server-IP:443
User-Agent: Go-http-client/1.1
Content-Length: 222
Accept: application/json
Authorization: Basic YWRtaW46UE1NQDIwMjM=
Content-Type: application/json
Accept-Encoding: gzip

{“node_type”:“GENERIC_NODE”,“node_name”:“stg01”,“address”:“10.40.132.90”,“machine_id”:“/machine_id/89e7fc6417744276bf72e5719f5496cd”,“distro”:“linux”,“reregister”:true,“metrics_mode”:“AUTO”,“disable_collectors”:null}
component=setup
DEBU[2023-08-04T10:33:21.414+04:00] HTTP/1.1 500 Internal Server Error
Content-Length: 92
Connection: keep-alive
Content-Type: application/json
Date: Fri, 04 Aug 2023 06:33:21 GMT
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;

{
“error”: “Internal server error.”,
“code”: 13,
“message”: “Internal server error.”
} component=setup
DEBU[2023-08-04T10:33:21.414+04:00] Register error: &node.RegisterNodeDefault{_statusCode:500, Payload:(*node.RegisterNodeDefaultBody)(0xc00025da70)} component=setup
[root@stg01 ~]#

13

Dear @matthewb ,
can you please suggest the way forward from here?

14

Honestly, at this point, I would just uninstall everything and start over. There clearly was a missed step somewhere and rather than waste time debugging, easier to wipe and start over.

15

Dear @matthewb,
wipe out PMM-Server container or client rpm?

I have tried couple of times removing rpm of PMM-Client and re-install but facing the same problem.
can you suggest the steps for clean removal of client?

16

I would do both, PMM Server and Client. You seem to have some deep issue at hand. Do you have SELinux running? Have you tried stopping that to see if it works?

Basic question, are you able to access the Web UI of PMM server in your current state? If so, then the issue is between your two servers’ networking. If you cannot access the UI, then you have a basic misconfiguration of PMM Server.

17

Dear @matthewb,
I figured it out by myself.

to remove all the images and network we need to run system prune command and rmi images then load the image and continue further and it worked.
Thanks.