Unable to register client with server

Percona Monitoring and Management (PMM) PMM 2.x
1

I am attempting to install PMM 2.31.0. I get to the point where I am attempting to register the client with the server and I always get and error. I should mention that my postgresql database, the pmm server and pmm client are all running on the same server. I should also mention when attempting to bring the https://XX.XX.XX.XX web page it fails. However, using http://XX.XX.XX.XX succeeds and I can log into PMM using the admin/admin username and password.

Below is a listing of the steps I took to register the client:

[hdrews2@cougar-pg-01 ~]$ sudo -A -i docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-109.gitcccb291.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      cccb291/1.13.1
 Built:           Tue Mar  3 17:21:24 2020
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-109.gitcccb291.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      cccb291/1.13.1
 Built:           Tue Mar  3 17:21:24 2020
 OS/Arch:         linux/amd64
 Experimental:    false


[hdrews2@cougar-pg-01 ~]$ sudo -A -i docker pull percona/pmm-server:2
Trying to pull repository docker.io/percona/pmm-server ...
2: Pulling from docker.io/percona/pmm-server
2d473b07cdd5: Pull complete
5c1be54dc1c3: Pull complete
Digest: sha256:69a07b0c75d47cdaf29636dfca32138c80e85b9572f52de1cfe0eb75381ac795
Status: Downloaded newer image for docker.io/percona/pmm-server:2


[hdrews2@cougar-pg-01 ~]$ sudo -A -i docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
docker.io/percona/pmm-server   2                   ddec0521f4bd        4 weeks ago         1.98 GB


[hdrews2@cougar-pg-01 ~]$ sudo -A -i docker volume create pmm-data
pmm-data


[hdrews2@cougar-pg-01 ~]$ sudo -A -i docker run --detach \
>               --restart always \
>               --publish 443:443 \
>               --publish 80:80 \
>               --volume pmm-data:/srv \
>               --name pmm-server percona/pmm-server:2
c0a8e7eb83317cfdca9e4e4fc29d5217ff65dbe69c2a35d0c826c875f4ce5d0d


[hdrews2@cougar-pg-01 ~]$ sudo -A -i docker container ls
CONTAINER ID        IMAGE                  COMMAND                CREATED             STATUS                                     PORTS                                      NAMES
c0a8e7eb8331        percona/pmm-server:2   "/opt/entrypoint.sh"   7 seconds ago       Up Less than a second (health: starting)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   pmm-server


[hdrews2@cougar-pg-01 ~]$ sudo -A -i pmm-admin --version
ProjectName: pmm-admin
Version: 2.31.0
PMMVersion: 2.31.0
Timestamp: 2022-09-23 13:21:08 (UTC)
FullCommit: 86df37b0a6913391ee28b100a46761174052097b


[hdrews2@cougar-pg-01 ~]$ sudo -A -i pmm-admin config --server-insecure-tls --server-url=https://admin:admin@XX.XX.XX.XX:443 --debug
DEBUG 2022-10-27 13:18:39.892740981Z: Running: pmm-agent --server-address=XX.XX.XX.XX:443 --server-username=admin --server-password=admin --listen-port=7777 --server-insecure-tls --log-level=warn --debug --log-lines-count=1024 setup --metrics-mode=auto XX.XX.XX.XX generic cougar-pg-01
DEBUG 2022-10-27 13:18:39.936961582Z: Result: &commands.configResult{Warning:"", Output:"Checking local pmm-agent status...\npmm-agent is running.\nRegistering pmm-agent on PMM Server...\nFailed to register pmm-agent on PMM Server: Post \"https://XX.XX.XX.XX:443/v1/management/Node/Register\": dial tcp XX.XX.XX.XX:443: connect: connection refused."}
DEBUG 2022-10-27 13:18:39.937709863Z: Error: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc000368000), Stderr:[]uint8{0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6d, 0x79, 0x73, 0x71, 0x6c, 0x64, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x6d, 0x6f, 0x6e, 0x67, 0x6f, 0x64, 0x62, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x70, 0x6f, 0x73, 0x74, 0x67, 0x72, 0x65, 0x73, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x70, 0x72, 0x6f, 0x78, 0x79, 0x73, 0x71, 0x6c, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x72, 0x64, 0x73, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x61, 0x7a, 0x75, 0x72, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x49, 0x4e, 0x46, 0x4f, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x36, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x55, 0x73, 0x69, 0x6e, 0x67, 0x20, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x65, 0x78, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x72, 0x73, 0x2f, 0x76, 0x6d, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x36, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x32, 0x37, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x50, 0x4f, 0x53, 0x54, 0x20, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x48, 0x6f, 0x73, 0x74, 0x3a, 0x20, 0x31, 0x32, 0x37, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x31, 0x3a, 0x37, 0x37, 0x37, 0x37, 0xd, 0xa, 0x55, 0x73, 0x65, 0x72, 0x2d, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x3a, 0x20, 0x47, 0x6f, 0x2d, 0x68, 0x74, 0x74, 0x70, 0x2d, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x33, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x67, 0x7a, 0x69, 0x70, 0xd, 0xa, 0xd, 0xa, 0x7b, 0x7d, 0xa, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x33, 0x31, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4f, 0x4b, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x32, 0x34, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x54, 0x68, 0x75, 0x2c, 0x20, 0x32, 0x37, 0x20, 0x4f, 0x63, 0x74, 0x20, 0x32, 0x30, 0x32, 0x32, 0x20, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x20, 0x47, 0x4d, 0x54, 0xd, 0xa, 0x47, 0x72, 0x70, 0x63, 0x2d, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2d, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x67, 0x72, 0x70, 0x63, 0xd, 0xa, 0xd, 0xa, 0x7b, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x20, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x72, 0x75, 0x6e, 0x73, 0x5f, 0x6f, 0x6e, 0x5f, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x20, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x3a, 0x20, 0x20, 0x6e, 0x75, 0x6c, 0x6c, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x3a, 0x20, 0x20, 0x5b, 0x5d, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x70, 0x61, 0x74, 0x68, 0x22, 0x3a, 0x20, 0x20, 0x22, 0x2f, 0x75, 0x73, 0x72, 0x2f, 0x6c, 0x6f, 0x63, 0x61, 0x6c, 0x2f, 0x70, 0x65, 0x72, 0x63, 0x6f, 0x6e, 0x61, 0x2f, 0x70, 0x6d, 0x6d, 0x32, 0x2f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x2f, 0x70, 0x6d, 0x6d, 0x2d, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x79, 0x61, 0x6d, 0x6c, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x3a, 0x20, 0x20, 0x22, 0x32, 0x2e, 0x33, 0x31, 0x2e, 0x30, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x20, 0x20, 0x22, 0x22, 0x2c, 0xa, 0x20, 0x20, 0x22, 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x75, 0x70, 0x74, 0x69, 0x6d, 0x65, 0x22, 0x3a, 0x20, 0x20, 0x30, 0xa, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x33, 0x32, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x20, 0x3c, 0x6e, 0x69, 0x6c, 0x3e, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x33, 0x34, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x50, 0x4f, 0x53, 0x54, 0x20, 0x2f, 0x76, 0x31, 0x2f, 0x6d, 0x61, 0x6e, 0x61, 0x67, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x2f, 0x4e, 0x6f, 0x64, 0x65, 0x2f, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x48, 0x6f, 0x73, 0x74, 0x3a, 0x20, 0x31, 0x30, 0x2e, 0x32, 0x33, 0x38, 0x2e, 0x32, 0x35, 0x2e, 0x36, 0x31, 0x3a, 0x34, 0x34, 0x33, 0xd, 0xa, 0x55, 0x73, 0x65, 0x72, 0x2d, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x3a, 0x20, 0x47, 0x6f, 0x2d, 0x68, 0x74, 0x74, 0x70, 0x2d, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2f, 0x31, 0x2e, 0x31, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x3a, 0x20, 0x32, 0x30, 0x36, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x42, 0x61, 0x73, 0x69, 0x63, 0x20, 0x59, 0x57, 0x52, 0x74, 0x61, 0x57, 0x34, 0x36, 0x59, 0x57, 0x52, 0x74, 0x61, 0x57, 0x34, 0x3d, 0xd, 0xa, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0xd, 0xa, 0x41, 0x63, 0x63, 0x65, 0x70, 0x74, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x67, 0x7a, 0x69, 0x70, 0xd, 0xa, 0xd, 0xa, 0x7b, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3a, 0x22, 0x47, 0x45, 0x4e, 0x45, 0x52, 0x49, 0x43, 0x5f, 0x4e, 0x4f, 0x44, 0x45, 0x22, 0x2c, 0x22, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x63, 0x6f, 0x75, 0x67, 0x61, 0x72, 0x2d, 0x70, 0x67, 0x2d, 0x30, 0x31, 0x22, 0x2c, 0x22, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x22, 0x3a, 0x22, 0x31, 0x30, 0x2e, 0x32, 0x33, 0x38, 0x2e, 0x32, 0x35, 0x2e, 0x36, 0x31, 0x22, 0x2c, 0x22, 0x6d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x22, 0x2f, 0x6d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x5f, 0x69, 0x64, 0x2f, 0x33, 0x39, 0x65, 0x34, 0x37, 0x38, 0x36, 0x37, 0x63, 0x34, 0x33, 0x61, 0x34, 0x66, 0x35, 0x62, 0x61, 0x34, 0x36, 0x32, 0x39, 0x34, 0x30, 0x62, 0x34, 0x32, 0x39, 0x30, 0x64, 0x30, 0x62, 0x65, 0x22, 0x2c, 0x22, 0x64, 0x69, 0x73, 0x74, 0x72, 0x6f, 0x22, 0x3a, 0x22, 0x6c, 0x69, 0x6e, 0x75, 0x78, 0x22, 0x2c, 0x22, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x5f, 0x6d, 0x6f, 0x64, 0x65, 0x22, 0x3a, 0x22, 0x41, 0x55, 0x54, 0x4f, 0x22, 0x2c, 0x22, 0x64, 0x69, 0x73, 0x61, 0x62, 0x6c, 0x65, 0x5f, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73, 0x22, 0x3a, 0x6e, 0x75, 0x6c, 0x6c, 0x7d, 0xa, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x44, 0x45, 0x42, 0x55, 0x1b, 0x5b, 0x30, 0x6d, 0x5b, 0x32, 0x30, 0x32, 0x32, 0x2d, 0x31, 0x30, 0x2d, 0x32, 0x37, 0x54, 0x31, 0x33, 0x3a, 0x31, 0x38, 0x3a, 0x33, 0x39, 0x2e, 0x39, 0x33, 0x35, 0x2b, 0x30, 0x30, 0x3a, 0x30, 0x30, 0x5d, 0x20, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x20, 0x26, 0x75, 0x72, 0x6c, 0x2e, 0x45, 0x72, 0x72, 0x6f, 0x72, 0x7b, 0x4f, 0x70, 0x3a, 0x22, 0x50, 0x6f, 0x73, 0x74, 0x22, 0x2c, 0x20, 0x55, 0x52, 0x4c, 0x3a, 0x22, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x31, 0x30, 0x2e, 0x32, 0x33, 0x38, 0x2e, 0x32, 0x35, 0x2e, 0x36, 0x31, 0x3a, 0x34, 0x34, 0x33, 0x2f, 0x76, 0x31, 0x2f, 0x6d, 0x61, 0x6e, 0x61, 0x67, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x2f, 0x4e, 0x6f, 0x64, 0x65, 0x2f, 0x52, 0x65, 0x67, 0x69, 0x73, 0x74, 0x65, 0x72, 0x22, 0x2c, 0x20, 0x45, 0x72, 0x72, 0x3a, 0x28, 0x2a, 0x6e, 0x65, 0x74, 0x2e, 0x4f, 0x70, 0x45, 0x72, 0x72, 0x6f, 0x72, 0x29, 0x28, 0x30, 0x78, 0x63, 0x30, 0x30, 0x30, 0x32, 0x31, 0x32, 0x30, 0x66, 0x30, 0x29, 0x7d, 0x20, 0x20, 0x1b, 0x5b, 0x33, 0x37, 0x6d, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e, 0x74, 0x1b, 0x5b, 0x30, 0x6d, 0x3d, 0x73, 0x65, 0x74, 0x75, 0x70, 0xa}}
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Post "https://XX.XX.XX.XX:443/v1/management/Node/Register": dial tcp XX.XX.XX.XX:443: connect: connection refused.
DEBUG 2022-10-27 13:18:39.938556019Z: exit status 1, stderr:
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
INFO[2022-10-27T13:18:39.926+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
DEBU[2022-10-27T13:18:39.927+00:00] POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}
  component=setup
DEBU[2022-10-27T13:18:39.931+00:00] HTTP/1.1 200 OK
Content-Length: 241
Content-Type: application/json
Date: Thu, 27 Oct 2022 13:18:39 GMT
Grpc-Metadata-Content-Type: application/grpc

{
  "agent_id":  "",
  "runs_on_node_id":  "",
  "server_info":  null,
  "agents_info":  [],
  "config_filepath":  "/usr/local/percona/pmm2/config/pmm-agent.yaml",
  "agent_version":  "2.31.0",
  "node_name":  "",
  "connection_uptime":  0
}  component=setup
DEBU[2022-10-27T13:18:39.932+00:00] Status error: <nil>                           component=setup
DEBU[2022-10-27T13:18:39.934+00:00] POST /v1/management/Node/Register HTTP/1.1
Host: XX.XX.XX.XX:443
User-Agent: Go-http-client/1.1
Content-Length: 206
Accept: application/json
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
Accept-Encoding: gzip

{"node_type":"GENERIC_NODE","node_name":"cougar-pg-01","address":"XX.XX.XX.XX","machine_id":"/machine_id/39e47867c43a4f5ba462940b4290d0be","distro":"linux","metrics_mode":"AUTO","disable_collectors":null}
  component=setup
DEBU[2022-10-27T13:18:39.935+00:00] Register error: &url.Error{Op:"Post", URL:"https://XX.XX.XX.XX:443/v1/management/Node/Register", Err:(*net.OpError)(0xc0002120f0)}  component=setup


[hdrews2@cougar-pg-01 ~]$ sudo -A -i  curl -k -v -u admin:admin https://XX.XX.XX.XX/ping
* About to connect() to XX.XX.XX.XX port 443 (#0)
*   Trying XX.XX.XX.XX...
* Connected to XX.XX.XX.XX (XX.XX.XX.XX) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: O=Main Org.
*       start date: Sep 23 13:32:52 2022 GMT
*       expire date: Sep 23 13:32:52 2023 GMT
*       common name: (nil)
*       issuer: O=Main Org.
* Server auth using Basic with user 'admin'
> GET /ping HTTP/1.1
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.29.0
> Host: XX.XX.XX.XX
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 27 Oct 2022 13:25:03 GMT
< Content-Type: application/json
< Content-Length: 2
< Connection: keep-alive
< Grpc-Metadata-Content-Type: application/grpc
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-control: no-cache
< Pragma: no-cache
< Strict-Transport-Security: max-age=63072000; includeSubdomains;
<
* Connection #0 to host XX.XX.XX.XX left intact
1 Like
2

when you say “it fails” on the https port what do you see in a browser?

Are you getting a security warning or more of a network error (your command seem to show “connection refused” in the output which seems to be the latter)?

PMM client requires a secure connection to the pmm server (doesn’t have to be on port 443 but when the connection is made the client will only talk gRPC over HTTPS) so if you can’t get to the https port that is very likely where to start troubleshooting.

Looking at the command for getting your pmm-server up, it appears to be correct so my suspicions go right to iptables and some sort of firewall rules (maybe selinux or fail2ban?) that would be blocking access. if you paste the output of iptables --list -n (at least the INPUT and DOCKER sections), something might jump out.

1 Like
3

hmmm…but I see this succeeded!!! now I’m stumped… what happens if you change that to /graph instead of /ping?

1 Like
4

Here is the output from the iptables command. The docker IP is represented by YY.YY.YY.YY:

[hdrews2@cougar-pg-01 ~]$ sudo -A -i iptables --list -n 2>&1 | more
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            YY.YY.YY.YY           tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            YY.YY.YY.YY           tcp dpt:80

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_direct (1 references)
target     prot opt source               destination

Chain FWDI_public (2 references)
target     prot opt source               destination
FWDI_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0

Chain FWDI_public_allow (1 references)
target     prot opt source               destination

Chain FWDI_public_deny (1 references)
target     prot opt source               destination

Chain FWDI_public_log (1 references)
target     prot opt source               destination

Chain FWDO_public (2 references)
target     prot opt source               destination
FWDO_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_allow  all  --  0.0.0.0/0            0.0.0.0/0

Chain FWDO_public_allow (1 references)
target     prot opt source               destination

Chain FWDO_public_deny (1 references)
target     prot opt source               destination

Chain FWDO_public_log (1 references)
target     prot opt source               destination

Chain INPUT_ZONES (1 references)
target     prot opt source               destination
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain INPUT_direct (1 references)
target     prot opt source               destination

Chain IN_public (2 references)
target     prot opt source               destination
IN_public_log  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0

Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp limit: avg 25/min burst 100
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW,UNTRACKED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:591 ctstate NEW,UNTRACKED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8082 ctstate NEW,UNTRACKED
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:161 ctstate NEW,UNTRACKED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5432 ctstate NEW,UNTRACKED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 ctstate NEW,UNTRACKED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7777 ctstate NEW,UNTRACKED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW,UNTRACKED

Chain IN_public_deny (1 references)
target     prot opt source               destination

Chain IN_public_log (1 references)
target     prot opt source               destination

Chain OUTPUT_direct (1 references)
target     prot opt source               destination
1 Like
5

The output from the curl command to too much to add to a reply.

1 Like
6

The curl output is quite large.
curl_graph.txt (63.5 KB)

1 Like
7

Also, get this when attempting to load the home page in a web browser:

ERR_CONNECTION_CLOSED

1 Like
8

It is probably your firewall rules. You can attempt the following:

sudo iptables -F
sudo systemctl restart docker

This will get to you a point where you only have the forwarding rules in place for docker container. Then test with your browser / pmm-admin config and you should hopefully see successful registration. If that passes, have a look at why your forwarding rule is being blocked.

1 Like
9

Still have the same issue as before. Those commands didn’t make any difference.

1 Like
10

When you’re on the server your curl commands are working correctly (that long output from /graph IS the login page loading) so I’m stumped why the pmm-admin command on the same host gets refused?!?!

Complete shot in the dark but in your curl command you leave the port number off but you include it in the pmm-admin command… can you try with just https://admin:admin@XX.XX.XX.XX or even https://admin:admin@localhost

still doesn’t explain why a browser also can’t open the https page but if we treat them separate (even though they may not be).

1 Like
11

I did a systemctl status firewalld and found out that I’m getting WARNING: COMMAND_FAILED when setting up interface docker0 with the -D FORWARD option, “Bad rule”. I’ll have to talk to our SA’s to see if they can help out

1 Like
12

I get the same response when I leave the port number off, with both XXX.XX.XX.XX and localhost.

I ran a iptables --list-rules DOCKER command and got this:

[hdrews2@cougar-pg-01 ~]$ sudo -A -i iptables --list-rules DOCKER
-N DOCKER
-A DOCKER -d YY.YY.YY.YY/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d YY.YY.YY.YY/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT

Could the “! -i docker0” be causing the issue?

1 Like
13

I don’t think so…I looked at my instance and I have the exact same rule for my DOCKER network and it’s not blocking me.

Have you looked in /var/log/audit/audit.log as you try to hit it to see if selinux is the culprit?

alternatively you can switch from a port forwarding configuration in docker to host networking…instead of

>               --restart always \
>               --publish 443:443 \
>               --publish 80:80 \
>               --volume pmm-data:/srv \
>               --name pmm-server percona/pmm-server:2

you’d instead run:

sudo -A -i docker run --detach \
>               --restart always \
>               --network host \
>               --volume pmm-data:/srv \
>               --name pmm-server percona/pmm-server:2

which will bind directly to your machines 80 and 443 without using forwarding rules…my fear is that you’ve got some restrictive policy in selinux, firewalld or similar preventing you from using port 443 (either locally bound or as a forward) so even removing iptables from the mix there’s still some mandatory enforcement of another policy.

1 Like
14

After making the recommended change to the docker run command, here is the results:

[hdrews2@cougar-pg-01 ~]$
[hdrews2@cougar-pg-01 ~]$ sudo -A -i pmm-admin config --server-insecure-tls --server-url=https://admin:admin@XX.XX.XX.XX:443
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Post "https://XX.XX.XX.XX:443/v1/management/Node/Register": dial tcp XX.XX.XX.XX:443: connect: connection refused.
[hdrews2@cougar-pg-01 ~]$
[hdrews2@cougar-pg-01 ~]$ sudo -A -i lsof -i -P | grep LISTEN
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
rpcbind     1391      rpc    9u  IPv4   22826      0t0  TCP *:111 (LISTEN)
rpcbind     1391      rpc   12u  IPv6   22829      0t0  TCP *:111 (LISTEN)
sshd        1831     root    3u  IPv4   25490      0t0  TCP *:22 (LISTEN)
sshd        1831     root    4u  IPv6   25492      0t0  TCP *:22 (LISTEN)
pmm-agent   1832     root    6u  IPv4   28233      0t0  TCP localhost:38019 (LISTEN)
pmm-agent   1832     root    9u  IPv4   24182      0t0  TCP localhost:7777 (LISTEN)
postmaste   1859 postgres    7u  IPv4   27287      0t0  TCP XX.XX.XX.XX:5432 (LISTEN)
master      2153     root   14u  IPv4   29739      0t0  TCP localhost:25 (LISTEN)
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
lsof: no pwd entry for UID 26
postgres  108940       26    6u  IPv4 2087719      0t0  TCP localhost:5432 (LISTEN)
nginx     108943     root    6u  IPv4 2100241      0t0  TCP *:80 (LISTEN)
nginx     108943     root    7u  IPv4 2100242      0t0  TCP *:443 (LISTEN)
victoriam 108947 sysmaint   13u  IPv4 2100255      0t0  TCP localhost:9090 (LISTEN)
vmalert   108951 sysmaint    3u  IPv4 2098298      0t0  TCP localhost:8880 (LISTEN)
alertmana 108953 sysmaint    3u  IPv4 2091998      0t0  TCP localhost:9093 (LISTEN)
nginx     109006  polkitd    6u  IPv4 2100241      0t0  TCP *:80 (LISTEN)
nginx     109006  polkitd    7u  IPv4 2100242      0t0  TCP *:443 (LISTEN)
nginx     109007  polkitd    6u  IPv4 2100241      0t0  TCP *:80 (LISTEN)
nginx     109007  polkitd    7u  IPv4 2100242      0t0  TCP *:443 (LISTEN)
[hdrews2@cougar-pg-01 ~]$
[hdrews2@cougar-pg-01 ~]$ ps -fu 26
UID         PID   PPID  C STIME TTY          TIME CMD
26       108940 108789  6 12:38 ?        00:00:00 /usr/pgsql-14/bin/postgres -D /srv/postgres14 -c shared_preload_libraries=pg_stat_
26       108990 108940  0 12:38 ?        00:00:00 postgres: logger
26       109025 108940  0 12:38 ?        00:00:00 postgres: checkpointer
26       109027 108940  0 12:38 ?        00:00:00 postgres: background writer
26       109029 108940  0 12:38 ?        00:00:00 postgres: walwriter
26       109030 108940  0 12:38 ?        00:00:00 postgres: autovacuum launcher
26       109031 108940  0 12:38 ?        00:00:00 postgres: stats collector
26       109032 108940  0 12:38 ?        00:00:00 postgres: logical replication launcher

Now I’m getting a connection refused when I attempt to register. I also see a new set of PostgreSQL processes using UID 26. I’m curious about those processes, since they are started from my locally installed PostgreSQL 14 installation.

1 Like
15

There’s actually no change there. UID 26 is the UID of postgres running inside the container…but because the container is sort of a “jail” running out of the hosts kernel you can see the processes that it’s running inside the container on the host…but the host isn’t looking at /etc/passwd inside the container to translate the UID to username.

something is blocking access to port 443 somehow and I can’t for the life of me figure out what. It might be time to talk to your sysadmin team about what policy they configure on systems that might interfere. At this point the only other idea I have is to run pmm’s https listening on a non-standard port (I know I’m asking you to recreate the pmm-server container again and I’m sorry…but not sure how else to try and get you up and running).

>               --restart always \
>.              --publish 8443:443 \
>               --publish 443:443 \
>               --publish 80:80 \
>               --volume pmm-data:/srv \
>               --name pmm-server percona/pmm-server:2

that’ll give you a listener on 80, 443 and 8443 so you should be able to access the UI and register using https://XX.XX.XX.XX:8443 at that point…

1 Like
16

It was a long time coming, but I finally figured out the issue. The server I was installing PMM onto was FIPS enabled. As soon as I disabled FIPS mode, everything started working as expected.

1 Like
17

Well I’m happy you were able to get it all working…but…I assume it was in FIPS mode for a reason!

What OS are you using…not that I’m overflowing with free time these days but I’d be interested to see how to keep FIPS mode on a system and still be able to leverage PMM. As far as I know we use all the latest cryptography so perhaps it’s blocking you because the cert we ship PMM with is untrusted?

1 Like
18

We are using Redhat Enterprise version 7.

1 Like