Hi
1.) pmm works fine for a period like 2 weeks or 1 months, but facing “failed to established two way communication” issue on some of the agents. we are not using any proxy behind the connectivity for server to agent connection.
2.) some times restarting pmm-server helps to reconnect agent, but pmm2-admin list command not showing any output, simply stalled. Then again it went to the same issue.
3.) we have a oneway port opening from client to server.
pmm-admin list output
Failed to get PMM Server parameters from local pmm-agent: pmm-agent is not connected to PMM Server.
pmm-agent service logs
● pmm-agent.service - pmm-agent
Loaded: loaded (/lib/systemd/system/pmm-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-12-21 07:26:48 UTC; 3 weeks 0 days ago
Main PID: 17829 (pmm-agent)
Tasks: 26 (limit: 4915)
CGroup: /system.slice/pmm-agent.service
└─17829 /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
Jan 11 11:02:54 pmm-agent[17829]: INFO[2023-01-11T11:02:54.388+00:00] Establishing two-way communication channel … component=client
Jan 11 11:02:59 pmm-agent[17829]: ERRO[2023-01-11T11:02:59.386+00:00] Failed to establish two-way communication channel: context canceled. component=client
Jan 11 11:03:10 pmm-agent[17829]: INFO[2023-01-11T11:03:10.868+00:00] Connecting to https://admin:***@172.25.xx.xx:8443/ … component=client
Jan 11 11:03:10 pmm-agent[17829]: INFO[2023-01-11T11:03:10.871+00:00] Connected to 172.25.xx.xx:8443. component=client
Jan 11 11:03:10 pmm-agent[17829]: INFO[2023-01-11T11:03:10.871+00:00] Establishing two-way communication channel … component=client
Jan 11 11:03:15 pmm-agent[17829]: ERRO[2023-01-11T11:03:15.868+00:00] Failed to establish two-way communication channel: context canceled. component=client
Jan 11 11:03:29 pmm-agent[17829]: INFO[2023-01-11T11:03:29.805+00:00] Connecting to https://admin:***@172.25.xx.xx:8443/ … component=client
Jan 11 11:03:29 pmm-agent[17829]: INFO[2023-01-11T11:03:29.808+00:00] Connected to 172.25.xx.xx:8443. component=client
Jan 11 11:03:29 pmm-agent[17829]: INFO[2023-01-11T11:03:29.808+00:00] Establishing two-way communication channel … component=client
Jan 11 11:03:34 pmm-agent[17829]: ERRO[2023-01-11T11:03:34.808+00:00] Failed to establish two-way communication channel: context canceled. component=client
