PMM server is not receiving metrics from client

Percona Monitoring and Management (PMM) PMM 3.x
, ,
1

Recently, we received a PMM client update request. After updating the agents on all monitored nodes, the system has stopped functioning as expected. Specifically, the inbuilt dashboards are no longer displaying the required data correctly, although everything was working fine prior to the update.

To troubleshoot this issue, I have already:

  • Reconfigured both the PMM server and the PMM clients.
  • Followed the official documentation and troubleshooting steps available online.
  • Verified the pmm-server status and inventory list, both of which indicate that all components are running properly.

Despite these efforts, the issue persists

2

Hi @Sumit_sarova

  1. can you share what version of PMM Server and PMM client are you using?
  2. Is it MySQL, PostgreSQL or MongoDB?
  3. Can you run pmm-admin summary and check for errors in logs
  4. Can you call https://PMM_SERVER_URL/logs.zip and check errors there?
  5. From which version to which you have upgraded?
  6. Did you upgrade only PMM Cleints or PMM Server as well?
3

Hi @nurlan
attaching the screenshots for your reference for version info and logs.
It is for MYSQL
PMM server was updated automatically and I manually updated the pmm-client.
can’t give any confirmation of the last version.

errorlogs
errorlogs1890×966 107 KB

pmm-client
pmm-client1623×859 31.6 KB

4

Can you share logs for mysqld_exporter?

5

mysqld_logs
mysqld_logs1902×718 96.8 KB

attached the logs, as it is visible there is some connection issue and we tried to solve it,
please suggest something if you have any solution for this.

6

Your mysqld_exporter is trying to ping the MySQL instance on 127.0.0.1:3306. Can you check manually if you are able to telnet 127.0.0.1 3306 or manually connect to MySQL using 127.0.0.1 on 3306? Also, check the netstat -tnlp or bind address of MySQL.

Please note that your initial PMM client for MySQL was configured for the address 10.110.128.196:3306.

7

Hi,
we have disabled telnet so sharing the output of nc -vz, pls check.

nc -vz
nc -vz988×703 18.8 KB

8

Anyone have any solution for this…
@nurlan @Abhinav_Gupta

9

Hi @Sumit_sarova

Could you please delete the service account related to that server, re-register the agent, and try once?

Hope you’re using the same version of the PMM server and client.

PMM server: 3.2.0
PMM Client: 3.2.0

image
image1919×260 33 KB

10

after investigation we found pmm-client version 3.2.0 is not compatible with pmm-server 3.2.0, so after downgrading pmm-client 3.1.0 it is working as expected, I suggest the community do the same procedure if anyone faces the same issue.

sudo apt install pmm-client=3.1.0-7.noble

working
working934×389 11.4 KB

11

@Sumit_sarova Can you please upgrade the agent to 3.2.0 and try once?

12

I tried many times with PMM Client: 3.2.0 but it didn’t work.

13

@Sumit_sarova

Can you try deleting the all service accounts related to the server and try re-register the agent?

14

yes tried this also but still it didn’t work as expected

15

@Sumit_sarova Can you check if the PMM user password contains any special characters like $ or # or something like these?

16

it wasn’t password issue, as the same password was working earlier and working with the current version that I am using.

17

@Sumit_sarova

Please check this.

18

thanks for this info, but no $ or # special character in the password.

1 Like
19

Hi @Sumit_sarova,
Can you share version of MySQL?

20

Hi,
mysql Ver 8.0.42-0ubuntu0.24.04.1 for Linux on x86_64 ((Ubuntu)), this is the version I am using