Data is falling behind on PMM

Percona Monitoring and Management (PMM)
1

I am monitoring for a percona mysql cluster with pmm, but I have been seeing data with a delay of approximately 15 minutes on pmm dashboards for 2 weeks. I would like your information about the cause and solution of this.

Note: I did not have such a problem 2 weeks ago.

ProjectName: pmm-managed
Version: 2.32.0
PMMVersion: 2.32.0
Timestamp: 2022-11-02 09:05:53 (UTC)
FullCommit: 166887e98892fc8a7fde409de9af5f10cb28b193

2

Hello @bthnklc,
What’s the CPU load on your PMM server? Can you share some screenshots? When you select ‘Last 5m’ you see no data?

3

Hi @matthewb,
I when select ‘Last 5m’ i don’t see no data. The data lags behind for about 15-16 minutes.

image
image1822×811 54.2 KB

image
image1283×601 34.9 KB

4

Hmm. Because it is lagging exactly 15m, I suspect this is a timezone/config issue. Can you check the time on your server, both OS (date) and MySQL (SELECT NOW();) and also check the timezone setting in PMM (click last 5 min menu and at the bottom is a timezone selector).

5

Hello, I checked the time zone configuration. There doesn’t seem to be a problem, only the time zone of the pmm server was 3 hours behind, I fixed it, but nothing changed. Finally, I rebooted the pmm server. There is no such thing as 15 minutes behind time sometimes it is 11 minutes behind.
I can’t see the last 5 minutes data at all, some of the metrics are already seen as nodata.

6

Have you examined the logs of PMM server? You need to exec into the docker container and look through all the logs to see if there are any errors. Also, check the logs of the pmm-client on the mysql server.

7

Hi,

I temporarily solved the problem by reinstalling. The problem recurred in the past few days, and when I checked it, I think I understood the cause of the problem. When a Docker image backup is taken, it corrupts the time displayed by Docker and the OS and Docker time differs, so we need to fix this manually again. Although I wasn’t sure, I solved the problem this way. I wanted to share it for informational purposes.