Hi Team, After adding 100 servers to PMM, we are seeing a high CPU load, and the GUI is not responsive. Can someone please help us resolve this issue? [image] [image]

hello, Could you please share the hardware specs of the pmm server host. and what kind of instances have you added to the PMM, are they rds / ec2 ? and could please share the graph from pmm dash board of the pmm server server itself thanks

Hi @Mughees_Ahmed Thanks for the reply. Installation Type: Docker CPU: 16 RAM: 32 GB OS: Rocky Linux 8 MariaDB DB servers: All are on-premise Linux servers Service Running: All are MariaDB servers. I am unable to check PMM dashboards, we are seeing only this page. [image] or [image]

I have noticed the below errors in Grafana’s log. error=“context canceled” traceID= error=“database is locked” traceID= logger=context t=2023-08-11T04:34:35.701564308Z level=error msg=“invalid API key” error=“context canceled” traceID= logger=context userId=0 orgId=0 uname= t=2023-08-11T04:34:35…

We also encountered same issue. It seems to be related to the fact that sqlite database is used for grafana and probably will be fixed with next release: [PMM-4466] Migrate Grafana from using SQLite to PostgreSQL - Percona JIRA , https://jira.percona.com/projects/PMM/issues/PMM-12173 One workaraoun…

@Lauri Thanks for the details. How many nodes did you add to the PMM server?

@Lauri I’m not sure, but why are we seeing too much high CPU load because of the SQLite DB locks? So after execution of the below command, the CPU load has come down for you, or is CPU still high? – sqlite3 grafana.db ‘pragma journal_mode=wal;’

I do not know, why load went up because of sqlite locks, but using ‘wal’ definitely helped: no “db locked” messages anymore in logs, and PMM server cpu load also went down after some time.

@Lauri But for me, issues still exist. Again, I am trying to add the servers to PMM, and again, I am facing the same issue. :smiling_face_with_tear:

Do you still see “database is locked” errors in grafana log? And, if you upgraded PMM meanwhile, then maybe you have to use this workaround command again for sqlite?

After Adding 100 Servers PMM Load is Too High

Lauri August 15, 2023, 7:47pm 7

we have added ~ 500 nodes

Database is Locked pmm 2.38.1

Topic		Replies	Views
After upgrade to 2.39.0, the grafana takes too much CPU PMM 2.x	6	1482	September 27, 2023
Database is Locked pmm 2.38.1 PMM 2.x	44	3532	August 22, 2024
PMM doesn't start under high load and consumes all allocated CPU resources PMM 2.x pmm	3	883	July 17, 2023
Grafana has failed to load Percona Monitoring and Management (PMM)	20	3142	October 15, 2023
When i monitor 149 systems ,102 DB instance ， the page of grafana'web met some error PMM 1.x	5	612	September 20, 2018

After Adding 100 Servers PMM Load is Too High

Related topics