After upgrading all my percona servers from 8.0.39 to 8.0.41, two of my servers show very high CPU load. At night we sync data from external sources with means allot of deletes, inserts and some analytical queries. A peak in CPU usage was normal, but since the update it spikes to 100% for hours. The other serves, where the issues are not present have the same hardware, same percona version, and similar type of data and load, but don’t have the hours long spiking.
System:
Debian bookworm - kernel 6.1.0-31-amd64 with percona 8.0.41-32
Using pt-config-diff I verified the config between working servers, and troubled server is the same.
I’m a bit stuck in how to troubleshoot any further.
Additional info: I have PMM 2.44 to help me troubleshoot (alhtough this has not really helped so far).
Thanks in advance for any help.
@alexanderdeprez
What is the role of those 2 servers where you observed high CPU load ? I mean are they part of any cross region (DR) or the Source (Read/Write) or if only serves the Read (Replica) ?
If (OS, DB )everything looks exactly same then reviewing the workload might be good idea. You can check the slow query logs or pt-query-digest report to get down to the bottleneck queries.
Since you having PMM then you can check QAN section as well to measure the query performance and differences
You can also check various [MySQL/Innodb] specific dashboard to check for the performance or other differences to find the hotspot.
High CPU usage mainly relatable to the unoptimized workload or queries. Fixing such DML’s would for sure reduce the consumption.
This blogpost - A Simple Approach to Troubleshooting High CPU in MySQL might come handy in order to get down to the heavily resource (CPU) consumed DML’s.