We have Percona MySQL 5.7 running on AWS EC2 with 752 GB of memory and 128 CPUs, using Rocky 8. We typically see around 36K QPS during peak times, with approximately 20–40 running threads.
However, we have experienced outages when the number of running threads increased to over 1,000. At that point, every query became slower, and we saw hundreds of threads appear to be in the “freeing item” state, although the majority of queries still completed within one second. When we throttled the traffic, the system recovered almost immediately.
We understand that this is load-related and that the database is reaching a tipping point. We found some older information suggesting that jemalloc may handle high concurrency better than glibc and that we could use jemalloc to replace glibc. However, most of this information dates back almost 10 years.
We also don’t see memory usage increasing consistently. What would be a good indicator that we should consider replacing glibc with jemalloc?
So the problem here is “at high active threads, the query becomes slower” and processlist shows “freeing items” state.
I’d disable query cache. (do test)
set global query_cache_size=0; set global query_cache=OFF;
This is more of a problem at concurrency than at memory, as you clearly state that you don’t see memory usage increase!
We might want to look at overall query pattern and other configuration as well! Eg. there is low table cache and your queries are making multi table joins - with high concurrency it could be a problem.
Anyways, to answer your other question, I have a counter. “most of this information dates back almost 10 years” - Yes, but then the engine you’re using is also decade old. If your doubt is on memory, that’s a recommended solution. But wait… here is your next question
“What would be a good indicator that we should consider replacing glibc with jemalloc?”
- a good indicator as a proof will be us proving it that the majority of your time is spent for memory specific ops and not innodb internals/query cache/disk tables… There is this article that elaborates on how to identify scaling issues.
Well, yea, 5.7 died, like completely died, a long time ago. Even 8.0 is pretty well dead.
Curious, why are you trying to fix something that has no support, and no updates?
Have you tried the thread_pool plugin? This is designed specifically to handle high active thread count environments.
jemalloc is a memory allocator/manager. If you are not seeing mysqld holding on to memory for extended periods of time, then you don’t necessarily need jemalloc. Will it hurt to use it? Probably not.
We understand that MySQL 5.7 is EOL. We’re moving to MySQL 8; it just takes time. This outage happened on MySQL 5.7.
Our production environment is OLTP, but there are some long-running queries. With thread_pool, I assume queue scheduling could be challenging and may need to be adjusted based on the traffic pattern.
Due to the outage, I was advised to look into jemalloc, which is supposed to handle memory allocation better than glibc. The “freeing items” state may be related to memory allocation, so if jemalloc is faster than glibc, it could potentially improve our throughput.
I was told that jemalloc is very popular and has been adopted by many Percona MySQL customers. I don’t think using jemalloc would hurt, but adopting a third-party library does add maintenance overhead for us. Therefore, I’m looking for opinions on how the Percona MySQL community views the use of jemalloc.
Based on the current information, I would not recommend changing the memory allocator yet. The evidence clearly shows a high-concurrency scalability problem, but it does not identify the root cause. The freeing item state and stable memory usage are not sufficient evidence of glibc allocator contention.