# Seemingly Random Slowdowns in CPU-bound Workload

**URL:** <https://forums.percona.com/t/seemingly-random-slowdowns-in-cpu-bound-workload/1766>\
**Category:** Other MySQL® Questions\
**Created:** [February 20, 2012, 7:28pm UTC](https://forums.percona.com/t/seemingly-random-slowdowns-in-cpu-bound-workload/1766 "2012-02-20T19:28:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jamesgolick](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jamesgolick](https://forums.percona.com/u/jamesgolick)\
**Post date:** [February 20, 2012, 7:28pm UTC](https://forums.percona.com/t/seemingly-random-slowdowns-in-cpu-bound-workload/1766/1 "2012-02-20T19:28:22Z")

</div>

Server specs:

24 cores (E5670)  
128GB mem  
10 x X25E RAID 10 for data  
2 x X25E RAID 1 for OS/logs

According to our monitoring, it seems like when things slow down, they slow down across the board. All queries seem to take longer. The slowdowns seem to come in groups as you can see here: [https://skitch.com/jamesgolick/8djnw/fetlife-rails3-rubby-1](https://skitch.com/jamesgolick/8djnw/fetlife-rails3-rubby-1). 9.3-new-relic

Edit: I should add that disk array utilization and await remain constant during the slowdowns, as does the buffer pool hit rate.

Also, we’re mysql version: “Server version: 5.0.91-50-log Percona SQL Server, Revision 73 (GPL)”

My mind immediately jumped to this being a mutex contention issue, but I don’t see much in the SEMAPHORES section of show innodb status. Example innodb status here:  
[URL=“https&#58;&#47;&#47;[gist:176319fda167e411f258 · GitHub](http://gist.github.com/176319fda167e411f258)”][https://gist.github.com/176319fda167e411f258[/URL]](https://gist.github.com/176319fda167e411f258%5B/URL%5D)

I’m not graphing OS Waits yet, but I’m working on getting that in to ganglia now. Any other thoughts on where to look?

---

<div class="post-metadata">

**Author:** ![xaprb](https://avatars.discourse-cdn.com/v4/letter/x/49beb7/32.png) [@xaprb](https://forums.percona.com/u/xaprb)\
**Post date:** [February 20, 2012, 8:38pm UTC](https://forums.percona.com/t/seemingly-random-slowdowns-in-cpu-bound-workload/1766/2 "2012-02-20T20:38:11Z")

</div>

I don’t think you have enough information to diagnose the problem. I’d use pt-stalk to gather more information when it happens.

---

<div class="post-metadata">

**Author:** ![jamesgolick](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jamesgolick](https://forums.percona.com/u/jamesgolick)\
**Post date:** [February 21, 2012, 9:06pm UTC](https://forums.percona.com/t/seemingly-random-slowdowns-in-cpu-bound-workload/1766/3 "2012-02-21T21:06:15Z")

</div>

I used pmp to get the following when it happens: [URL][https://gist.github.com/ae48bf7797f78e718bca[/URL]](https://gist.github.com/ae48bf7797f78e718bca%5B/URL%5D)

Note that this is after I disabled the query cache, since the first few profiles showed some contention on that.

Is it normal to see this much waiting for open\_and\_lock\_tables? I’m not very familiar with the mysql codebase.
