# Performance Problems - High IO Wait

**URL:** <https://forums.percona.com/t/performance-problems-high-io-wait/757>\
**Category:** Other MySQL® Questions\
**Created:** [May 14, 2008, 11:56am UTC](https://forums.percona.com/t/performance-problems-high-io-wait/757 "2008-05-14T11:56:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![critzjm](https://avatars.discourse-cdn.com/v4/letter/c/cc9497/32.png) [@critzjm](https://forums.percona.com/u/critzjm)\
**Post date:** [May 14, 2008, 11:56am UTC](https://forums.percona.com/t/performance-problems-high-io-wait/757/1 "2008-05-14T11:56:35Z")

</div>

So my database has been running great for a while, and all of a sudden I’m seeing a lot of long running queries (that have been optimized and have been running fine) and my servers io wait starts growing to ~30%. My database is 22 GB, and all but one of my tables are InnoDB. The DB is doing around 1,000 qps.

My DB traffic stays pretty consistent so increased traffic doesn’t seem to cause the high io wait times. The only thing I can think of that has changed is the number of rows in my tables. I’ve got 2 tables that are around 30 million rows.

I’m not a dba. I have done quite a bit of reading on Mysql performance, though, and have tried to tune the Mysql as best I can. I assume our problems are coming from something being configured incorrectly.

Any help would be greatly appreciated.

Server Info

- CPU 4 x Dual Core AMD Opteron™ Processor 270
- RAM 15 GB

Linux

- Version 2.6.16.33-xenU (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #2 SMP
- Ubuntu 7.10 gutsy

Mysql

- Version 5.0.45-Debian\_1ubuntu3-log

- SHOW GLOBAL VARIABLES;  
| auto\_increment\_increment | 1 |  
| auto\_increment\_offset | 1 |  
| automatic\_sp\_privileges | ON |  
| back\_log | 50 |  
| basedir | /usr/ |  
| binlog\_cache\_size | 32768 |  
| bulk\_insert\_buffer\_size | 8388608 |  
| character\_set\_client | latin1 |  
| character\_set\_connection | latin1 |  
| character\_set\_database | latin1 |  
| character\_set\_filesystem | binary |  
| character\_set\_results | latin1 |  
| character\_set\_server | latin1 |  
| character\_set\_system | utf8 |  
| character\_sets\_dir | /usr/share/mysql/charsets/ |  
| collation\_connection | latin1\_swedish\_ci |  
| collation\_database | latin1\_swedish\_ci |  
| collation\_server | latin1\_swedish\_ci |  
| completion\_type | 0 |  
| concurrent\_insert | 1 |  
| connect\_timeout | 5 |  
| datadir | /mnt/mysql\_data/ |  
| date\_format | %Y-%m-%d |  
| datetime\_format | %Y-%m-%d %H:%i:%s |  
| default\_week\_format | 0 |  
| delay\_key\_write | ON |  
| delayed\_insert\_limit | 100 |  
| delayed\_insert\_timeout | 300 |  
| delayed\_queue\_size | 1000 |  
| div\_precision\_increment | 4 |  
| engine\_condition\_pushdown | OFF |  
| expire\_logs\_days | 10 |  
| flush | OFF |  
| flush\_time | 0 |  
| ft\_boolean\_syntax | + -\>\<()~\*:“”&| |  
| ft\_max\_word\_len | 84 |  
| ft\_min\_word\_len | 4 |  
| ft\_query\_expansion\_limit | 20 |  
| ft\_stopword\_file | (built-in) |  
| group\_concat\_max\_len | 1024 |  
| have\_archive | YES |  
| have\_bdb | NO |  
| have\_blackhole\_engine | YES |  
| have\_compress | YES |  
| have\_crypt | YES |  
| have\_csv | YES |  
| have\_dynamic\_loading | YES |  
| have\_example\_engine | NO |  
| have\_federated\_engine | YES |  
| have\_geometry | YES |  
| have\_innodb | YES |  
| have\_isam | NO |  
| have\_merge\_engine | YES |  
| have\_ndbcluster | DISABLED |  
| have\_openssl | DISABLED |  
| have\_ssl | DISABLED |  
| have\_query\_cache | YES |  
| have\_raid | NO |  
| have\_rtree\_keys | YES |  
| have\_symlink | YES |  
| init\_connect | |  
| init\_file | |  
| init\_slave | |  
| innodb\_additional\_mem\_pool\_size | 20971520 |  
| innodb\_autoextend\_increment | 8 |  
| innodb\_buffer\_pool\_awe\_mem\_mb | 0 |  
| innodb\_buffer\_pool\_size | 8589934592 |  
| innodb\_checksums | ON |  
| innodb\_commit\_concurrency | 0 |  
| innodb\_concurrency\_tickets | 500 |  
| innodb\_data\_file\_path | ibdata1:100M:autoextend |  
| innodb\_data\_home\_dir | |  
| innodb\_doublewrite | ON |  
| innodb\_fast\_shutdown | 1 |  
| innodb\_file\_io\_threads | 4 |  
| innodb\_file\_per\_table | OFF |  
| innodb\_flush\_log\_at\_trx\_commit | 2 |  
| innodb\_flush\_method | |  
| innodb\_force\_recovery | 0 |  
| innodb\_lock\_wait\_timeout | 20 |  
| innodb\_locks\_unsafe\_for\_binlog | OFF |  
| innodb\_log\_arch\_dir | |  
| innodb\_log\_archive | OFF |  
| innodb\_log\_buffer\_size | 8388608 |  
| innodb\_log\_file\_size | 335544320 |  
| innodb\_log\_files\_in\_group | 2 |  
| innodb\_log\_group\_home\_dir | ./ |  
| innodb\_max\_dirty\_pages\_pct | 90 |  
| innodb\_max\_purge\_lag | 0 |  
| innodb\_mirrored\_log\_groups | 1 |  
| innodb\_open\_files | 300 |  
| innodb\_rollback\_on\_timeout | OFF |  
| innodb\_support\_xa | ON |  
| innodb\_sync\_spin\_loops | 20 |  
| innodb\_table\_locks | ON |  
| innodb\_thread\_concurrency | 8 |  
| innodb\_thread\_sleep\_delay | 10000 |  
| interactive\_timeout | 28800 |  
| join\_buffer\_size | 131072 |  
| key\_buffer\_size | 33554432 |  
| key\_cache\_age\_threshold | 300 |  
| key\_cache\_block\_size | 1024 |  
| key\_cache\_division\_limit | 100 |  
| language | /usr/share/mysql/english/ |  
| large\_files\_support | ON |  
| large\_page\_size | 0 |  
| large\_pages | OFF |  
| lc\_time\_names | en\_US |  
| license | GPL |  
| local\_infile | ON |  
| locked\_in\_memory | OFF |  
| log | OFF |  
| log\_bin | ON |  
| log\_bin\_trust\_function\_creators | OFF |  
| log\_error | |  
| log\_queries\_not\_using\_indexes | OFF |  
| log\_slave\_updates | OFF |  
| log\_slow\_queries | ON |  
| log\_warnings | 1 |  
| long\_query\_time | 1 |  
| low\_priority\_updates | OFF |  
| lower\_case\_file\_system | OFF |  
| lower\_case\_table\_names | 0 |  
| max\_allowed\_packet | 16776192 |  
| max\_binlog\_cache\_size | 18446744073709551615 |  
| max\_binlog\_size | 104857600 |  
| max\_connect\_errors | 10 |  
| max\_connections | 500 |  
| max\_delayed\_threads | 20 |  
| max\_error\_count | 64 |  
| max\_heap\_table\_size | 16777216 |  
| max\_insert\_delayed\_threads | 20 |  
| max\_join\_size | 18446744073709551615 |  
| max\_length\_for\_sort\_data | 1024 |  
| max\_prepared\_stmt\_count | 16382 |  
| max\_relay\_log\_size | 0 |  
| max\_seeks\_for\_key | 18446744073709551615 |  
| max\_sort\_length | 1024 |  
| max\_sp\_recursion\_depth | 0 |  
| max\_tmp\_tables | 32 |  
| max\_user\_connections | 0 |  
| max\_write\_lock\_count | 18446744073709551615 |  
| multi\_range\_count | 256 |  
| myisam\_data\_pointer\_size | 6 |  
| myisam\_max\_sort\_file\_size | 9223372036854775807 |  
| myisam\_recover\_options | OFF |  
| myisam\_repair\_threads | 1 |  
| myisam\_sort\_buffer\_size | 8388608 |  
| myisam\_stats\_method | nulls\_unequal |  
| ndb\_autoincrement\_prefetch\_sz | 32 |  
| ndb\_force\_send | ON |  
| ndb\_use\_exact\_count | ON |  
| ndb\_use\_transactions | ON |  
| ndb\_cache\_check\_time | 0 |  
| ndb\_connectstring | |  
| net\_buffer\_length | 16384 |  
| net\_read\_timeout | 30 |  
| net\_retry\_count | 10 |  
| net\_write\_timeout | 60 |  
| new | OFF |  
| old\_passwords | OFF |  
| open\_files\_limit | 3582 |  
| optimizer\_prune\_level | 1 |  
| optimizer\_search\_depth | 62 |  
| pid\_file | /var/run/mysqld/mysqld.pid |  
| port | 3306 |  
| preload\_buffer\_size | 32768 |  
| profiling | OFF |  
| profiling\_history\_size | 15 |  
| protocol\_version | 10 |  
| query\_alloc\_block\_size | 8192 |  
| query\_cache\_limit | 1048576 |  
| query\_cache\_min\_res\_unit | 4096 |  
| query\_cache\_size | 0 |  
| query\_cache\_type | OFF |  
| query\_cache\_wlock\_invalidate | OFF |  
| query\_prealloc\_size | 8192 |  
| range\_alloc\_block\_size | 2048 |  
| read\_buffer\_size | 131072 |  
| read\_only | OFF |  
| read\_rnd\_buffer\_size | 262144 |  
| relay\_log\_purge | ON |  
| relay\_log\_space\_limit | 0 |  
| rpl\_recovery\_rank | 0 |  
| secure\_auth | OFF |  
| secure\_file\_priv | |  
| server\_id | 2 |  
| skip\_external\_locking | ON |  
| skip\_networking | OFF |  
| skip\_show\_database | OFF |  
| slave\_compressed\_protocol | OFF |  
| slave\_load\_tmpdir | /mnt/mysql\_data/tmp/ |  
| slave\_net\_timeout | 3600 |  
| slave\_skip\_errors | OFF |  
| slave\_transaction\_retries | 10 |  
| slow\_launch\_time | 2 |  
| socket | /var/run/mysqld/mysqld.sock |  
| sort\_buffer\_size | 2097144 |  
| sql\_big\_selects | ON |  
| sql\_mode | |  
| sql\_notes | ON |  
| sql\_warnings | OFF |  
| ssl\_ca | |  
| ssl\_capath | |  
| ssl\_cert | |  
| ssl\_cipher | |  
| ssl\_key | |  
| storage\_engine | InnoDB |  
| sync\_binlog | 0 |  
| sync\_frm | ON |  
| system\_time\_zone | UTC |  
| table\_cache | 1536 |  
| table\_lock\_wait\_timeout | 50 |  
| table\_type | InnoDB |  
| thread\_cache\_size | 32 |  
| thread\_stack | 131072 |  
| time\_format | %H:%i:%s |  
| time\_zone | SYSTEM |  
| timed\_mutexes | OFF |  
| tmp\_table\_size | 1024 |  
| tmpdir | /mnt/mysql\_data/tmp |  
| transaction\_alloc\_block\_size | 8192 |  
| transaction\_prealloc\_size | 4096 |  
| tx\_isolation | REPEATABLE-READ |  
| updatable\_views\_with\_limit | YES |  
| version | 5.0.45-Debian\_1ubuntu3-log |  
| version\_comment | Debian etch distribution |  
| version\_compile\_machine | x86\_64 |  
| version\_compile\_os | pc-linux-gnu |  
| wait\_timeout | 600

- SHOW GLOBAL STATUS;  
| Aborted\_clients | 2 |  
| Aborted\_connects | 0 |  
| Binlog\_cache\_disk\_use | 0 |  
| Binlog\_cache\_use | 3058177 |  
| Bytes\_received | 18816243950 |  
| Bytes\_sent | 182503223657 |  
| Com\_admin\_commands | 1 |  
| Com\_alter\_db | 0 |  
| Com\_alter\_table | 0 |  
| Com\_analyze | 0 |  
| Com\_backup\_table | 0 |  
| Com\_begin | 3062707 |  
| Com\_call\_procedure | 0 |  
| Com\_change\_db | 0 |  
| Com\_change\_master | 0 |  
| Com\_check | 0 |  
| Com\_checksum | 0 |  
| Com\_commit | 3057864 |  
| Com\_create\_db | 0 |  
| Com\_create\_function | 0 |  
| Com\_create\_index | 0 |  
| Com\_create\_table | 0 |  
| Com\_create\_user | 0 |  
| Com\_dealloc\_sql | 0 |  
| Com\_delete | 6583 |  
| Com\_delete\_multi | 0 |  
| Com\_do | 0 |  
| Com\_drop\_db | 0 |  
| Com\_drop\_function | 0 |  
| Com\_drop\_index | 0 |  
| Com\_drop\_table | 0 |  
| Com\_drop\_user | 0 |  
| Com\_execute\_sql | 0 |  
| Com\_flush | 0 |  
| Com\_grant | 0 |  
| Com\_ha\_close | 0 |  
| Com\_ha\_open | 0 |  
| Com\_ha\_read | 0 |  
| Com\_help | 0 |  
| Com\_insert | 913342 |  
| Com\_insert\_select | 0 |  
| Com\_kill | 0 |  
| Com\_load | 0 |  
| Com\_load\_master\_data | 0 |  
| Com\_load\_master\_table | 0 |  
| Com\_lock\_tables | 0 |  
| Com\_optimize | 0 |  
| Com\_preload\_keys | 0 |  
| Com\_prepare\_sql | 0 |  
| Com\_purge | 0 |  
| Com\_purge\_before\_date | 0 |  
| Com\_rename\_table | 0 |  
| Com\_repair | 0 |  
| Com\_replace | 0 |  
| Com\_replace\_select | 0 |  
| Com\_reset | 0 |  
| Com\_restore\_table | 0 |  
| Com\_revoke | 0 |  
| Com\_revoke\_all | 0 |  
| Com\_rollback | 4821 |  
| Com\_savepoint | 0 |  
| Com\_select | 34569207 |  
| Com\_set\_option | 11383 |  
| Com\_show\_binlog\_events | 0 |  
| Com\_show\_binlogs | 0 |  
| Com\_show\_charsets | 0 |  
| Com\_show\_collations | 0 |  
| Com\_show\_column\_types | 0 |  
| Com\_show\_create\_db | 0 |  
| Com\_show\_create\_table | 0 |  
| Com\_show\_databases | 0 |  
| Com\_show\_errors | 0 |  
| Com\_show\_fields | 36910 |  
| Com\_show\_grants | 0 |  
| Com\_show\_innodb\_status | 0 |  
| Com\_show\_keys | 0 |  
| Com\_show\_logs | 0 |  
| Com\_show\_master\_status | 0 |  
| Com\_show\_ndb\_status | 0 |  
| Com\_show\_new\_master | 0 |  
| Com\_show\_open\_tables | 0 |  
| Com\_show\_privileges | 0 |  
| Com\_show\_processlist | 0 |  
| Com\_show\_slave\_hosts | 1 |  
| Com\_show\_slave\_status | 0 |  
| Com\_show\_status | 7743104 |  
| Com\_show\_storage\_engines | 0 |  
| Com\_show\_tables | 10952 |  
| Com\_show\_triggers | 0 |  
| Com\_show\_variables | 202 |  
| Com\_show\_warnings | 0 |  
| Com\_slave\_start | 0 |  
| Com\_slave\_stop | 0 |  
| Com\_stmt\_close | 0 |  
| Com\_stmt\_execute | 0 |  
| Com\_stmt\_fetch | 0 |  
| Com\_stmt\_prepare | 0 |  
| Com\_stmt\_reset | 0 |  
| Com\_stmt\_send\_long\_data | 0 |  
| Com\_truncate | 0 |  
| Com\_unlock\_tables | 0 |  
| Com\_update | 3213818 |  
| Com\_update\_multi | 0 |  
| Com\_xa\_commit | 0 |  
| Com\_xa\_end | 0 |  
| Com\_xa\_prepare | 0 |  
| Com\_xa\_recover | 0 |  
| Com\_xa\_rollback | 0 |  
| Com\_xa\_start | 0 |  
| Compression | OFF |  
| Connections | 6872 |  
| Created\_tmp\_disk\_tables | 187729 |  
| Created\_tmp\_files | 211 |  
| Created\_tmp\_tables | 376632 |  
| Delayed\_errors | 0 |  
| Delayed\_insert\_threads | 0 |  
| Delayed\_writes | 0 |  
| Flush\_commands | 1 |  
| Handler\_commit | 10249469 |  
| Handler\_delete | 0 |  
| Handler\_discover | 0 |  
| Handler\_prepare | 6115724 |  
| Handler\_read\_first | 192304 |  
| Handler\_read\_key | 392210003 |  
| Handler\_read\_next | 9155741769 |  
| Handler\_read\_prev | 0 |  
| Handler\_read\_rnd | 3134865 |  
| Handler\_read\_rnd\_next | 32034534 |  
| Handler\_rollback | 5138 |  
| Handler\_savepoint | 0 |  
| Handler\_savepoint\_rollback | 0 |  
| Handler\_update | 0 |  
| Handler\_write | 30488911 |  
| Innodb\_buffer\_pool\_pages\_data | 483900 |  
| Innodb\_buffer\_pool\_pages\_dirty | 63756 |  
| Innodb\_buffer\_pool\_pages\_flushed | 708235 |  
| Innodb\_buffer\_pool\_pages\_free | 1 |  
| Innodb\_buffer\_pool\_pages\_latched | 8 |  
| Innodb\_buffer\_pool\_pages\_misc | 40387 |  
| Innodb\_buffer\_pool\_pages\_total | 524288 |  
| Innodb\_buffer\_pool\_read\_ahead\_rnd | 1359 |  
| Innodb\_buffer\_pool\_read\_ahead\_seq | 6970 |  
| Innodb\_buffer\_pool\_read\_requests | 27856201125 |  
| Innodb\_buffer\_pool\_reads | 698690 |  
| Innodb\_buffer\_pool\_wait\_free | 0 |  
| Innodb\_buffer\_pool\_write\_requests | 40718811 |  
| Innodb\_data\_fsyncs | 94825 |  
| Innodb\_data\_pending\_fsyncs | 0 |  
| Innodb\_data\_pending\_reads | 0 |  
| Innodb\_data\_pending\_writes | 0 |  
| Innodb\_data\_read | 18081370112 |  
| Innodb\_data\_reads | 893070 |  
| Innodb\_data\_writes | 6570956 |  
| Innodb\_data\_written | 29564379648 |  
| Innodb\_dblwr\_pages\_written | 708235 |  
| Innodb\_dblwr\_writes | 12758 |  
| Innodb\_log\_waits | 0 |  
| Innodb\_log\_write\_requests | 7043610 |  
| Innodb\_log\_writes | 6103194 |  
| Innodb\_os\_log\_fsyncs | 69265 |  
| Innodb\_os\_log\_pending\_fsyncs | 0 |  
| Innodb\_os\_log\_pending\_writes | 0 |  
| Innodb\_os\_log\_written | 6354055168 |  
| Innodb\_page\_size | 16384 |  
| Innodb\_pages\_created | 15931 |  
| Innodb\_pages\_read | 1074474 |  
| Innodb\_pages\_written | 708235 |  
| Innodb\_row\_lock\_current\_waits | 0 |  
| Innodb\_row\_lock\_time | 2015890 |  
| Innodb\_row\_lock\_time\_avg | 143 |  
| Innodb\_row\_lock\_time\_max | 21384 |  
| Innodb\_row\_lock\_waits | 14088 |  
| Innodb\_rows\_deleted | 6589 |  
| Innodb\_rows\_inserted | 913341 |  
| Innodb\_rows\_read | 9435790975 |  
| Innodb\_rows\_updated | 3177083 |  
| Key\_blocks\_not\_flushed | 0 |  
| Key\_blocks\_unused | 26792 |  
| Key\_blocks\_used | 14 |  
| Key\_read\_requests | 8113318 |  
| Key\_reads | 81332 |  
| Key\_write\_requests | 1394173 |  
| Key\_writes | 0 |  
| Last\_query\_cost | 0.000000 |  
| Max\_used\_connections | 279 |  
| Ndb\_cluster\_node\_id | 0 |  
| Ndb\_config\_from\_host | |  
| Ndb\_config\_from\_port | 0 |  
| Ndb\_number\_of\_data\_nodes | 0 |  
| Not\_flushed\_delayed\_rows | 0 |  
| Open\_files | 18 |  
| Open\_streams | 0 |  
| Open\_tables | 919 |  
| Opened\_tables | 925 |  
| Prepared\_stmt\_count | 0 |  
| Qcache\_free\_blocks | 0 |  
| Qcache\_free\_memory | 0 |  
| Qcache\_hits | 0 |  
| Qcache\_inserts | 0 |  
| Qcache\_lowmem\_prunes | 0 |  
| Qcache\_not\_cached | 0 |  
| Qcache\_queries\_in\_cache | 0 |  
| Qcache\_total\_blocks | 0 |  
| Questions | 44894895 |  
| Rpl\_status | NULL |  
| Select\_full\_join | 0 |  
| Select\_full\_range\_join | 0 |  
| Select\_range | 243349 |  
| Select\_range\_check | 0 |  
| Select\_scan | 241007 |  
| Slave\_open\_temp\_tables | 0 |  
| Slave\_retried\_transactions | 0 |  
| Slave\_running | OFF |  
| Slow\_launch\_threads | 0 |  
| Slow\_queries | 38471 |  
| Sort\_merge\_passes | 103 |  
| Sort\_range | 247677 |  
| Sort\_rows | 29168476 |  
| Sort\_scan | 405301 |  
| Ssl\_accept\_renegotiates | 0 |  
| Ssl\_accepts | 0 |  
| Ssl\_callback\_cache\_hits | 0 |  
| Ssl\_cipher | |  
| Ssl\_cipher\_list | |  
| Ssl\_client\_connects | 0 |  
| Ssl\_connect\_renegotiates | 0 |  
| Ssl\_ctx\_verify\_depth | 0 |  
| Ssl\_ctx\_verify\_mode | 0 |  
| Ssl\_default\_timeout | 0 |  
| Ssl\_finished\_accepts | 0 |  
| Ssl\_finished\_connects | 0 |  
| Ssl\_session\_cache\_hits | 0 |  
| Ssl\_session\_cache\_misses | 0 |  
| Ssl\_session\_cache\_mode | NONE |  
| Ssl\_session\_cache\_overflows | 0 |  
| Ssl\_session\_cache\_size | 0 |  
| Ssl\_session\_cache\_timeouts | 0 |  
| Ssl\_sessions\_reused | 0 |  
| Ssl\_used\_session\_cache\_entries | 0 |  
| Ssl\_verify\_depth | 0 |  
| Ssl\_verify\_mode | 0 |  
| Ssl\_version | |  
| Table\_locks\_immediate | 46497124 |  
| Table\_locks\_waited | 0 |  
| Tc\_log\_max\_pages\_used | 0 |  
| Tc\_log\_page\_size | 0 |  
| Tc\_log\_page\_waits | 1 |  
| Threads\_cached | 2 |  
| Threads\_connected | 275 |  
| Threads\_created | 858 |  
| Threads\_running | 74 |  
| Uptime | 59291 |  
| Uptime\_since\_flush\_status | 59291

- mysqlreport  
MySQL 5.0.45-Debian\_1ub uptime 0 16:29:5 Wed May 14 17:24:55 2008

\_\_ Key \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_  
Buffer used 14.00k of 32.00M %Used: 0.04  
Current 5.84M %Usage: 18.24  
Write hit 100.00%  
Read hit 99.00%

\_\_ Questions \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Total 44.96M 757.7/s  
DMS 38.76M 653.2/s %Total: 86.21  
Com\_ 13.95M 235.0/s 31.02  
-Unknown 7.75M 130.7/s 17.24  
COM\_QUIT 6.87k 0.1/s 0.02  
Slow (1) 38.47k 0.6/s 0.09 %DMS: 0.10 Log: ON  
DMS 38.76M 653.2/s 86.21  
SELECT 34.62M 583.4/s 77.00 89.32  
UPDATE 3.22M 54.2/s 7.16 8.30  
INSERT 914.94k 15.4/s 2.03 2.36  
DELETE 6.59k 0.1/s 0.01 0.02  
REPLACE 0 0/s 0.00 0.00  
Com\_ 13.95M 235.0/s 31.02  
show\_status 7.75M 130.7/s 17.24  
begin 3.07M 51.7/s 6.82  
commit 3.06M 51.6/s 6.81

\_\_ SELECT and Sort \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Scan 241.29k 4.1/s %SELECT: 0.70  
Range 243.76k 4.1/s 0.70  
Full join 0 0/s 0.00  
Range check 0 0/s 0.00  
Full rng join 0 0/s 0.00  
Sort scan 405.92k 6.8/s  
Sort range 248.06k 4.2/s  
Sort mrg pass 103 0.0/s

\_\_ Table Locks \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Waited 0 0/s %Total: 0.00  
Immediate 46.57M 784.7/s

\_\_ Tables \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_  
Open 919 of 1536 %Cache: 59.83  
Opened 925 0.0/s

\_\_ Connections \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Max used 279 of 500 %Max: 55.80  
Total 6.88k 0.1/s

\_\_ Created Temp \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Disk table 188.00k 3.2/s  
Table 377.19k 6.4/s Size: 1.0k  
File 211 0.0/s

\_\_ Threads \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_  
Running 73 of 275  
Cached 2 of 32 %Hit: 87.52  
Created 858 0.0/s  
Slow 0 0/s

\_\_ Aborted \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_  
Clients 2 0.0/s  
Connects 0 0/s

\_\_ Bytes \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_  
Sent 182.78G 3.1M/s  
Received 18.84G 317.5k/s

\_\_ InnoDB Buffer Pool \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Usage 8.00G of 8.00G %Used: 100.00  
Read hit 100.00%  
Pages  
Free 0 %Total: 0.00  
Data 483.90k 92.30 %Drty: 13.26  
Misc 40384 7.70  
Latched 3 0.00  
Reads 27.90G 470.1k/s  
From file 699.26k 11.8/s 0.00  
Ahead Rnd 1363 0.0/s  
Ahead Sql 6976 0.1/s  
Writes 40.79M 687.3/s  
Flushes 708.89k 11.9/s  
Wait Free 0 0/s

\_\_ InnoDB Lock \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Waits 14118 0.2/s  
Current 0  
Time acquiring  
Total 2016944 ms  
Average 142 ms  
Max 21384 ms

\_\_ InnoDB Data, Pages, Rows \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
Data  
Reads 893.81k 15.1/s  
Writes 6.58M 110.9/s  
fsync 94.91k 1.6/s  
Pending  
Reads 0  
Writes 0  
fsync 0

Pages  
Created 15.96k 0.3/s  
Read 1.08M 18.1/s  
Written 708.89k 11.9/s

Rows  
Deleted 6.60k 0.1/s  
Inserted 914.94k 15.4/s  
Read 9.45G 159.2k/s  
Updated 3.18M 53.6/s

---

<div class="post-metadata">

**Author:** ![critzjm](https://avatars.discourse-cdn.com/v4/letter/c/cc9497/32.png) [@critzjm](https://forums.percona.com/u/critzjm)\
**Post date:** [May 14, 2008, 2:25pm UTC](https://forums.percona.com/t/performance-problems-high-io-wait/757/2 "2008-05-14T14:25:19Z")

</div>

Here is a snippet from top:

top - 19:54:22 up 60 days, 21:20, 1 user, load average: 8.73, 10.58, 10.41  
Tasks: 52 total, 1 running, 51 sleeping, 0 stopped, 0 zombie  
Cpu(s): 30.4%us, 3.2%sy, 0.0%ni, 32.4%id, 22.0%wa, 0.1%hi, 1.7%si, 10.4%st  
Mem: 15728640k total, 15703600k used, 25040k free, 150332k buffers  
Swap: 0k total, 0k used, 0k free, 4520588k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  
20249 mysql 15 0 9724m 9.1g 5840 S 139 60.8 1113:22 mysqld

---

<div class="post-metadata">

**Author:** ![edsuit](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@edsuit](https://forums.percona.com/u/edsuit)\
**Post date:** [May 28, 2008, 8:22am UTC](https://forums.percona.com/t/performance-problems-high-io-wait/757/3 "2008-05-28T08:22:59Z")

</div>

I see your Innodb buffer pool is used for 100%. That means it is probably too small, which can cause major performance problems (and other problems as well).

See: [URL=“http&#58;&#47;&#47;[hackmysql.com/mysqlreportguide](http://hackmysql.com/mysqlreportguide)”][http://hackmysql.com/mysqlreportguide[/URL]](http://hackmysql.com/mysqlreportguide%5B/URL%5D) and [http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-](http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-) in-mysql-server-after-installation/ for more info on that (I’m no expert either).

---

<div class="post-metadata">

**Author:** ![critzjm](https://avatars.discourse-cdn.com/v4/letter/c/cc9497/32.png) [@critzjm](https://forums.percona.com/u/critzjm)\
**Post date:** [May 28, 2008, 10:47am UTC](https://forums.percona.com/t/performance-problems-high-io-wait/757/4 "2008-05-28T10:47:36Z")

</div>

Thanks for the reply!

I used to have my buffer pool size set to 12GB, but mysql wouldn’t start - I’m pretty sure I ran out of memory. I bumped it back down to 8GB, but I could probably get it back up to around 10GB. But that may only buy me a little time with my database growing.

I guess my question is: since my DB is 22GB+, and I only have 15GB of RAM on the server, does that mean I need to start splitting my database up, or are there other ways to keep performance up with a DB larger than the amount of RAM on the server?