Out of memory

This is sending me a little crazy, so please… any suggestions welcome.

A mySQL server is running fine, but then suddenly stops processing queries and returns all requests (even ‘show databases’ or ‘show variables’) with an “Out of memory” error.:

ERROR 5 (HY000): Out of memory
(mySQL error log shows… 090119 14:04:41 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 16711680 bytes))

This usually happens when the memory usage figures of ‘top’ show about 23.n% used by the mySQL process. Restarting mySQL clears the problem until memory usage again creeps up.

There are some bug reports similar to this in the mySQL bugs system, but all lack feedback and have been closed.

mySQL 5.1.30 (also observed on 5.0.x)
Xeon quad core
32bit CentOS 5.2 (with PAE kernel 2.6.18-53.1.14.el5PAE)
4GB physical RAM
1GB swap
/proc/sys/vm/overcommit_memory = 0

‘free’ shows:

total used free shared buffers cachedMem: 4148920 3993992 154928 0 135020 2763528-/+ buffers/cache: 1095444 3053476Swap: 1028120 88 1028032

‘strace’ of a ‘show variables’ command from the CLI…

11866 <… read resumed> “\17\0\0\0”, 4) = 411866 read(2783, “\3show variables”, 15) = 1511866 fcntl64(2783, F_SETFL, O_RDWR|O_NONBLOCK) = 011866 gettimeofday({1232373727, 307606}, NULL) = 011866 sched_setscheduler(11866, SCHED_OTHER, { 6 }) = -1 EINVAL (Invalid argument)11866 gettimeofday({1232373727, 307776}, NULL) = 011866 mmap2(NULL, 15720448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)11866 mmap2(NULL, 15720448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)11866 brk(0x11618000) = 0x106fa00011866 mmap2(NULL, 15851520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)11866 time(NULL) = 123237372711866 write(2, “090119 14:02:07 [ERROR] /usr/sbi”…, 80) = 8011866 sched_setscheduler(11866, SCHED_OTHER, { 8 }) = -1 EINVAL (Invalid argument)11866 write(2783, “.\0\0\1\377\5\0#HY000Out of memory (Need”…, 50) = 5011866 gettimeofday({1232373727, 308411}, NULL) = 011866 read(2783, 0x28e1e790, 4) = -1 EAGAIN (Resource temporarily unavailable)11866 time(NULL) = 123237372711866 fcntl64(2783, F_SETFL, O_RDWR) = 011866 read(2783, <unfinished …>

Are you using myisam_mmap?