# Out of Memory **URL:** \ **Category:** Other MySQL® Questions\ **Created:** [January 15, 2008, 8:31am UTC](https://forums.percona.com/t/out-of-memory/599 "2008-01-15T08:31:43Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![orensol](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@orensol](https://forums.percona.com/u/orensol)\ **Post date:** [January 15, 2008, 8:31am UTC](https://forums.percona.com/t/out-of-memory/599/1 "2008-01-15T08:31:43Z")
Hello, This is my first post here, after I have tried to search existing posts for an answer to my problem, and could not find it. I have recently moved my database from a single 2GB RAM machine (that ran the entire LAMP), to a cluster configuration, in which the database is on its own machine with 4GB RAM, and also being replicated. Since then, I started receiving a lot of “Out of Memory” errors in mysql error log. They come in bulks, and at random times. Here is a sample: | [B]Quote:[/B] | | 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1042860 bytes) 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 5:20:02 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 5:20:04 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 5:20:04 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:18:29 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:18:33 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 9:18:33 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 9:18:42 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 9:20:01 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:20:01 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 9:27:21 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 15:52:20 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 15:52:20 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 16:01:19 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 16:01:21 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 16:01:22 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 16:01:23 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) | One time they also caused the server to crash and failover to the slave. Note, that it seems the process only needed 1M of allocation before it terminated… At first I thought it had to do with the OS being a 32-bit (rhel4), and the mysqld process larger than 2GB - I had innodb\_buffer\_pool\_size of 2000M - so I reduced it to 1000M, and I see that mysqld is under 2GB (at the time of writing: VIRT - 1855m, RES - 1.4g), but I am still getting these errors. I also tried to isolate a specific query that does it, so I turned on query logging with error messages, and I see that several queries cause this. Here is my.cnf: | [B]Quote:[/B] | | datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock skip-locking max\_connections = 530 default-character-set = utf8 max\_allowed\_packet = 16M query\_cache\_type = 1 query\_cache\_size = 256M query\_cache\_limit = 2M key\_buffer\_size = 256M max\_seeks\_for\_key = 500 # Try number of CPU’s\*2 for thread\_concurrency thread\_concurrency = 4 thread\_cache\_size = 16 table\_cache = 2048 tmp\_table\_size = 256M max\_heap\_table\_size = 257M ft\_min\_word\_len = 3 sort\_buffer\_size = 1M read\_buffer\_size = 1M read\_rnd\_buffer\_size = 1M join\_buffer\_size = 2M innodb\_buffer\_pool\_size = 1000M innodb\_additional\_mem\_pool\_size = 20M innodb\_log\_file\_size = 512M innodb\_log\_buffer\_size = 8M innodb\_flush\_log\_at\_trx\_commit = 1 innodb\_lock\_wait\_timeout = 50 innodb\_data\_file\_path = ibdata1:2000M;ibdata2:10M:autoextend log-slow-queries=slow-log long\_query\_time=1 log-bin=db-bin-log server-id = 1 slave-skip-errors=126,1062 | Any help would be greatly appreciated. Or ---
**Author:** ![MarkRose](https://avatars.discourse-cdn.com/v4/letter/m/85f322/32.png) [@MarkRose](https://forums.percona.com/u/MarkRose)\ **Post date:** [January 15, 2008, 2:25pm UTC](https://forums.percona.com/t/out-of-memory/599/2 "2008-01-15T14:25:13Z")
Is /proc/sys/vm/overcommit\_memory set to 0? If not, I believe it’s a setting in /etc/sysctl.conf on RedHat (I’m not a Redhat user). Make sure it’s set to 0 if it’s not. You can also change the value until next reboot by running “echo 0 \> /proc/sys/vm/overcommit\_memory” and see if it solves the problem. ---
**Author:** ![orensol](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@orensol](https://forums.percona.com/u/orensol)\ **Post date:** [January 15, 2008, 3:11pm UTC](https://forums.percona.com/t/out-of-memory/599/3 "2008-01-15T15:11:29Z")
It is set to 0: # cat /proc/sys/vm/overcommit\_memory0 I forgot to mention, this is mysql 5.0.45. Also, now the server from time to time gets huge loads (over 100!), mysqld is burning cpu on close to 200%, and I get over 95% IO waits, and temporary tables start to pile up in /var/lib/mysql… I am not sure if this problem is connected, or an entirely new problem. Or ---
**Author:** ![orensol](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@orensol](https://forums.percona.com/u/orensol)\ **Post date:** [January 15, 2008, 3:57pm UTC](https://forums.percona.com/t/out-of-memory/599/4 "2008-01-15T15:57:06Z")
The change in the machine was from OEM server with SATA drives to a HP with RAID10. The problems could be connected as you say, however the memory problems have been going on for a month (since the machine switching), but the IO problems are brand new (started today). More info that might help, is when this IO load occured, I restarted mysql, and upon restart I get this: | [B]Quote:[/B] | | 080115 14:26:34 mysqld started 080115 14:26:35 InnoDB: Started; log sequence number 8 1135678335 080115 14:26:35 [Note] /usr/sbin/mysqld: ready for connections. Version: ‘5.0.45-community-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 MySQL Community Edition (GPL) 080115 14:27:42 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:43 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:44 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:45 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:46 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:47 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:47 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:51 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:27:53 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080115 14:28:02 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080115 14:28:05 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) | Memory errors as soon as it starts up? Weird no? Or ---
**Author:** ![MarkRose](https://avatars.discourse-cdn.com/v4/letter/m/85f322/32.png) [@MarkRose](https://forums.percona.com/u/MarkRose)\ **Post date:** [January 15, 2008, 4:03pm UTC](https://forums.percona.com/t/out-of-memory/599/5 "2008-01-15T16:03:09Z")
That is indeed. Can you paste in my.cnf and the output of “show status;”? I’m not an expert, but another set of eyes never hurts. ---
**Author:** ![erkules](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@erkules](https://forums.percona.com/u/erkules)\ **Post date:** [January 15, 2008, 5:46pm UTC](https://forums.percona.com/t/out-of-memory/599/6 "2008-01-15T17:46:29Z")
| [B]orensol wrote on Tue, 15 January 2008 09:01[/B] | | | [B]Quote:[/B] | | 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1043352 bytes) 080113 5:20:00 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1042860 bytes) [snip] 080113 16:01:19 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 16:01:21 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) 080113 16:01:22 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1044576 bytes) 080113 16:01:23 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 1045840 bytes) | One time they also caused the server to crash and failover to the slave. Note, that it seems the process only needed 1M of allocation before it terminated… | This looks like a new connection, but you have no memory left. As you are running an 32bit OS you AFAIK cant use all the 4GB. Anyway habe a look at your system also. Something like top or vmstat look into /var/log/messages. Type into your mysql-client “show global status like ‘Threads\_connected’” next time it happens. “Show status like ‘Max\_used\_connections’” should give you a good clue. etc. ---
**Author:** ![orensol](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@orensol](https://forums.percona.com/u/orensol)\ **Post date:** [January 16, 2008, 4:09am UTC](https://forums.percona.com/t/out-of-memory/599/7 "2008-01-16T04:09:39Z")
| [B]Quote:[/B] | | USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND mysql 15 0 196 404:07.09 34.7 1811m 1.4g 4688 S mysqld | mysqld is burning the cpu (196%!!)… How can I reduce cpu usage? Maybe I should reduce number of threads? Or