Tweak Innodb parameters

MySQL & MariaDB Other MySQL® Questions
1

Hi,

The machine has 12 gb ram. It only houses one huge cache table (flushed twice per day). The table is BLOB. Currently about 10 GB is cached, so there’s room for using more ram.

I could really need some tips how to tweak it.

This is my.cnf:

Uncomment the following if you are using InnoDB tables

innodb_data_home_dir = /var/lib/mysql/
innodb_data_file_path = ibdata1:200M:autoextend
innodb_log_group_home_dir = /var/lib/mysql/
innodb_log_arch_dir = /var/lib/mysql/

You can set …_buffer_pool_size up to 50 - 80 %

of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 384M
innodb_additional_mem_pool_size = 20M

Set …_log_file_size to 25 % of buffer pool size

innodb_log_file_size = 100M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

Here’s output of status for innodb params.

Com_show_innodb_status 0
Innodb_buffer_pool_pages_data 24475
Innodb_buffer_pool_pages_dirty 10365
Innodb_buffer_pool_pages_flushed 26657589
Innodb_buffer_pool_pages_free 71
Innodb_buffer_pool_pages_latched 52
Innodb_buffer_pool_pages_misc 30
Innodb_buffer_pool_pages_total 24576
Innodb_buffer_pool_read_ahead_rnd 566045
Innodb_buffer_pool_read_ahead_seq 162349
Innodb_buffer_pool_read_requests 3165607053
Innodb_buffer_pool_reads 12897128
Innodb_buffer_pool_wait_free 0
Innodb_buffer_pool_write_requests 1052043302
Innodb_data_fsyncs 86361799
Innodb_data_pending_fsyncs 0
Innodb_data_pending_reads 1
Innodb_data_pending_writes 0
Innodb_data_read 4257353728
Innodb_data_reads 15859383
Innodb_data_writes 92817184
Innodb_data_written 1746445824
Innodb_dblwr_pages_written 26657589
Innodb_dblwr_writes 367841
Innodb_log_waits 0
Innodb_log_write_requests 644676246
Innodb_log_writes 85553632
Innodb_os_log_fsyncs 85632317
Innodb_os_log_pending_fsyncs 0
Innodb_os_log_pending_writes 0
Innodb_os_log_written 70796288
Innodb_page_size 16384
Innodb_pages_created 18758388
Innodb_pages_read 50853538
Innodb_pages_written 26657589
Innodb_row_lock_current_waits 0
Innodb_row_lock_time 261
Innodb_row_lock_time_avg 0
Innodb_row_lock_time_max 25
Innodb_row_lock_waits 1301
Innodb_rows_deleted 1906012
Innodb_rows_inserted 84545781
Innodb_rows_read 213763389
Innodb_rows_updated 0

Any tips?

2

Do you have any specific problems with performance?
Because the only thing I can say generic is that these two:
innodb_buffer_pool_size = 384M
innodb_additional_mem_pool_size = 20M
could be increased a lot, up to about 80% of RAM which could mean something like this:
innodb_buffer_pool_size = 9G
innodb_additional_mem_pool_size = 200M
But that depends on how big you database actually is, because if your DB is smaller than that this setting is worthless and you should maybe focus on allowing more memory to be used for sorting or join buffers etc.
That is why it is important to say if you experience any specific problems.

When you say “flushed twice per day”, what do you actually mean?
That it is deleted and rebuilt or what?

If you are experiencing slow insert/updates but the rest is going fast then you should change this:
innodb_flush_log_at_trx_commit = 1
to a value of 2 instead.
But if you do you should also be aware that if the server crashes then you could loose the last changes that you have performed in the database.

3

Do you know how to calculate innodb_additional_mem_pool_size?

4

  1. no specific problem. The database can hold up to 40-50G of data.

  2. It’s cached data, so the whole table (theres really only one table on it) gets dropped and rebuilt twice a day.

  3. No, I dont!

Thanks for your reply

5

If it’s only one table then you can keep the setting:
innodb_additional_mem_pool_size = 20M
There is no use to allow more since it will never be used.

And actually the value of 200M is very large and I want to retract that statement from my previous post. Sorry about that! :wink:

erkules:
No unfortunately I don’t have a good way to calculate it.
But it doesn’t really matter since the only thing that will happen if this buffer is too small is that MySQL will allocate the memory anyway but it will write error messages about it to the MySQL error log.
So if you see these errors turn up then you can just increase the setting.

6

Hi!

Did not work out very well.

The only thing I can think of would be the key_buffer already so high so innodb buffer pool could not allocate its 3G I chose.
(And having a key buffer above say 16M makes no sense in this case as I only have one table, an innodb so I will of course lower it)
What do you think?

080414 8:22:47 InnoDB: Error: cannot allocate 3221241856 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 39688400 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most 32-bit computers the process
InnoDB: memory space is limited to 2 GB or 4 GB.
InnoDB: We keep retrying the allocation for 60 seconds…
InnoDB: Fatal error: cannot allocate the memory for the buffer pool
^G/usr/sbin/mysqld: Out of memory (Needed 3814433792 bytes)
^G/usr/sbin/mysqld: Out of memory (Needed 2860824576 bytes)
080414 8:23:47 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.0.37-standard-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 MySQL Community Edition - Standard (GPL)
080414 8:24:02 [ERROR] /usr/sbin/mysqld: Incorrect information in file: ‘./something/somethingelse.frm’

Linux something.net 2.6.9-55.0.12.ELsmp #1 SMP Fri Nov 2 11:19:08 EDT 2007 i686 i686 i386 GNU/Linux

cat /proc/meminfo

MemTotal: 12469000 kB

Example MySQL config file for very large systems.

This is for a large system with memory of 1G-2G where the system runs mainly

MySQL.

You can copy this file to

/etc/my.cnf to set global options,

mysql-data-dir/my.cnf to set server-specific options (in this

installation this directory is /usr/local/var) or

~/.my.cnf to set user-specific options.

In this file, you can use all long options that a program supports.

If you want to know which options a program supports, run the program

with the “–help” option.

The following options will be passed to all MySQL clients

[client]
#password = your_password
port = 3306
socket = /var/lib/mysql/mysql.sock
default-character-set=utf8

Here follows entries for some specific programs

The MySQL server

[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-locking
key_buffer = 4096M
#Edited by PM default is 1M
max_allowed_packet = 40M
table_cache = 1024
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M

Try number of CPU’s*2 for thread_concurrency

thread_concurrency = 8
log_slow_queries=/var/log/mysql/slow-queries.log
long_query_time=2

log-long-format

wait_timeout=600
max_connections=2000
max_connect_errors=1000
default-character-set=utf8

Don’t listen on a TCP/IP port at all. This can be a security enhancement,

if all processes that need to connect to mysqld run on the same host.

All interaction with mysqld must be made via Unix sockets or named pipes.

Note that using this option without enabling named pipes on Windows

(via the “enable-named-pipe” option) will render mysqld useless!

#skip-networking

Replication Master Server (default)

binary logging is required for replication

#log-bin=mysql-bin

required unique id between 1 and 2^32 - 1

defaults to 1 if master-host is not set

but will not function as a master if omitted

server-id = 1

Replication Slave (comment out master section to use this)

To configure this host as a replication slave, you can choose between

two methods :

1) Use the CHANGE MASTER TO command (fully described in our manual) -

the syntax is:

CHANGE MASTER TO MASTER_HOST=, MASTER_PORT=,

MASTER_USER=, MASTER_PASSWORD= ;

where you replace , , by quoted strings and

by the master’s port number (3306 by default).

Example:

CHANGE MASTER TO MASTER_HOST=‘125.564.12.1’, MASTER_PORT=3306,

MASTER_USER=‘joe’, MASTER_PASSWORD=‘secret’;

OR

2) Set the variables below. However, in case you choose this method, then

start replication for the first time (even unsuccessfully, for example

if you mistyped the password in master-password and the slave fails to

connect), the slave will create a master.info file, and any later

change in this file to the variables’ values below will be ignored and

overridden by the content of the master.info file, unless you shutdown

the slave server, delete master.info and restart the slaver server.

For that reason, you may want to leave the lines below untouched

(commented) and instead use CHANGE MASTER TO (see above)

required unique id between 2 and 2^32 - 1

(and different from the master)

defaults to 2 if master-host is set

but will not function as a slave if omitted

#server-id = 2

The replication master for this slave - required

#master-host =

The username the slave will use for authentication when connecting

to the master - required

#master-user =

The password the slave will authenticate with when connecting to

the master - required

#master-password =

The port the master is listening on.

optional - defaults to 3306

#master-port =

binary logging - not required for slaves, but recommended

#log-bin=mysql-bin

Point the following paths to different dedicated disks

#tmpdir = /tmp/
#log-update = /path-to-dedicated-directory/hostname

Uncomment the following if you are using BDB tables

#bdb_cache_size = 384M
#bdb_max_lock = 100000

Uncomment the following if you are using InnoDB tables

innodb_data_home_dir = /var/lib/mysql/
innodb_data_file_path = ibdata1:200M:autoextend
innodb_log_group_home_dir = /var/lib/mysql/
innodb_log_arch_dir = /var/lib/mysql/

You can set …_buffer_pool_size up to 50 - 80 %

of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 3G
innodb_additional_mem_pool_size = 20M

Set …_log_file_size to 25 % of buffer pool size

innodb_log_file_size = 256M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

Remove the next comment character if you are not familiar with SQL

#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

7

I guess you’re using 32bit OS.

Using 64bit OS to overcome this limitation.

8

What exactly is the limitation?

9

You should of course lower it but I don’t think that is the problem because this buffer will not be allocated unless there are any indexes to use ti.

No, your problem is probably OS related instead.

Which OS are you running?
Not a 64bit version i assume?

Either that or you have a ulimit set globally for how much memory each process are allowed to use.

On 32bit OS’es one process in itself can very rarely be allowed to allocate more than about 2-3GB due to limitations with using 32bit memory address space.

And on some Linux/BSD distributions they have a global ulimit that doesn’t allow user processes to be larger than a certain size.
First check the ulimit limit with:

ulimit -a

If it says unlimited then it is not a ulimit problem and then unfortunately you need to run a 64bit version of your OS to be able to use more memory (which I think you should anyway since you have a pretty hefty server with a lot of RAM available).

And when you upgrade your OS to a 64 bit version you should also make sure that you are running a 64 bit version of MySQL

10

unlimited.

I guess I will have to reinstall servers etc.
I will revoke this thread once all is done, probably a few weeks.

Thanks for your answers guys!