Replication generates huge disk write IO

MySQL & MariaDB Other MySQL® Questions
#1

Noticed that MySQL replication is generating enormous disk write activity

iotop 10-20Mb/s

AWS volume monitor - 700-800 IOPS (while only 150 IOPS on master)

And it lasts for hours. Historical graphs show that this repeats every day for 8-10 hours. (I think until the moment when catch up is finished)

Replication processes in the state: “Reading event from the relay log” and “Waiting for master to send event”
Replication is lagging behind master, it is running but VERY slow.

There are no active processes in MySQL that can generate this activity. No backups at this time. Replication activity is constant but not so high to generate this problem.

If I stop slave SQL thread - activity drops to zero immediately. After starting slave it increases again. Restarting mysqld doesn’t help.

What parameters/configs/metrics sould I check?

Any help would be appreciated.

Percona server 5.5 on CentOS
Replicated database about 500Gb, about 300 tables.

#2

Hi, I would like to suggest you to use pt-stalk (Percona utility for collecting data about MySQL when problem occur including trace and tcpdump ). You can get more information here that how to use it.

http://www.percona.com/doc/percona-toolkit/2.2/pt-stalk.html
http://www.mysqlperformanceblog.com/2013/01/03/percona-toolkit-by-example-pt-stalk/

Generally you have to specify some trigger condition so it can start to collect data. i.e with some function, variable values, threshold etc. If you want to collect information right now and without waiting for any trigger occur then you can simply run pt-stalk --no-stalk and check the results files to figure out where is the problem.

#3

Thank you.
What impact on the system will it have? Because any additional workload can crash the system.

#4

Hi,

It will not impact that much on system but I would suggest to read documentation properly with all options and test is on stage server before running on prod.

#5

I met the same situation.and more ,where the slave process start,the slave will eat up all the memory,all the swap space , and finally ,the mysqld process run outofmemory,and the kernel killed the mysqld worker process ,then ,mysql restart.

#6

Could it be the case https://www.percona.com/blog/2014/01/21/beware-mysql-5-6-server-uuid-cloning-slaves/ ?

#7

no ,it’s not the same case

#8

If you are using row-based replication and tables missing primary key/unique key then you probably hitting this bug https://bugs.mysql.com/bug.php?id=53375
Try to find out If any of database tables have missing PK/UK http://datacharmer.blogspot.com/2011/09/finding-tables-without-primary-keys.html And add an PK auto-inc If required.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2021 Percona LLC. All rights reserved.