# Replication generates huge disk write IO **URL:** \ **Category:** Other MySQL® Questions\ **Created:** [June 11, 2013, 3:53pm UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749 "2013-06-11T15:53:36Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Serge\_Shakhov](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@Serge\_Shakhov](https://forums.percona.com/u/Serge_Shakhov)\ **Post date:** [June 11, 2013, 3:53pm UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/1 "2013-06-11T15:53:36Z")
Noticed that MySQL replication is generating enormous disk write activity iotop 10-20Mb/s AWS volume monitor - 700-800 IOPS (while only 150 IOPS on master) And it lasts for hours. Historical graphs show that this repeats every day for 8-10 hours. (I think until the moment when catch up is finished) Replication processes in the state: “Reading event from the relay log” and “Waiting for master to send event” Replication is lagging behind master, it is running but VERY slow. There are no active processes in MySQL that can generate this activity. No backups at this time. Replication activity is constant but not so high to generate this problem. If I stop slave SQL thread - activity drops to zero immediately. After starting slave it increases again. Restarting mysqld doesn’t help. What parameters/configs/metrics sould I check? Any help would be appreciated. Percona server 5.5 on CentOS Replicated database about 500Gb, about 300 tables. ---
**Author:** ![niljoshi](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/niljoshi/32/16889_2.png) [@niljoshi](https://forums.percona.com/u/niljoshi)\ **Post date:** [June 12, 2013, 2:58am UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/2 "2013-06-12T02:58:25Z")
Hi, I would like to suggest you to use pt-stalk (Percona utility for collecting data about MySQL when problem occur including trace and tcpdump ). You can get more information here that how to use it. [URL][http://www.percona.com/doc/percona-toolkit/2.2/pt-stalk.html[/URL]](http://www.percona.com/doc/percona-toolkit/2.2/pt-stalk.html%5B/URL%5D) [URL][http://www.mysqlperformanceblog.com/2013/01/03/percona-toolkit-by-example-pt-stalk/[/URL]](http://www.mysqlperformanceblog.com/2013/01/03/percona-toolkit-by-example-pt-stalk/%5B/URL%5D) Generally you have to specify some trigger condition so it can start to collect data. i.e with some function, variable values, threshold etc. If you want to collect information right now and without waiting for any trigger occur then you can simply run pt-stalk --no-stalk and check the results files to figure out where is the problem. ---
**Author:** ![Serge\_Shakhov](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@Serge\_Shakhov](https://forums.percona.com/u/Serge_Shakhov)\ **Post date:** [June 13, 2013, 7:15pm UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/3 "2013-06-13T19:15:15Z")
> [@niljoshi;10533](#): > > Hi, I would like to suggest you to use pt-stalk (Percona utility for collecting data about MySQL when problem occur including trace and tcpdump ). You can get more information here that how to use it. Thank you. What impact on the system will it have? Because any additional workload can crash the system. ---
**Author:** ![niljoshi](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/niljoshi/32/16889_2.png) [@niljoshi](https://forums.percona.com/u/niljoshi)\ **Post date:** [June 19, 2013, 3:20am UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/4 "2013-06-19T03:20:19Z")
Hi, It will not impact that much on system but I would suggest to read documentation properly with all options and test is on stage server before running on prod. ---
**Author:** ![467675761](https://avatars.discourse-cdn.com/v4/letter/4/f14d63/32.png) [@467675761](https://forums.percona.com/u/467675761)\ **Post date:** [January 10, 2016, 8:17am UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/5 "2016-01-10T08:17:39Z")
I met the same situation.and more ,where the slave process start,the slave will eat up all the memory,all the swap space , and finally ,the mysqld process run outofmemory,and the kernel killed the mysqld worker process ,then ,mysql restart. ---
**Author:** ![carakod](https://avatars.discourse-cdn.com/v4/letter/c/ba8739/32.png) [@carakod](https://forums.percona.com/u/carakod)\ **Post date:** [January 11, 2016, 3:11pm UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/6 "2016-01-11T15:11:00Z")
Could it be the case [url][https://www.percona.com/blog/2014/01/21/beware-mysql-5-6-server-uuid-cloning-slaves/[/url]](https://www.percona.com/blog/2014/01/21/beware-mysql-5-6-server-uuid-cloning-slaves/%5B/url%5D) ? ---
**Author:** ![467675761](https://avatars.discourse-cdn.com/v4/letter/4/f14d63/32.png) [@467675761](https://forums.percona.com/u/467675761)\ **Post date:** [February 3, 2016, 10:04am UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/7 "2016-02-03T10:04:07Z")
> [@carakod;43571](#): > > Could it be the case [https://www.percona.com/blog/2014/01/21/beware-mysql-5-6-server-uuid-cloning-slaves/](https://www.percona.com/blog/2014/01/21/beware-mysql-5-6-server-uuid-cloning-slaves/) ? no ,it’s not the same case ---
**Author:** ![mirfan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mirfan/32/16890_2.png) [@mirfan](https://forums.percona.com/u/mirfan)\ **Post date:** [February 9, 2016, 4:15am UTC](https://forums.percona.com/t/replication-generates-huge-disk-write-io/2749/8 "2016-02-09T04:15:35Z")
If you are using row-based replication and tables missing primary key/unique key then you probably hitting this bug [url][MySQL Bugs: #53375: RBR + no PK =\> High load on slave (table scan/cpu) =\> slave failure](https://bugs.mysql.com/bug.php?id=53375%5B/url%5D) Try to find out If any of database tables have missing PK/UK [url][http://datacharmer.blogspot.com/2011/09/finding-tables-without-primary-keys.html[/url]](http://datacharmer.blogspot.com/2011/09/finding-tables-without-primary-keys.html%5B/url%5D) And add an PK auto-inc If required.