# Replication thread spends too much time in "freeing items" state **URL:** \ **Category:** Other MySQL® Questions\ **Created:** [June 5, 2015, 4:14am UTC](https://forums.percona.com/t/replication-thread-spends-too-much-time-in-freeing-items-state/4249 "2015-06-05T04:14:09Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Alex.by](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/alex.by/32/1073_2.png) [@Alex.by](https://forums.percona.com/u/Alex.by)\ **Post date:** [June 5, 2015, 4:14am UTC](https://forums.percona.com/t/replication-thread-spends-too-much-time-in-freeing-items-state/4249/1 "2015-06-05T04:14:09Z")
Hi guys. I have a Master-Slave replication (both are 5.6.21-70.0-log Percona Server). Slave is used for XtraBackup + some SELECT queries are executed once in awhile. We also have percona agent running on a server (for Percona Cloud Tools). Red bars (CPU usage) on image are caused by XtraBackup: [ATTACH=CONFIG]temp\_337\_1433497825592\_323[/ATTACH] ( [url][http://i.imgur.com/ZkUuIQa.png[/url]](http://i.imgur.com/ZkUuIQa.png%5B/url%5D) ) Replication gap grows as shown on image (graphs show last 7 days). I can reboot slave - things will be fine for 2-3 days but gap will start growing again as shown on a graph. [ATTACH=CONFIG]temp\_335\_1433497845981\_524[/ATTACH] ( [url][http://i.imgur.com/sudZKPv.png[/url]](http://i.imgur.com/sudZKPv.png%5B/url%5D) ) The problem is - gap appears even before XtraBackup is scheduled to run. It just appears when traffic goes up a bit and becomes worth and worth every day. In 9 out of 10 “SHOW PROCESSLIST” cases thread is in “freeing items”: mysql\> show processlist; | 212 | system user | | NULL | Connect | 861178 | Waiting for master to send event | NULL | 0 | 0 | | 104717 | percona-agent | localhost | NULL | Sleep | 0 | | NULL | 403 | 403 | | 104723 | system user | | NULL | Connect | 7368 | freeing items | NULL | 0 | 0 | So replication gap is 7368 seconds. No queries are executed at that time. It doesn’t matter whether xtrabackup is running or not - amount of time thread spends at ‘freeing items’ visually looks the same for the whole day. I assume the reason gap goes down is because we have 2 times less traffic at that period of a day (US oriented service). /etc/my.cnf: [mysqld\_safe] malloc-lib = /usr/lib64/libjemalloc.so.1 [mysqld] … default-storage-engine = innodb replicate-wild-do-table=db\_name.% key\_buffer\_size = 16K max\_allowed\_packet = 1M read\_buffer\_size = 256K read\_rnd\_buffer\_size = 256K net\_buffer\_length = 2K max\_connections = 2000 server-id = 2 log-bin=mysql-bin expire\_logs\_days=4 max\_binlog\_size=20873534 query\_cache\_limit = 0 query\_cache\_size = 0 query\_cache\_type = 0 innodb\_buffer\_pool\_size = 23G innodb\_log\_file\_size = 2G innodb\_log\_buffer\_size = 8M innodb\_flush\_log\_at\_trx\_commit = 2 innodb\_flush\_method = O\_DIRECT innodb\_file\_per\_table innodb\_read\_io\_threads = 4 innodb\_write\_io\_threads = 8 innodb\_thread\_concurrency = 0 innodb\_io\_capacity = 600 innodb\_support\_xa = 0 innodb\_stats\_on\_metadata = 0 innodb\_buffer\_pool\_instances = 4 tmp\_table\_size = 128M max\_heap\_table\_size = 128M long\_query\_time = 1.0 slow\_query\_log = 0 slow\_query\_log\_file = /var/lib/mysql/slow-query.log slow\_query\_log\_timestamp\_always = 1 slow\_query\_log\_timestamp\_precision = microsecond slow\_query\_log\_use\_global\_control = all log\_slow\_verbosity = full Below is “SHOW GLOBAL STATUS” difference with 300 seconds interval: Binlog\_snapshot\_position diff is 1204 Binlog\_cache\_use diff is 1 Binlog\_stmt\_cache\_use diff is 5 Bytes\_received diff is 177512897 Bytes\_sent diff is 1138503753 Com\_admin\_commands diff is 52 Com\_begin diff is 467110 Com\_commit diff is 467111 Com\_create\_index diff is 1 Com\_create\_table diff is 2 Com\_delete diff is 2222 Com\_drop\_table diff is 3 Com\_insert diff is 6109 Com\_select diff is 8307 Com\_set\_option diff is 12 Com\_show\_slave\_status diff is 17 Com\_show\_status diff is 307 Com\_show\_variables diff is 1 Com\_update diff is 467563 Connections diff is 35 Created\_tmp\_disk\_tables diff is 3 Created\_tmp\_files diff is 3 Created\_tmp\_tables diff is 314 Handler\_commit diff is 951268 Handler\_delete diff is 18768 Handler\_external\_lock diff is 968327 Handler\_prepare diff is 4 Handler\_read\_first diff is 6 Handler\_read\_key diff is 613326 Handler\_read\_last diff is 1 Handler\_read\_next diff is 12186897 Handler\_read\_rnd diff is 1008274 Handler\_read\_rnd\_next diff is 25303902 Handler\_update diff is 1541282 Handler\_write diff is 6675629 Innodb\_background\_log\_sync diff is 298 Innodb\_buffer\_pool\_pages\_data diff is 924 Innodb\_buffer\_pool\_bytes\_data diff is 15138816 Innodb\_buffer\_pool\_pages\_dirty diff is 1476 Innodb\_buffer\_pool\_bytes\_dirty diff is 24182784 Innodb\_buffer\_pool\_pages\_flushed diff is 62591 Innodb\_buffer\_pool\_pages\_free diff is 3 Innodb\_buffer\_pool\_pages\_made\_not\_young diff is 65532 Innodb\_buffer\_pool\_pages\_made\_young diff is 99828 Innodb\_buffer\_pool\_pages\_old diff is 341 Innodb\_buffer\_pool\_read\_requests diff is 72325656 Innodb\_buffer\_pool\_reads diff is 15456 Innodb\_buffer\_pool\_write\_requests diff is 3302830 Innodb\_data\_fsyncs diff is 6258 Innodb\_data\_read diff is 310525952 Innodb\_data\_reads diff is 18954 Innodb\_data\_writes diff is 531176 Innodb\_data\_written diff is 2468012032 Innodb\_dblwr\_pages\_written diff is 62591 Innodb\_dblwr\_writes diff is 706 Innodb\_history\_list\_length diff is 108 Innodb\_ibuf\_merged\_delete\_marks diff is 27967 Innodb\_ibuf\_merged\_deletes diff is 8659 Innodb\_ibuf\_merged\_inserts diff is 6907 Innodb\_ibuf\_merges diff is 8535 Innodb\_log\_write\_requests diff is 360723 Innodb\_log\_writes diff is 467335 Innodb\_lsn\_current diff is 177732798 Innodb\_lsn\_flushed diff is 177628594 Innodb\_lsn\_last\_checkpoint diff is 192034095 Innodb\_master\_thread\_active\_loops diff is 298 Innodb\_max\_trx\_id diff is 937646 Innodb\_mutex\_os\_waits diff is 2568 Innodb\_mutex\_spin\_rounds diff is 112751 Innodb\_mutex\_spin\_waits diff is 16936 Innodb\_os\_log\_fsyncs diff is 495 Innodb\_os\_log\_written diff is 417008640 Innodb\_pages\_created diff is 652 Innodb\_pages\_read diff is 18953 Innodb\_pages\_written diff is 62591 Innodb\_purge\_trx\_id diff is 937647 Innodb\_rows\_deleted diff is 18768 Innodb\_rows\_inserted diff is 48277 Innodb\_rows\_read diff is 32417934 Innodb\_rows\_updated diff is 512089 Innodb\_s\_lock\_os\_waits diff is 492 Innodb\_s\_lock\_spin\_rounds diff is 24043 Innodb\_s\_lock\_spin\_waits diff is 2070 Innodb\_x\_lock\_os\_waits diff is 1163 Innodb\_x\_lock\_spin\_rounds diff is 52583 Innodb\_x\_lock\_spin\_waits diff is 2282 Key\_read\_requests diff is 25033329 Key\_reads diff is 8320794 Key\_write\_requests diff is 3064203 Key\_writes diff is 3052392 Opened\_files diff is 281 Opened\_table\_definitions diff is 6 Opened\_tables diff is 411 Queries diff is 951980 Questions diff is 8686 Select\_range diff is 8253 Select\_scan diff is 317 Slow\_queries diff is 10214 Table\_locks\_immediate diff is 484163 Table\_open\_cache\_hits diff is 484162 Table\_open\_cache\_misses diff is 3 Threads\_connected diff is 1 Uptime diff is 300 Uptime\_since\_flush\_status diff is 300 Few more things: - XtraBackup is running on the same machine. It works with NFS directory. - Slave was running well for months. Issue started in a week after XtraBackup was configured on a server. - Server is running on CentOS 6 Appreciate any help around the issue. Will be happy to provide more data if needed. Thanks in advance! Alex