# sudden transaction rate slowdown during sysbench test **URL:** \ **Category:** Percona XtraDB Cluster 8.x\ **Created:** [August 13, 2020, 5:06am UTC](https://forums.percona.com/t/sudden-transaction-rate-slowdown-during-sysbench-test/7901 "2020-08-13T05:06:13Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![split-brain](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/split-brain/32/65_2.png) [@split-brain](https://forums.percona.com/u/split-brain)\ **Post date:** [August 13, 2020, 5:06am UTC](https://forums.percona.com/t/sudden-transaction-rate-slowdown-during-sysbench-test/7901/1 "2020-08-13T05:06:13Z")
Hi there! I’m testing PXC 8 with Sysbench and ran into some weird problem, where after about and hour of testing, the QPS rate went down from approx. 10000 qps to approx. 2000 qps. From what I’ve observed so far, the only suspicious value is TCP retransmission rate, which (at the time of slowdown)  went from approx. 0 ops for all nodes to approx. 3 ops for node1 and node2, to approx. 13 ops for node3 (the writer node). At 8:47 UTC node1 sent couple of FC messages and at 8:49 UTC the cluster latency went up from around 1.5ms to around 20ms and the qps rate went down from approx. 10000 qps to approx. 2000 qps. From 8:49 UTC there were no additional FC messages being sent and the cluster just almost stalled. This actually happened several times, but now I can document it. Can you please point me to the cause of this problem? There are 3 PXC nodes in one AWS region, each residing in separate AZ. The fourth node is running ProxySQL and I’m running Sysbench on that node locally. ProxySQL has no query rules defined (no R/W split), only simple mysql\_galera\_hostgroups set with max\_writers=1 (node3 is the writer here, node1 and node2 is only applying wsrep). PMM is running on fifth node. All of them are t3.medium type with 4GB of RAM. ```auto sysbench /usr/share/sysbench/oltp_read_write.lua --db-driver=mysql --mysql-host=127.0.0.1 --mysql-user='sbuser' --mysql-password='sbpass' --mysql-port=6033 --mysql-db=sbtest --tables=1 --table_size=1000000 --db-ps-mode=disable --threads=16 --report-interval=1 --time=3600 --skip-trx=off --mysql-ignore-errors=all run ``` ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/4a49b6588628fed14f0d468b00e0785e071ad32a.png "Image: /uploads/percona1/original/1X/4a49b6588628fed14f0d468b00e0785e071ad32a.png") ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/c0c3411420c2890f83ed32203c03663367167329.png) ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/ae12846ea263a24e4b7616e1eca81f92573b9368.png) ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/59af55b3845fa477c6f2c4fdee68139f8c794ca4.png) ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/b7ca60a2542bcc3915dd317356f5799317f01e7f.png) ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/a2676ba79f4f1580cacc2f9c64be4b60a6b79d44.png) ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/3476ffce7a52ab1df3f1ad9fe05fb55850015ac4.png) ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/d84ea94ecefd8fe6107e788cc023947d3c722c49.png "Image: /uploads/percona1/original/1X/d84ea94ecefd8fe6107e788cc023947d3c722c49.png") ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/cdcf641da71381f52bb2015ef13bcb1a1176e44b.png "Image: /uploads/percona1/original/1X/cdcf641da71381f52bb2015ef13bcb1a1176e44b.png") ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/66aeca2c85bcc8933194be3017afd60b023c2060.png "Image: /uploads/percona1/original/1X/66aeca2c85bcc8933194be3017afd60b023c2060.png") [sysbench\_report\_from\_the\_time\_of\_slowndown.txt](https://forums.percona.com/uploads/short-url/lMIMTojFjhJnCKI54N6xUljIYgj.txt) (13.1 KB) [node1-myq\_status\_from\_the\_time\_of\_slowndown…txt](https://forums.percona.com/uploads/short-url/u4GDMCUyplboUwdT14RDAa6ybEg.txt) (48.7 KB) ---
**Author:** ![split-brain](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/split-brain/32/65_2.png) [@split-brain](https://forums.percona.com/u/split-brain)\ **Post date:** [August 17, 2020, 6:08am UTC](https://forums.percona.com/t/sudden-transaction-rate-slowdown-during-sysbench-test/7901/2 "2020-08-17T06:08:04Z")
I’m testing it again and now I have only 2 nodes in the cluster. After some time there is surge at receive/send queue and cluster hangs due to FC. mysqld.log shows some gcache pages being created/deleted. shall I increase the gcache size? > [@](#): > > 2020-08-17T11:04:04.333437Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000036 of size 134217728 bytes > 2020-08-17T11:04:04.534960Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000035 > 2020-08-17T11:06:25.791539Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000037 of size 134217728 bytes > 2020-08-17T11:06:26.007878Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000036 > 2020-08-17T11:08:47.387363Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000038 of size 134217728 bytes > 2020-08-17T11:08:47.544381Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000037 > 2020-08-17T11:11:09.973613Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000039 of size 134217728 bytes > 2020-08-17T11:11:10.269839Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000038 > 2020-08-17T11:13:31.655732Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000040 of size 134217728 bytes > 2020-08-17T11:13:31.753867Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000039 > 2020-08-17T11:15:53.698146Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000041 of size 134217728 bytes > 2020-08-17T11:15:53.805761Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000040 > 2020-08-17T11:18:16.027883Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/gcache.page.000042 of size 134217728 bytes > 2020-08-17T11:18:16.380256Z 0 [Note] [MY-000000] [Galera] Deleted page /var/lib/mysql/gcache.page.000041 ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/e5007047c3aeae3af1c3d1dc4145a000a67b897b.png)