# Slow full state transfer / wsrep over 10GBit

**URL:** <https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986>\
**Category:** Percona XtraDB Cluster 8.x\
**Tags:** mysql, percona\
**Created:** [April 7, 2021, 6:54am UTC](https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986 "2021-04-07T06:54:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![derbode](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@derbode](https://forums.percona.com/u/derbode)\
**Post date:** [April 7, 2021, 6:54am UTC](https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986/1 "2021-04-07T06:54:35Z")

</div>

Hi together,

we have build up an five node percona-xtradb-cluster (version 8.0.22-13-1) on Ubuntu 20.04. Each Server has two dedicated 10GBit/s nics, 36 cores and 256 GB RAM and performs well when we do stuff like copy files etc.

Our config looks like:

```
[...]
wsrep_provider_options="gcache.size=5G;gcache.page_size=1G;gmcast.listen_addr=tcp://10.10.20.11:4567;socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"
wsrep_slave_threads=18
wsrep_sst_method=xtrabackup-v2
performance_schema = True 
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem
inno-apply-opts="--use-memory=20G"
encrypt_threads=16
[...]

```

When I add a new node, a full state transfer is being started, but the network utilization is only 100-140 MB/s. We tried a lot by playing with options like

- inno-backup-opts=“–parallel=4”
- encrypt\_threads=16
- backup\_threads=8
- compressor=“pigz”
- decompressor=“pigz -d”
- sockopt=“,cipher=AES128”

but never come over 400 MB/sec with no load on the servers. After we copy the settings and add a new node, we come back to 100-140 MB/sec.

Can someone give me a tip what I can still try to use more network bandwidth?

---

<div class="post-metadata">

**Author:** ![Marcelo\_Altmann](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/marcelo_altmann/32/11299_2.png) [@Marcelo\_Altmann](https://forums.percona.com/u/Marcelo_Altmann)\
**Post date:** [April 7, 2021, 11:40am UTC](https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986/2 "2021-04-07T11:40:37Z")

</div>

Hi @derbode ,

You can try adding more parallel copy and stream threads:

```auto
[sst]
. . .
inno-backup-opts="--parallel=16"
xbstream-opts="--parallel=16"

```

Please give it a try.

---

<div class="post-metadata">

**Author:** ![derbode](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@derbode](https://forums.percona.com/u/derbode)\
**Post date:** [April 7, 2021, 3:48pm UTC](https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986/3 "2021-04-07T15:48:15Z")

</div>

holy sh… you got it… Now I have over 800 MB/s. can you explain how You came across the parallel=16 values?

---

<div class="post-metadata">

**Author:** ![Marcelo\_Altmann](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/marcelo_altmann/32/11299_2.png) [@Marcelo\_Altmann](https://forums.percona.com/u/Marcelo_Altmann)\
**Post date:** [April 7, 2021, 4:26pm UTC](https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986/4 "2021-04-07T16:26:02Z")

</div>

The value of 16 I got from your encrypt threads, but the main bottleneck I belive you were facing is at stream.

SST will do the work (read/copy file, compress, encrypt ) then it will stream it to whatever streamming method you use. By default that is xbstream, which uses a single thread, unless you use `--parallel` at `xbstream-opts` .

---

<div class="post-metadata">

**Author:** ![derbode](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@derbode](https://forums.percona.com/u/derbode)\
**Post date:** [April 8, 2021, 7:11am UTC](https://forums.percona.com/t/slow-full-state-transfer-wsrep-over-10gbit/9986/5 "2021-04-08T07:11:14Z")

</div>

Thanks a lot… this was what we was looking for.