Slow full state transfer / wsrep over 10GBit

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

Hi together,

we have build up an five node percona-xtradb-cluster (version 8.0.22-13-1) on Ubuntu 20.04. Each Server has two dedicated 10GBit/s nics, 36 cores and 256 GB RAM and performs well when we do stuff like copy files etc.

Our config looks like:

[...]
wsrep_provider_options="gcache.size=5G;gcache.page_size=1G;gmcast.listen_addr=tcp://10.10.20.11:4567;socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem"
wsrep_slave_threads=18
wsrep_sst_method=xtrabackup-v2
performance_schema = True 
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem
inno-apply-opts="--use-memory=20G"
encrypt_threads=16
[...]

When I add a new node, a full state transfer is being started, but the network utilization is only 100-140 MB/s. We tried a lot by playing with options like

  • inno-backup-opts=“–parallel=4”
  • encrypt_threads=16
  • backup_threads=8
  • compressor=“pigz”
  • decompressor=“pigz -d”
  • sockopt=“,cipher=AES128”

but never come over 400 MB/sec with no load on the servers. After we copy the settings and add a new node, we come back to 100-140 MB/sec.

Can someone give me a tip what I can still try to use more network bandwidth?

1 Like
2

Hi @derbode ,

You can try adding more parallel copy and stream threads:

[sst]
. . .
inno-backup-opts="--parallel=16"
xbstream-opts="--parallel=16"

Please give it a try.

3 Likes
3

holy sh… you got it… Now I have over 800 MB/s. can you explain how You came across the parallel=16 values?

1 Like
4

The value of 16 I got from your encrypt threads, but the main bottleneck I belive you were facing is at stream.

SST will do the work (read/copy file, compress, encrypt ) then it will stream it to whatever streamming method you use. By default that is xbstream, which uses a single thread, unless you use --parallel at xbstream-opts .

1 Like
5

Thanks a lot… this was what we was looking for.

1 Like