No SST with TLS enabled

Hi Percona xtradb users,

I just set up my first Percona xtradb cluster using Ubuntu 24.04.2.

These packages are currently installed:

    ii  percona-release                      1.0-30.generic                          all          Package to install Percona gpg key and APT repos
    ii  percona-telemetry-agent              1.0.3-4.noble                           amd64        Percona Telemetry Agent gathers information and metrics from Percona products installed on the host.
    ii  percona-xtrabackup-84                8.4.0-2-1.noble                         amd64        Open source backup tool for InnoDB and XtraDB
    ii  percona-xtrabackup-dbg-84            8.4.0-2-1.noble                         amd64        Debug symbols for Percona XtraBackup
    ii  percona-xtradb-cluster               1:8.0.41-32-1.noble                     amd64        Percona XtraDB Cluster with Galera
    ii  percona-xtradb-cluster-client        1:8.0.41-32-1.noble                     amd64        Percona XtraDB Cluster database client binaries
    ii  percona-xtradb-cluster-common        1:8.0.41-32-1.noble                     amd64        Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
    ii  percona-xtradb-cluster-server        1:8.0.41-32-1.noble                     amd64        Percona XtraDB Cluster database server binaries

Here you have the config of the first node.

    # Template my.cnf for PXC
    # Edit to your requirements.
    [client]
    socket=/var/run/mysqld/mysqld.sock

    [mysqld]
    server-id=1
    datadir=/var/lib/mysql
    socket=/var/run/mysqld/mysqld.sock
    log-error=/var/log/mysql/error.log
    pid-file=/var/run/mysqld/mysqld.pid
    user= mysql

    # Binary log expiration period is 604800 seconds, which equals 7 days
    binlog_expire_logs_seconds=604800

    ######## wsrep ###############
    # Path to Galera library
    wsrep_provider=/usr/lib/galera4/libgalera_smm.so

    # Cluster connection URL contains IPs of nodes
    #If no IP is found, this implies that a new cluster needs to be created,
    #in order to do that you need to bootstrap this node
    wsrep_cluster_address=gcomm://192.168.1.10,192.168.1.11,192.168.1.12,

    # In order for Galera to work correctly binlog format should be ROW
    binlog_format=ROW

    # Slave thread to use
    wsrep_slave_threads=8

    wsrep_log_conflicts

    # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
    innodb_autoinc_lock_mode=2

    # Node IP address
    wsrep_node_address=192.168.1.10
    # Cluster name
    wsrep_cluster_name=testcluster

    [mysqld]

    # TLS/SSL Usage
    wsrep_provider_options         = "socket.ssl_cert=/etc/ssl/galera/cluster_1/node01.domain.tld.crt;socket.ssl_key=/etc/ssl/galera/cluster_1/node01.domain.tld.key;socket.ssl_ca=/etc/ssl/certs/CACRT.pem;ssl_cipher=tlsv1.2"

    pxc_encrypt_cluster_traffic=ON
    #
    [sst]
    encrypt=4
        ssl-ca=/etc/ssl/certs/CACRT.pem
    ssl-cert=/etc/ssl/galera/cluster_1/node01.domain.tld.crt
    ssl-key=/etc/ssl/galera/cluster_1/node01.domain.tld.key

    #If wsrep_node_name is not specified,  then system hostname will be used
    wsrep_node_name=node01.domain.tld

    #pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
    pxc_strict_mode=ENFORCING

    # SST method
    wsrep_sst_method=xtrabackup-v2

The configs of the other two nodes differs in wsrep_node_address, wsrep_node_name, and certname(key|crt).
Bootstrapping the first node can be executed without errors.

As soon as I start node2 to join the cluster, I receive error messages on the donor node

  2025-05-06T12:30:25.821752Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error

And these messages on node2:

2025-05-06T12:30:18.313375Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: invalid padding: certificate signature failure

I used self signed certs on both nodes, signed with the same CA, which is integrated into the local ca store of the os.
Checking the self signed cert show no errors:

openssl verify -verbose -CAfile /etc/ssl/certs/CACRT.pem  /etc/ssl/galera/cluster_1/node01.domain.tld.crt
/etc/ssl/galera/cluster_1/node01.domain.tld.crt: OK

The user mysql is able to read the needed cert files.

I also tried adding “socat_options=openssl-cipher=TLSv1.2,TLSv1.3” and ‘wsrep_provider_options = “
;ssl_cipher=tlsv1.2”’ to the config with no success.

Performing a SST is posible if I switch off the transport security “pxc_encrypt_cluster_traffic=OFF”.
Do I have an issue with Percona xtradb cluster under Ubuntu 24, where TLS 1.0, 1.1 and DTLS 1.0 are forcefully disabled Ubuntu 24.04 LTS (Noble Numbat) Release Notes - Release - Ubuntu Community Hub
Or are there some other issues within my installation?

Thx in advance
mrks

Hey @mrks,

The Common Name value used for the server and client keys and certificates must differ from that value used for the CA certificate.

Check the docs on this, Encrypt PXC traffic - Percona XtraDB Cluster

Ensure the CN is correct for CA and the server certs.

Thanks for your reply. I checked this and this was indeed an issue.
The first crt just had multiple Subject Alternative Name set. But no CN. So I created new certs on every cluster member

root@node01:# openssl x509 -in /etc/ssl/galera/cluster_1/node01.domain.tld.crt -subject -issuer -noout
subject=..., CN = node01.domain.tld, ...
issuer=..., CN = SELF_CA

Started to bootstrap on node1, and tried to join with node2. Without any specification of tls settings (socat_options or wsrep_provider_options).
But the error still occurs.

donor node:

2025-05-06T14:49:47.397900Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error

joiner node:

2025-05-06T14:49:47.400133Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: invalid padding: certificate signature failure
2025-05-06T14:49:48.390713Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Connection refused

:thinking:

Support for the TLSv1 and TLSv1.1 connection protocols is removed as of MySQL 8.0.28.

Make sure your certs are TLS v1.2, or better, v1.3 compatible.

The supported and used TLS protocols for the secured connection depends on the available ciphers (on both ends of the connection server/client). At least that’s how I understood it. And this should depend on the available ciphers provided by openssl.

openssl version
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

That should be sufficient.

Some older certificates with old and unsecure algorithms (SHA-1 or 1024-Bit-RSA) may not be useable anymore.
The used certificates where generated with modern standards (Public Key Algorithm: rsaEncryption, Public-Key: 4096 bit),

Here https://dev.mysql.com/doc/refman/8.0/en/encrypted-connection-protocols-ciphers.html#encrypted-connection-deprecated-protocols it’s mentioned, that newer mysqld versions (MySQL 8.0.28 and above) only support TLSv1.2, TLSv1.3.

mysqld --version
/usr/sbin/mysqld  Ver 8.0.41-32.1 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel32, Revision 9cd31bf, WSREP version 26.1.4.3)

This should be the case, based on the docs, with mysqld Ver 8.0.41-32.1.

So, why do I get those log messages:

2025-05-06T14:49:47.397900Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error

When TLSv1 is not available in mysqld Ver 8.0.41-32.1.

I’m the instructor for our Percona XtraDB Cluster Tutorial. In the class, we use SSL/TLS everywhere (sst, ist, galera, mysql-client, etc) on the latest PXC version. Here are the certs that are used, along with some docs for generating certs: training-aws/packer/etc_ssl_mysql at master · percona/training-aws · GitHub

Can you try generating certs similar to what we use in the class, and see if that works? I’m not an SSL expert, so the best I can do is show you exactly what we use, and what we know works. Maybe there’s some difference between my cert generation and yours.

Thank you for sharing the link to your sources for creating those certs. I used your readme to create the needed certs, copied them to the servers, fix ownership and permissions

root@node01:/var/lib/mysql# ls -lah /etc/ssl/galera/cluster_1/
total 40K
dr-x------ 2 mysql mysql 4,0K May  7 08:55 .
dr-x------ 3 mysql mysql 4,0K May  6 16:46 ..
-rw-r--r-- 1 mysql mysql 1,3K May  7 08:55 ca.pem
-r-------- 1 mysql mysql 1,9K May  5 13:34 galera_rep.crt
-r-------- 1 mysql mysql 3,2K May  5 13:34 galera_rep.key
-r-------- 1 mysql mysql 1,4K May  7 08:55 node01.cert.pem
-r-------- 1 mysql mysql 1,7K May  7 08:55 node01.key.pem
-r-------- 1 mysql mysql 1,1K May  7 08:55 node01.req.pem

I checked my my.cnf again and used your hint Percona XtraDB cluster not available after bootstrap the first node - #4 by matthewb and grouped the sst section to the end at the config file, and removed the doubled [mysqld] section.

# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
user= mysql

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800

######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.1.10,192.168.1.11,192.168.1.12

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Slave thread to use
wsrep_slave_threads=8

wsrep_log_conflicts

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=192.168.1.10
# Cluster name
wsrep_cluster_name=testcluster

# TLS/SSL Usage
wsrep_provider_options="gcache.size=512M;cert.log_conflicts=yes;socket.ssl=yes;socket.ssl_cert=/etc/ssl/galera/cluster_1/node01.cert.pem;socket.ssl_key=/etc/ssl/galera/cluster_1/node01.key.pem;socket.ssl_ca=/etc/ssl/galera/cluster_1/ca.pem"

pxc_encrypt_cluster_traffic=ON

#If wsrep_node_name is not specified,  then system hostname will be used
wsrep_node_name=node1

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

[sst]
encrypt=4
ssl-ca=/etc/ssl/galera/cluster_1/ca.pem
ssl-cert=/etc/ssl/galera/cluster_1/node01.cert.pem
ssl-key=/etc/ssl/galera/cluster_1/node01.key.pem

In my opinion, this looks ok. After bootstrapping, the donor and joiner node react unchanged.
The I tried to check the certificate and the available tls versions.

So I used this command, to interact with the donner node

openssl s_client -connect 192.168.1.10:4567 -tls1_2

Here I saw

 0 s:CN = MySQL_Server_8.0.41-32_Auto_Generated_Server_Certificate
   i:CN = MySQL_Server_8.0.41-32_Auto_Generated_CA_Certificate

While I am not sure, if on port 4567 the SST cert should be used, this first looked strange to me.
So I copied my self signed certs to /var/lib/mysql and symlinked them to mysql default certs.
After that, I bootstrapped the cluster again.
The errors on doner and joiner node are unchanged and the second node cant join. Using the same cert as on node01 or using a different cert, does not matter.
But the CN changed as expected.

depth=1 C = US, ST = Anywhere, L = MyCity, O = Percona, OU = TrainingDept, CN = MyCoolCA
verify return:1
depth=0 C = US, ST = Anywhere, L = MyCity, O = Percona, OU = TrainingDept, CN = node01
verify return:1

So I checked the bootstrapped node and here I saw that although wsrep_provider_options is set in the config file, this values are not loaded.

Donor node, bootstrapping state:

mysql> SHOW GLOBAL VARIABLES LIKE '%wsrep_provider_options%';
| Variable_name  | Value
+-------------+-------------+
| wsrep_provider_options | allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 178.63.126.95; base_port = 4567; cert.log_conflicts = yes; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 4; evs.version = 1; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 512M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = ssl://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 192.168.1.10; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;  |
+-------------+-------------+
1 row in set (0,01 sec)

and

mysql> SHOW GLOBAL VARIABLES LIKE 'ssl%';
+---------------------------+-----------------+
| Variable_name             | Value           |
+---------------------------+-----------------+
| ssl_ca                    | ca.pem          |
| ssl_capath                |                 |
| ssl_cert                  | server-cert.pem |
| ssl_cipher                |                 |
| ssl_crl                   |                 |
| ssl_crlpath               |                 |
| ssl_fips_mode             | OFF             |
| ssl_key                   | server-key.pem  |
| ssl_session_cache_mode    | ON              |
| ssl_session_cache_timeout | 300             |
+---------------------------+-----------------+
10 rows in set (0,00 sec)

Does the bootstrap process uses a different my.cnf? Or is my config faulty?

Add pxc_encrypt_cluster_traffic=ON to [mysqld], though this is the default and should copy all SSL related parameters to wsrep_provider_options.

That’s the case. I pasted the latest config of the donor node here No SST with TLS enabled - #7 by mrks

I spun up some new EC2 instances to test this out. Some output is removed for brevity. tl;dr: SSL SST/IST works as expected.

Nothing redacted from my.cnf. This is the exact file contents: (same for mysql2 just different IP)

[root@mysql1-T1 ~]# cat /etc/my.cnf
# The section 'client' is used by any application which links against
# the 'libmysqlclient' library, which is most tools and languages.
[client]
socket = /var/lib/mysql/mysql.sock
ssl_ca = /etc/ssl/mysql/ca.pem
ssl_cert = /etc/ssl/mysql/client-cert.pem
ssl_key = /etc/ssl/mysql/client-key.pem

# The 'mysql' section is used only by the 'mysql' CLI
[mysql]
prompt = "mysql1-T1 mysql> "

# The 'mysqld' section is specific to parameters which modify the behavior of the mysql server-daemon itself
[mysqld]
user = mysql
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysqld.pid
event_scheduler = off
percona_telemetry_disable = 1

innodb_buffer_pool_size = 3G

server_id = 17

log-bin = mysqld-bin
binlog_format = ROW
sync_binlog = 0
log_replica_updates

gtid-mode = ON
enforce-gtid-consistency = 1

skip-name-resolve
log-error = /var/log/mysqld.log
log_error_verbosity = 3

slow_query_log = on
slow_query_log_file = '/var/lib/mysql/slow.log'
log_slow_verbosity = full
long_query_time = 0
secure-file-priv = /var/lib/mysql-files/

explicit_defaults_for_timestamp = 1
sysdate-is-now = 1

ssl_ca = /etc/ssl/mysql/ca.pem
ssl_cert = /etc/ssl/mysql/server-cert.pem
ssl_key = /etc/ssl/mysql/server-key.pem

wsrep_provider                  = /usr/lib64/galera4/libgalera_smm.so
wsrep_cluster_address           = gcomm://mysql1,mysql2,mysql3
wsrep_node_address              = 10.11.95.234
wsrep_cluster_name              = mycluster
wsrep_node_name                 = mysql1
wsrep_log_conflicts

Checking:

[root@mysql1-T1 ~]# mysql
Server version: 8.0.41-32.1 Percona XtraDB Cluster (GPL), Release rel32, Revision 9cd31bf, WSREP version 26.1.4.3

mysql1-T1 mysql> SHOW GLOBAL VARIABLES LIKE '%ssl%';
+-------------------------------------+--------------------------------+
| Variable_name                       | Value                          |
+-------------------------------------+--------------------------------+
...
| have_openssl                        | YES                            |
| have_ssl                            | YES                            |
| ssl_ca                              | /etc/ssl/mysql/ca.pem          |
| ssl_capath                          |                                |
| ssl_cert                            | /etc/ssl/mysql/server-cert.pem |
| ssl_cipher                          |                                |
| ssl_crl                             |                                |
| ssl_crlpath                         |                                |
| ssl_fips_mode                       | OFF                            |
| ssl_key                             | /etc/ssl/mysql/server-key.pem  |

TLS checks:

# openssl s_client -connect 127.0.0.1:4567 -tls1
Connecting to 127.0.0.1
CONNECTED(00000003)
00CE37EEF67F0000:error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available:ssl/statem/statem_lib.c:153:
---
no peer certificate available
---
No client certificate CA names sent

TLS 1.2

# openssl s_client -connect 127.0.0.1:4567 -tls1_2
Connecting to 127.0.0.1
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolCA
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolCA
verify return:1
depth=0 C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolServer
verify return:1
000E4877177F0000:error:0A000410:SSL routines:ssl3_read_bytes:ssl/tls alert handshake failure:ssl/record/rec_layer_s3.c:909:SSL alert number 40

Certificate chain
 0 s:C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolServer
   i:C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolCA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec  3 21:39:06 2024 GMT; NotAfter: Dec  1 21:39:06 2034 GMT
 1 s:C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolCA
   i:C=US, ST=Anywhere, L=MyCity, O=Percona, OU=TrainingDept, CN=MyCoolCA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec  3 21:38:47 2024 GMT; NotAfter: Dec  1 21:38:47 2034 GMT
...
    New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
   Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    No ALPN negotiated
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    ...
    Verify return code: 19 (self-signed certificate in certificate chain)

Verified that the donor, mysql1, does not support TLS1 nor 1.1, and does support 1.2.

Watch SST, port 4444, packets from mysql1 → mysql2 when starting mysql2 (which has no data and thus must force SST). port 4567 is regular Galera traffic, and port 4568 is IST.

# ngrep -d eth0 port 4444
interface: eth0 (10.11.0.0/255.255.0.0)
filter: ( port 4444 ) and ((ip || ip6) || (vlan && (ip || ip6)))
####
T 10.11.95.234:40334 -> 10.11.4.178:4444 [AP] #4
  ....d...`...W.a....,..7.....GA.....z....A{. .rV.=..\.D......R..&0...>.b5."\U.t.........,.0.............].a.S.+./.......\.`.R.$.(.k.s.w...#.'.g.r.v.......9.
  ......3.E.....Q.....P.=...<...5.../.A.......................#.............". .................................+........-.....3.G.E...A...9m...FHT..9.......
  +H.j...nU.L.9.....*.s.`U..G............I...........

Definitely encrypted SST. Checking logs, we see explicit SSL on 4567, but nothing explicit for 4444. The assumption is that if IST is SSL’d, then so is SST.

2025-05-07T15:56:44.095035Z 0 [Note] [MY-000000] [Galera] (7fc08081-a26f, 'ssl://0.0.0.0:4567') connection established to df89f5b3-94cd ssl://10.11.4.178:4567
2025-05-07T15:56:44.589748Z 0 [Note] [MY-000000] [Galera] declaring df89f5b3-94cd at ssl://10.11.4.178:4567 stable
...

This is mysql2 (joiner) log entries:

2025-05-07T16:05:54.880413Z 0 [Note] [MY-000000] [Galera] (27d4544d-9bfb, 'ssl://0.0.0.0:4567') connection established to 7fc08081-a26f ssl://10.11.95.234:4567
2025-05-07T16:05:55.369977Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2025-05-07T16:05:55.370056Z 0 [Note] [MY-000000] [Galera] declaring 7fc08081-a26f at ssl://10.11.95.234:4567 stable
...
2025-05-07T16:05:56.728458Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://10.11.4.178:4568
2025-05-07T16:05:56.728536Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl
2025-05-07T16:05:56.728706Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-4, listening at: ssl://10.11.4.178:4568
2025-05-07T16:05:56.729220Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mysql2) requested state transfer from '*any*'. Selected 1.0 (mysql1)(SYNCED) as donor.

I did notice that SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options'; gets truncated on output. We can’t see all the SSL variables, however, if you look in the error log, all provider_options are displayed:

2025-05-07T15:39:44.373926Z 0 [Note] [MY-000000] [Galera] Skipped GCache ring buffer recovery: could not determine history UUID.
2025-05-07T15:39:44.376663Z 0 [Note] [MY-000000] [Galera] Passing config to GCS:
...
socket.ssl = YES; socket.ssl_ca = /etc/ssl/mysql/ca.pem; 
socket.ssl_cert = /etc/ssl/mysql/server-cert.pem; 
socket.ssl_cipher = ; 
socket.ssl_key = /etc/ssl/mysql/server-key.pem; 
socket.ssl_reload = 1;

I suggest comparing what I’ve got with what you’ve got and see where the issue originates.

1 Like

I just wiped the systems to start from scratch.
Thank your for your detailed info.

Now I got it working. Thanks for your support @matthewb
These are my roughly setup instructions, in case someone has/had the same problem.

  • Setup 3 Ubuntu 24 hosts
  • Installation of percona-xtradb-cluster 8.0
echo "install percona-xtradb-cluster 8.0"
sudo apt update
sudo apt install -y wget gnupg2 lsb-release curl
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
sudo dpkg -i percona-release_latest.generic_all.deb
sudo apt update
sudo percona-release setup pxc80 
sudo apt install -y percona-xtradb-cluster
  • Installation of percona-xtrabackup 8.0
echo "install percona-xtrabackup 8.0"
sudo apt update
sudo apt install curl
curl -O https://repo.percona.com/apt/percona-release_latest.generic_all.deb
sudo apt install gnupg2 lsb-release ./percona-release_latest.generic_all.deb
sudo apt update
sudo percona-release setup pxb-80
sudo apt install percona-xtrabackup-80
xtrabackup --version
ssl_ca = /etc/ssl/mysql/ca.pem
ssl_cert = /etc/ssl/mysql/server-cert.pem
ssl_key = /etc/ssl/mysql/server-key.pem
  • Modified all three /etc/mysql/mysql.conf.d/mysqld.cnf on all three nodes.
  • changed wsrep_cluster_address=gcomm://host1,host2, host3 (same value an each node)
  • Set wsrep_node_address (to IP adress of each machine)
  • Set wsrep_cluster_name (same value an each node)
  • Set wsrep_node_name=hostname of each node
  • Set ssl_ca = /etc/ssl/mysql/ca.pem (in mysqld section)
  • Set ssl_cert = /etc/ssl/mysql/server-cert.pem (in mysqld section)
  • Set ssl_key = /etc/ssl/mysql/server-key.pem (in mysqld section)

Bootstrapped first node:

systemctl start mysql@bootstrap.service

Started mysql on node2 to join the cluster

service mysql start

After node2 successfully synced with the cluster, started mysql on node3 to join the cluster

service mysql start

After node3 successfully synced with the cluster, stopped bootstrapping process on node1

systemctl stop mysql@bootstrap.service

when it’s stopped, start mysql the normal way without bootstrapping

service mysql start

Awesome! Glad you got it working!