Percona xtraDB cluster fails to come up on IPv6

MySQL & MariaDB Percona XtraDB Cluster 5.x
, ,
1

We are using percone-xtradb-cluster version 5.7.36-31.55 running in kubernetes environment configured on IPv6.

I have put this configuration in helm chart while bootstrapping:

 configuration: |
     [mysqld]
     pxc_strict_mode=PERMISSIVE
     bind_address=::
     wsrep_provider_options="gmcast.listen_addr=tcp://[::]:4567;debug=yes;ist.recv_addr=percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4568"
     wsrep_debug=ON
     wsrep_sst_receive_address=percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444
     [sst]
     sockopt=',pf=ipv6'

This is a 3 node cluster with 3 replicas expected. percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc is the first percona pod.
However I see that joiner(2nd pod) fails to do SST from 1st pod with this failure
Joiner side(percona-db-pxc-db-pxc-1) logs:

2022-11-03T17:56:29.069377Z 2 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1
2022-11-03T17:56:29.069444Z 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 1 -> 2)
2022-11-03T17:56:29.069539Z 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-11-03T17:56:29.069644Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '5.7.36-39-57'   '' : 32 (Broken pipe)

Donor side(percona-db-pxc-db-pxc-0) logs:

2022/11/03 17:56:59 socat[9767] E connect(7, AF=10 [fd00:0109:0000:0000:0000:0000:0000:070b]:4444, 28): Connection refused
	2022-11-03T17:56:59.718113Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************
	2022-11-03T17:56:59.721502Z WSREP_SST: [ERROR] Error while sending data to joiner node:  exit codes: 0 1
	2022-11-03T17:56:59.724580Z WSREP_SST: [ERROR] ******************************************************
	2022-11-03T17:56:59.728128Z WSREP_SST: [ERROR] Cleanup after exit with status:32
	2022-11-03T17:56:59.734309Z WSREP_SST: [DEBUG] Cleaning up temporary directories
2022-11-03T17:56:59.749056Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.36-39-57'   '' --gtid 'eb066fe8-5b9c-11ed-ab4d-7a7983fb2636:23' : 32 (Broken pipe)
2022-11-03T17:56:59.749166Z 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.36-39-57'   '' --gtid 'eb066fe8-5b9c-11ed-ab4d-7a7983fb2636:23'
2022-11-03T17:56:59.749191Z 0 [Warning] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():779: Could not find peer: 95cfc76d-5ba0-11ed-b214-d3b63deaca52
2022-11-03T17:56:59.749276Z 0 [Warning] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():786: 0.0 (percona-db-pxc-db-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)

Is there any configuration that I am missing ? Does percona support IPv6 on 5.7 version ? Please advice.

1 Like
2

Hi @Supreeth_Shivanand thanks for posting

Looks like you have a typo in your configuration, it should be:

[sst]
sockopt=“pf=ip6”
1 Like
3

Hello @Michael_Coburn , Thanks for helping me out. I tried setting the configuration as pointed out by you, but I still see that failing with broken pipe.

	2022-11-04T17:30:01.688938Z WSREP_SST: [DEBUG] Evaluating socat -u openssl-listen:4444,reuseaddr,cert=/etc/mysql/ssl-internal/tls.crt,key=/etc/mysql/ssl-internal/tls.key,cafile=/etc/mysql/ssl-internal/ca.crt,verify=1,pf=ip6,retry=30 stdio | pv -f  -i 10 -N joiner -F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p'  | xbstream -x  $xbstream_eopts
	2022-11-04T17:30:01.885470Z WSREP_SST: [DEBUG] 1259: Entering loop body : 2
	2022-11-04T17:30:01.924852Z WSREP_SST: [DEBUG] 1268: Examining pid: 818
2022-11-04T17:30:02.043064Z 0 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:check_liveness():1242: continuing relaying for PT0.514806S
	2022-11-04T17:30:02.094937Z WSREP_SST: [DEBUG] 1285: sockets: 57279931|57279932|57279933
	2022-11-04T17:30:02.111492Z WSREP_SST: [DEBUG] 1325: wait_for_listen() exiting
2022-11-04T17:30:02.112475Z 1 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1
2022-11-04T17:30:02.112559Z 1 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 1 -> 2)
2022-11-04T17:30:02.112942Z 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-11-04T17:30:02.113026Z 1 [Note] WSREP: galera/src/certification.cpp:assign_initial_position():989: Assign initial position for certification: 23, protocol version: 4
2022-11-04T17:30:02.113121Z 0 [Note] WSREP: galera/src/galera_service_thd.cpp:thd_func():46: Service thread queue flushed.
2022-11-04T17:30:02.113295Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_state_request():578: Check if state gap can be serviced using IST
2022-11-04T17:30:02.113354Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_for_IST():515: Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: eb066fe8-5b9c-11ed-ab4d-7a7983fb2636
2022-11-04T17:30:02.113461Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_state_request():584: State gap can't be serviced using IST. Switching to SST
2022-11-04T17:30:02.113484Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_state_request():586: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (eb066fe8-5b9c-11ed-ab4d-7a7983fb2636): 1 (Operation not permitted)
	 at galera/src/replicator_str.cpp:prepare_for_IST():518. IST will be unavailable.
2022-11-04T17:30:02.113520Z 1 [Note] [Debug] WSREP: gcs/src/gcs.cpp:gcs_request_state_transfer():1835: ist_uuid[00000000-0000-0000-0000-000000000000], ist_seqno[-1]
2022-11-04T17:30:02.115384Z 0 [Note] WSREP: gcs/src/gcs_group.cpp:group_select_donor():1389: Member 0.0 (percona-db-pxc-db-pxc-1) requested state transfer from '*any*'. Selected 1.0 (percona-db-pxc-db-pxc-0)(SYNCED) as donor.
2022-11-04T17:30:02.115458Z 0 [Note] WSREP: gcs/src/gcs.cpp:gcs_shift_state():645: Shifting PRIMARY -> JOINER (TO: 23)
2022-11-04T17:30:02.115578Z 1 [Note] WSREP: galera/src/replicator_str.cpp:send_state_request():711: Requesting state transfer: success, donor: 1
2022-11-04T17:30:02.115626Z 1 [Note] WSREP: gcache/src/GCache_seqno.cpp:seqno_reset():38: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> eb066fe8-5b9c-11ed-ab4d-7a7983fb2636:23
2022-11-04T17:30:02.543191Z 0 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:check_liveness():1242: continuing relaying for PT0.0146638S
2022-11-04T17:30:03.043345Z 0 [Note] WSREP: gcomm/src/gmcast.cpp:check_liveness():1293: (4ea2d568, 'ssl://[::]:4567') turning message relay requesting off
2022-11-04T17:30:32.914085Z 0 [Warning] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():786: 1.0 (percona-db-pxc-db-pxc-0): State transfer to 0.0 (percona-db-pxc-db-pxc-1) failed: -32 (Broken pipe)

Any ideas what might be going wrong ?

1 Like
4

Does percona operator and the init scripts support IPv6 ? Because I see it not prefixing [] around the addresses, also percona pxc runs xtrabackup-v2 with half parsed ipv6 addess like below.

Example:
wsrep_sst_xtrabackup-v2 --role ‘joiner’ –address ‘fd00’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘1’ --mysqld-version ‘5.7.36-39-57’ ‘’ : 32 (Broken pipe)

1 Like