# Percona xtraDB cluster fails to come up on IPv6 **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Tags:** community, mysql, percona\ **Created:** [November 3, 2022, 6:09pm UTC](https://forums.percona.com/t/percona-xtradb-cluster-fails-to-come-up-on-ipv6/18365 "2022-11-03T18:09:15Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Supreeth\_Shivanand](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/supreeth_shivanand/32/7765_2.png) [@Supreeth\_Shivanand](https://forums.percona.com/u/Supreeth_Shivanand)\ **Post date:** [November 3, 2022, 6:09pm UTC](https://forums.percona.com/t/percona-xtradb-cluster-fails-to-come-up-on-ipv6/18365/1 "2022-11-03T18:09:15Z")
We are using percone-xtradb-cluster version 5.7.36-31.55 running in kubernetes environment configured on IPv6. I have put this configuration in helm chart while bootstrapping: ```auto configuration: | [mysqld] pxc_strict_mode=PERMISSIVE bind_address=:: wsrep_provider_options="gmcast.listen_addr=tcp://[::]:4567;debug=yes;ist.recv_addr=percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4568" wsrep_debug=ON wsrep_sst_receive_address=percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444 [sst] sockopt=',pf=ipv6' ``` This is a 3 node cluster with 3 replicas expected. percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc is the first percona pod. However I see that joiner(2nd pod) fails to do SST from 1st pod with this failure Joiner side(percona-db-pxc-db-pxc-1) logs: ```auto 2022-11-03T17:56:29.069377Z 2 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1 2022-11-03T17:56:29.069444Z 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 1 -> 2) 2022-11-03T17:56:29.069539Z 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2022-11-03T17:56:29.069644Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '5.7.36-39-57' '' : 32 (Broken pipe) ``` Donor side(percona-db-pxc-db-pxc-0) logs: ```auto 2022/11/03 17:56:59 socat[9767] E connect(7, AF=10 [fd00:0109:0000:0000:0000:0000:0000:070b]:4444, 28): Connection refused 2022-11-03T17:56:59.718113Z WSREP_SST: [ERROR] *******************FATAL ERROR********************** 2022-11-03T17:56:59.721502Z WSREP_SST: [ERROR] Error while sending data to joiner node: exit codes: 0 1 2022-11-03T17:56:59.724580Z WSREP_SST: [ERROR] ****************************************************** 2022-11-03T17:56:59.728128Z WSREP_SST: [ERROR] Cleanup after exit with status:32 2022-11-03T17:56:59.734309Z WSREP_SST: [DEBUG] Cleaning up temporary directories 2022-11-03T17:56:59.749056Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.36-39-57' '' --gtid 'eb066fe8-5b9c-11ed-ab4d-7a7983fb2636:23' : 32 (Broken pipe) 2022-11-03T17:56:59.749166Z 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address 'percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.36-39-57' '' --gtid 'eb066fe8-5b9c-11ed-ab4d-7a7983fb2636:23' 2022-11-03T17:56:59.749191Z 0 [Warning] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():779: Could not find peer: 95cfc76d-5ba0-11ed-b214-d3b63deaca52 2022-11-03T17:56:59.749276Z 0 [Warning] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():786: 0.0 (percona-db-pxc-db-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe) ``` Is there any configuration that I am missing ? Does percona support IPv6 on 5.7 version ? Please advice. ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [November 4, 2022, 3:18pm UTC](https://forums.percona.com/t/percona-xtradb-cluster-fails-to-come-up-on-ipv6/18365/2 "2022-11-04T15:18:23Z")
Hi @Supreeth_Shivanand thanks for posting > [@Supreeth\_Shivanand](#): > > ```auto > [sst] > sockopt=',pf=ipv6' > > ``` Looks like you have a typo in your configuration, it should be: ```auto [sst] sockopt=“pf=ip6” ``` > [@SST not working with xtrabackup-v2, only with rsync. IPv6 issue ?](https://forums.percona.com/t/sst-not-working-with-xtrabackup-v2-only-with-rsync-ipv6-issue/6138/4): > > Hello. I have found two solutions and I’m using the 2nd. 1st solution - (not good, need edit the script): edit /usr/bin/wsrep\_sst\_xtrabackup-v2, line 532: replace tcmd=“socat -u TCP-LISTEN:${TSST\_PORT},reuseaddr${sockopt} stdio” to tcmd=“socat -u TCP6-LISTEN:${TSST\_PORT},reuseaddr${sockopt} stdio” restart mysql and sst will work using xtrabackup-v2. 2nd solution: (good, edit the config file): before any changes, check joiner log: 2018-01-25T15:23:38.623734Z WSREP\_SST: [DEBUG] The xtra… ---
**Author:** ![Supreeth\_Shivanand](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/supreeth_shivanand/32/7765_2.png) [@Supreeth\_Shivanand](https://forums.percona.com/u/Supreeth_Shivanand)\ **Post date:** [November 4, 2022, 5:34pm UTC](https://forums.percona.com/t/percona-xtradb-cluster-fails-to-come-up-on-ipv6/18365/3 "2022-11-04T17:34:26Z")
Hello @Michael_Coburn , Thanks for helping me out. I tried setting the configuration as pointed out by you, but I still see that failing with broken pipe. ```auto 2022-11-04T17:30:01.688938Z WSREP_SST: [DEBUG] Evaluating socat -u openssl-listen:4444,reuseaddr,cert=/etc/mysql/ssl-internal/tls.crt,key=/etc/mysql/ssl-internal/tls.key,cafile=/etc/mysql/ssl-internal/ca.crt,verify=1,pf=ip6,retry=30 stdio | pv -f -i 10 -N joiner -F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p' | xbstream -x $xbstream_eopts 2022-11-04T17:30:01.885470Z WSREP_SST: [DEBUG] 1259: Entering loop body : 2 2022-11-04T17:30:01.924852Z WSREP_SST: [DEBUG] 1268: Examining pid: 818 2022-11-04T17:30:02.043064Z 0 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:check_liveness():1242: continuing relaying for PT0.514806S 2022-11-04T17:30:02.094937Z WSREP_SST: [DEBUG] 1285: sockets: 57279931|57279932|57279933 2022-11-04T17:30:02.111492Z WSREP_SST: [DEBUG] 1325: wait_for_listen() exiting 2022-11-04T17:30:02.112475Z 1 [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|percona-db-pxc-db-pxc-0.percona-db-pxc-db-pxc:4444/xtrabackup_sst//1 2022-11-04T17:30:02.112559Z 1 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 1) (Increment: 1 -> 2) 2022-11-04T17:30:02.112942Z 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2022-11-04T17:30:02.113026Z 1 [Note] WSREP: galera/src/certification.cpp:assign_initial_position():989: Assign initial position for certification: 23, protocol version: 4 2022-11-04T17:30:02.113121Z 0 [Note] WSREP: galera/src/galera_service_thd.cpp:thd_func():46: Service thread queue flushed. 2022-11-04T17:30:02.113295Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_state_request():578: Check if state gap can be serviced using IST 2022-11-04T17:30:02.113354Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_for_IST():515: Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: eb066fe8-5b9c-11ed-ab4d-7a7983fb2636 2022-11-04T17:30:02.113461Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_state_request():584: State gap can't be serviced using IST. Switching to SST 2022-11-04T17:30:02.113484Z 1 [Note] WSREP: galera/src/replicator_str.cpp:prepare_state_request():586: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (eb066fe8-5b9c-11ed-ab4d-7a7983fb2636): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():518. IST will be unavailable. 2022-11-04T17:30:02.113520Z 1 [Note] [Debug] WSREP: gcs/src/gcs.cpp:gcs_request_state_transfer():1835: ist_uuid[00000000-0000-0000-0000-000000000000], ist_seqno[-1] 2022-11-04T17:30:02.115384Z 0 [Note] WSREP: gcs/src/gcs_group.cpp:group_select_donor():1389: Member 0.0 (percona-db-pxc-db-pxc-1) requested state transfer from '*any*'. Selected 1.0 (percona-db-pxc-db-pxc-0)(SYNCED) as donor. 2022-11-04T17:30:02.115458Z 0 [Note] WSREP: gcs/src/gcs.cpp:gcs_shift_state():645: Shifting PRIMARY -> JOINER (TO: 23) 2022-11-04T17:30:02.115578Z 1 [Note] WSREP: galera/src/replicator_str.cpp:send_state_request():711: Requesting state transfer: success, donor: 1 2022-11-04T17:30:02.115626Z 1 [Note] WSREP: gcache/src/GCache_seqno.cpp:seqno_reset():38: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> eb066fe8-5b9c-11ed-ab4d-7a7983fb2636:23 2022-11-04T17:30:02.543191Z 0 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:check_liveness():1242: continuing relaying for PT0.0146638S 2022-11-04T17:30:03.043345Z 0 [Note] WSREP: gcomm/src/gmcast.cpp:check_liveness():1293: (4ea2d568, 'ssl://[::]:4567') turning message relay requesting off 2022-11-04T17:30:32.914085Z 0 [Warning] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():786: 1.0 (percona-db-pxc-db-pxc-0): State transfer to 0.0 (percona-db-pxc-db-pxc-1) failed: -32 (Broken pipe) ``` Any ideas what might be going wrong ? ---
**Author:** ![Supreeth\_Shivanand](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/supreeth_shivanand/32/7765_2.png) [@Supreeth\_Shivanand](https://forums.percona.com/u/Supreeth_Shivanand)\ **Post date:** [November 7, 2022, 7:02pm UTC](https://forums.percona.com/t/percona-xtradb-cluster-fails-to-come-up-on-ipv6/18365/4 "2022-11-07T19:02:35Z")
Does percona operator and the init scripts support IPv6 ? Because I see it not prefixing `[]` around the addresses, also percona pxc runs xtrabackup-v2 with half parsed ipv6 addess like below. Example: wsrep\_sst\_xtrabackup-v2 --role ‘joiner’ **–address ‘fd00’** --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘1’ --mysqld-version ‘5.7.36-39-57’ ‘’ : 32 (Broken pipe)