Openshift Operator provisioned xtradb cluster won't start

Hi,
i just installed the operator from the redhat marketplace and deployed an instance in my project. Only 1of3 MySQL instances boot up, the 2of3 is allways falling into error state and starts again.

Here is the logfile:

{"log":"2022-01-25T07:28:09.440694Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.128.3.172' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.23-14.1'  --binlog 'binlog' )\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:09.859612Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.172664Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|10.128.3.172:4444/xtrabackup_sst//1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.173185Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.173573Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: d5c055c5-7daf-11ec-b6a2-07f3a8a6a841\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.173945Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 39, STRv: 3\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.174297Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://10.128.3.172:4568\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.174546Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.175319Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-39, listening at: ssl://10.128.3.172:4568\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.176549Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mysql-pxc-1) requested state transfer from '*any*'. Selected 1.0 (mysql-pxc-0)(SYNCED) as donor.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.176902Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 39)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.177281Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 1\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.177715Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.178098Z 1 [Note] [MY-000000] [Galera] GCache history reset: d5c055c5-7daf-11ec-b6a2-07f3a8a6a841:0 -> d5c055c5-7daf-11ec-b6a2-07f3a8a6a841:39\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.707146Z 0 [Warning] [MY-000000] [Galera] 1.0 (mysql-pxc-0): State transfer to 0.0 (mysql-pxc-1) failed: -32 (Broken pipe)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.707969Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receive state. Need to abort.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.708005Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.708034Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:10.708125Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713090Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview (view_id(NON_PRIM,57e9e4e5-8e1e,10)\nmemb {\n\t57e9e4e5-8e1e,0\n\t}\njoined {\n\t}\nleft {\n\t}\npartitioned {\n\te6ef41f8-8b1b,0\n\t}\n)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713246Z 0 [Note] [MY-000000] [Galera] (57e9e4e5-8e1e, 'ssl://0.0.0.0:4567') turning message relay requesting off\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713280Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713303Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node\nview ((empty))\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713601Z 0 [Note] [MY-000000] [Galera] gcomm: closed\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713669Z 0 [Note] [MY-000000] [Galera] mysqld: Terminated.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713691Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.713708Z 0 [Note] [MY-000000] [WSREP] Terminating SST process\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"07:28:11 UTC - mysqld got signal 11 ;\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"Build ID: Not Available \nServer Version: 8.0.23-14.1 Percona XtraDB Cluster (GPL), Release rel14, Revision d3b9a1d, WSREP version 26.4.3, wsrep_26.4.3\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"Thread pointer: 0x0\nAttempting backtrace. You can use the following information to find out\nwhere mysqld died. If you see no messages after this, something went\nterribly wrong...\nstack_bottom = 0 thread_stack 0x46000\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.714523Z 0 [ERROR] [MY-000000] [WSREP-SST] Removing /var/lib/mysql//xtrabackup_galera_info file due to signal\n/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x2162191]\n/usr/sbin/mysqld(handle_fatal_signal+0x3c3) [0x12ded33]\n/lib64/libpthread.so.0(+0x12c20) [0x7ff520fa2c20]\n/lib64/libc.so.6(abort+0x203) [0x7ff51ee6ce91]\n/usr/lib64/galera4/libgalera_smm.so(+0x14a75) [0x7ff512b41a75]\n/usr/lib64/galera4/libgalera_smm.so(+0x17a826) [0x7ff512ca7826]\n/usr/lib64/galera4/libgalera_smm.so(+0x182966) [0x7ff512caf966]\n/lib64/libpthread.so.0(+0x817a) [0x7ff520f9817a]\n/lib64/libc.so.6(clone+0x43) [0x7ff51ef47dc3]\nYou may download the Percona XtraDB Cluster operations manual by visiting\nhttp://www.percona.com/software/percona-xtradb-cluster/. You may find information\nin the manual which will help you identify the cause of the crash.\nWriting a core file using lib coredumper\nPATH: (null)\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:11.717464Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:12.632207Z 0 [ERROR] [MY-000000] [WSREP-SST] SST script interrupted\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2022-01-25T07:28:12.632273Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** \n","file":"/var/lib/mysql/mysqld-error.log"}
1 Like

Hello @snooops ,

we will need to reproduce this issue.
What are the versions of Openshift and Operator that you are trying out?
Anything specific about your Openshift deployment? Like some custom built storage?

1 Like

Openshift Version: 4.8.28
Operator Version: percona-xtradb-cluster-operator-certified-rhmp.v.1.10.0
StorageClasses: standard - Default kubernetes.io/cinder and standard-csi cinder.csi.openstack.org

1 Like