Handshake Failure with certificate validation failed

cmg1986 · January 6, 2022, 9:43am

Hello,

I am using percona-xtradb-cluster:8.0.22-13.1 docker image in my solution with default SSL configuration and certificate generated by percona itself using below conf -

[mysqld]
pxc-encrypt-cluster-traffic=ON
ssl-ca=/etc/mysql/ssl-internal/ca.crt
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt

And thats absolutely working fine.

But due to some security vulnerabilities, updated dockerfile as per below -

github.com

cmg1986/percona-docker/blob/fix_go/percona-xtradb-cluster-8.0/Dockerfile

FROM golang:1.17.5 AS go_builder
WORKDIR /go/src/github.com/percona/percona-xtradb-cluster-operator/src

RUN go get k8s.io/apimachinery/pkg/util/sets; \
    curl -Lfk -o /go/src/github.com/percona/percona-xtradb-cluster-operator/src/peer-list.go https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v1.8.0/cmd/peer-list/main.go; \
	go mod init; go mod tidy; go build peer-list.go

FROM centos:8 AS centos8

LABEL org.opencontainers.image.authors="info@percona.com"
LABEL desc="This is Security Patched modified image with updated Golang and some Packages"

# check repository package signature in secure way
RUN set -ex; \
    export GNUPGHOME="$(mktemp -d)"; \
    gpg --batch --keyserver keyserver.ubuntu.com --recv-keys 430BDF5C56E7C94E848EE60C1C4CBDCDCD2EFD2A 99DB70FAE1D7CE227FB6488205B555B38483C65D 94E279EB8D8F25B21810ADF121EA45AB2F86D6A1; \
    gpg --batch --export --armor 430BDF5C56E7C94E848EE60C1C4CBDCDCD2EFD2A > ${GNUPGHOME}/RPM-GPG-KEY-Percona; \
    gpg --batch --export --armor 99DB70FAE1D7CE227FB6488205B555B38483C65D > ${GNUPGHOME}/RPM-GPG-KEY-centosofficial; \
    gpg --batch --export --armor 94E279EB8D8F25B21810ADF121EA45AB2F86D6A1 > ${GNUPGHOME}/RPM-GPG-KEY-EPEL-8; \
    rpmkeys --import ${GNUPGHOME}/RPM-GPG-KEY-Percona ${GNUPGHOME}/RPM-GPG-KEY-EPEL-8 ${GNUPGHOME}/RPM-GPG-KEY-centosofficial; \

This file has been truncated. show original

But after that when i am trying to create Node, pxc-0 is coming up but unable to take up pxc-1 and unable to join cluster. During POD logs, i found below issue while using same certs and configuration.

I am not sure whether its dependent on openssl package version as in this Dockerfile, its installing latest available version of openssl package which is different from earlier.

In Original Docker image → openssl : openssl-1.1.1g-12.el8_3.src.rpm
In modified Docker image → openssl : openssl-1.1.1k-5.el8_5.src.rpm

I am not so sure whether openssl version is something which is unable to verify certificates ?

*2022-01-06T08:20:19.156450Z 0 [Note] [MY-000000] [Galera] GMCast version 0*
*2022-01-06T08:20:19.158212Z 0 [Note] [MY-000000] [Galera] (7c1f7c75-b70b, 'ssl://[::]:4567') listening at ssl://[::]:4567*
*2022-01-06T08:20:19.158383Z 0 [Note] [MY-000000] [Galera] (7c1f7c75-b70b, 'ssl://[::]:4567') multicast: , ttl: 1*
*2022-01-06T08:20:19.158912Z 0 [Note] [MY-000000] [Galera] EVS version 1*
*2022-01-06T08:20:19.159182Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'mysql-0810-pxc-db-pxc', peer 'mysql-0810-pxc-db-pxc-0.mysql-0810-pxc-db-pxc:'*
*2022-01-06T08:20:19.164471Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://[240b:c0e0:202:55d8:b484:2:0:4f60]:4567 failed: asio.ssl:337047686: 'certificate verify failed' ( 337047686: 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed')*
*This error is often caused by SSL issues. For more information, please see:*
*  https://per.co.na/pxc/encrypt_cluster_traffic*

Any help appreciated please !!

Thanks
Chandra

matthewb · January 6, 2022, 3:07pm

Modify your dockerfile to use the original openssl version and re-test.

cmg1986 · January 7, 2022, 9:58am

Thanks @matthewb , I changed docker file like below -

FROM golang:1.17.5 AS go_builder
WORKDIR /go/src/github.com/percona/percona-xtradb-cluster-operator/src

RUN go get k8s.io/apimachinery/pkg/util/sets; \
    curl -Lfk -o /go/src/github.com/percona/percona-xtradb-cluster-operator/src/peer-list.go https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v1.8.0/cmd/peer-list/main.go; \
	go mod init; go mod tidy; go build peer-list.go

FROM centos:8 AS centos8

LABEL org.opencontainers.image.authors="info@percona.com"

# check repository package signature in secure way
RUN set -ex; \
    export GNUPGHOME="$(mktemp -d)"; \
    gpg --batch --keyserver keyserver.ubuntu.com --recv-keys 430BDF5C56E7C94E848EE60C1C4CBDCDCD2EFD2A 99DB70FAE1D7CE227FB6488205B555B38483C65D 94E279EB8D8F25B21810ADF121EA45AB2F86D6A1; \
    gpg --batch --export --armor 430BDF5C56E7C94E848EE60C1C4CBDCDCD2EFD2A > ${GNUPGHOME}/RPM-GPG-KEY-Percona; \
    gpg --batch --export --armor 99DB70FAE1D7CE227FB6488205B555B38483C65D > ${GNUPGHOME}/RPM-GPG-KEY-centosofficial; \
    gpg --batch --export --armor 94E279EB8D8F25B21810ADF121EA45AB2F86D6A1 > ${GNUPGHOME}/RPM-GPG-KEY-EPEL-8; \
    rpmkeys --import ${GNUPGHOME}/RPM-GPG-KEY-Percona ${GNUPGHOME}/RPM-GPG-KEY-EPEL-8 ${GNUPGHOME}/RPM-GPG-KEY-centosofficial; \
    curl -Lf -o /tmp/percona-release.rpm https://repo.percona.com/yum/percona-release-1.0-26.noarch.rpm; \
    rpmkeys --checksig /tmp/percona-release.rpm; \
    dnf install -y /tmp/percona-release.rpm; \
    rm -rf "$GNUPGHOME" /tmp/percona-release.rpm; \
    rpm --import /etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY; \
    dnf -y module disable mysql

RUN set -ex; \
	curl -Lf -o /tmp/pv.rpm https://download.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/pv-1.6.6-7.el8.x86_64.rpm; \
	rpmkeys --checksig /tmp/pv.rpm; \
	rpm -i /tmp/pv.rpm; \
	rm -rf /tmp/pv.rpm

ENV PXC_VERSION 8.0.22-13.1
ENV PXC_REPO release
ENV OS_VER el8
ENV FULL_PERCONA_XTRADBCLUSTER_VERSION "$PXC_VERSION.$OS_VER"

COPY openssl-1.1.1g-15.el8_3.x86_64.rpm ./
RUN set -ex; \
	percona-release setup -y ps-80; \
	rpm -e --nodeps tzdata; \
	dnf install -y \
		jemalloc \
		socat \
		# openssl \
		shadow-utils \
		hostname \
		# curl \
		tzdata \
		diffutils \
		libaio \
		which \
		pam \
		nss-pam-ldapd \
		numactl-libs \
		libev \
		jq \
		oniguruma \
		cracklib-dicts \
		dnf-utils; \
	dnf localinstall -y openssl-1.1.1g-15.el8_3.x86_64.rpm; \	
	dnf update -y \
		glib2 \
		dhcp-libs \
		bind-export-libs \
		nettle \
		gnutls; \
	dnf clean all; \
	rm -rf /var/cache/dnf; \
	rm -rf openssl-1.1.1g-15.el8_3.x86_64.rpm

# create mysql user/group before mysql installation
RUN groupadd -g 1001 mysql; \
	useradd -u 1001 -r -g 1001 -s /sbin/nologin \
		-c "Default Application User" mysql

# we need licenses from docs
RUN set -ex; \
	percona-release disable all; \
	percona-release enable pxc-80 ${PXC_REPO}; \
	repoquery -a --location \
		Percona-XtraDB-Cluster-server-${FULL_PERCONA_XTRADBCLUSTER_VERSION} \
		| xargs curl -Lf -o /tmp/Percona-XtraDB-Cluster-server.rpm; \
	repoquery -a --location \
		Percona-XtraDB-Cluster-shared-${FULL_PERCONA_XTRADBCLUSTER_VERSION} \
		| xargs curl -Lf -o /tmp/Percona-XtraDB-Cluster-shared.rpm; \
	repoquery -a --location \
		Percona-XtraDB-Cluster-shared-compat-${FULL_PERCONA_XTRADBCLUSTER_VERSION} \
		| xargs curl -Lf -o /tmp/Percona-XtraDB-Cluster-shared-compat.rpm; \
	repoquery -a --location \
		Percona-XtraDB-Cluster-client-${FULL_PERCONA_XTRADBCLUSTER_VERSION} \
		| xargs curl -Lf -o /tmp/Percona-XtraDB-Cluster-client.rpm; \
	rpm -iv /tmp/Percona-XtraDB-Cluster-server.rpm /tmp/Percona-XtraDB-Cluster-shared.rpm /tmp/Percona-XtraDB-Cluster-shared-compat.rpm /tmp/Percona-XtraDB-Cluster-client.rpm --nodeps; \
	rm -rf /tmp/Percona-XtraDB-Cluster-server.rpm /tmp/Percona-XtraDB-Cluster-shared.rpm /tmp/Percona-XtraDB-Cluster-shared-compat.rpm /tmp/Percona-XtraDB-Cluster-client.rpm; \
	dnf clean all; \
	rm -rf /usr/bin/mysqltest /usr/bin/perror /usr/bin/replace /usr/bin/resolve_stack_dump /usr/bin/resolveip; \
	rm -rf /var/cache/dnf /var/lib/mysql /usr/lib64/mysql/plugin/debug /usr/sbin/mysqld-debug /usr/lib64/mecab /usr/lib64/mysql/mecab /usr/bin/myisam*; \
	rpm -ql percona-xtradb-cluster-client | egrep -v "mysql$|mysqldump$|mysqladmin$" | xargs rm -rf;

COPY LICENSE /licenses/LICENSE.Dockerfile
RUN cp /usr/share/doc/percona-xtradb-cluster-galera/COPYING /licenses/LICENSE.galera; \
    cp /usr/share/doc/percona-xtradb-cluster-galera/LICENSE.* /licenses/

RUN set -ex; \
    rmdir /etc/my.cnf.d; \
    ln -s /etc/mysql/conf.d /etc/my.cnf.d; \
    rm -f /etc/percona-xtradb-cluster.conf.d/*.cnf; \
    echo '!include /etc/mysql/node.cnf' > /etc/my.cnf; \
    echo '!includedir /etc/my.cnf.d/' >> /etc/my.cnf; \
    echo '!includedir /etc/percona-xtradb-cluster.conf.d/' >> /etc/my.cnf

COPY dockerdir /
COPY --from=go_builder /go/src/github.com/percona/percona-xtradb-cluster-operator/src/peer-list /usr/bin/
RUN	mkdir -p /etc/mysql/conf.d/ /var/log/mysql /var/lib/mysql /docker-entrypoint-initdb.d /etc/percona-xtradb-cluster.conf.d; \
	chown -R 1001:1001 /etc/mysql/ /var/log/mysql /var/lib/mysql /docker-entrypoint-initdb.d /etc/percona-xtradb-cluster.conf.d; \
	chmod -R g=u /etc/mysql/ /var/log/mysql /var/lib/mysql /docker-entrypoint-initdb.d /etc/percona-xtradb-cluster.conf.d

ARG DEBUG
RUN if [[ -n $DEBUG ]] ; then \
                set -ex; \
		sed -i '/\[mysqld\]/a wsrep_log_conflicts\nlog_error_verbosity=3\nwsrep_debug=1' /etc/mysql/node.cnf; \
		mv /usr/sbin/mysqld /usr/sbin/mysqld-ps; \
		cp /usr/local/bin/mysqld-debug /usr/sbin/mysqld; \
		repoquery -a --location \
			Percona-XtraDB-Cluster-debuginfo-${FULL_PERCONA_XTRADBCLUSTER_VERSION} \
			| xargs curl -Lf -o /tmp/Percona-XtraDB-Cluster-debuginfo.rpm; \
		repoquery -a --location \
			Percona-XtraDB-Cluster-server-debuginfo-${FULL_PERCONA_XTRADBCLUSTER_VERSION} \
			| xargs curl -Lf -o /tmp/Percona-XtraDB-Cluster-server-debuginfo.rpm; \
		rpmkeys --checksig /tmp/Percona-XtraDB-Cluster-debuginfo.rpm /tmp/Percona-XtraDB-Cluster-server-debuginfo.rpm; \
		rpm -iv /tmp/Percona-XtraDB-Cluster-debuginfo.rpm /tmp/Percona-XtraDB-Cluster-server-debuginfo.rpm --nodeps; \
		rm -rf /tmp/Percona-XtraDB-Cluster-debuginfo.rpm /tmp/Percona-XtraDB-Cluster-server-debuginfo.rpm; \
		percona-release enable pdpxc-8.0 release; \
		dnf install -y \
			net-tools \
			telnet \
			nc \
			gdb \
			perf \
			strace \
			tcpdump \
			percona-toolkit; \
		percona-release disable pdpxc-8.0 release; \
		dnf clean all; \
		rm -rf /var/cache/dnf; \
	fi

USER 1001

VOLUME ["/var/lib/mysql", "/var/log/mysql"]

ENTRYPOINT ["/entrypoint.sh"]

EXPOSE 3306 4567 4568 33060
CMD ["mysqld"]

After using image created from above dockerfile, that SSl issue got resolved but still PXC-1 is unable to join cluster with below error -

2022-01-07T09:54:59.305701Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-07T09:54:59.310632Z 10 [Note] [MY-000000] [Galera] Recording CC from group: 61
2022-01-07T09:54:59.310701Z 10 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 8
2022-01-07T09:54:59.310743Z 10 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-01-07T09:55:00.699506Z 0 [Note] [MY-000000] [Galera] Member 1.0 (mysql-0999-pxc-db-pxc-1) requested state transfer from '*any*'. Selected 0.0 (mysql-0999-pxc-db-pxc-0)(SYNCED) as donor.
2022-01-07T09:55:00.699565Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> DONOR/DESYNCED (TO: 61)
2022-01-07T09:55:00.699655Z 10 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 175, req: STRv1
2022-01-07T09:55:00.699745Z 10 [Note] [MY-000000] [Galera] Cert index preload: 8 -> 61
2022-01-07T09:55:00.700724Z 10 [Note] [MY-000000] [Galera] IST sender using ssl
2022-01-07T09:55:00.713124Z 10 [Note] [MY-000000] [WSREP] Server status change synced -> donor
2022-01-07T09:55:00.713218Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-07T09:55:00.713299Z 0 [Note] [MY-000000] [Galera] async IST sender starting to serve ssl://[240b:c0e0:202:55d8:b484:2:0:402c]:4568 sending 8-61, preload starts from 8
2022-01-07T09:55:00.713859Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '[240b:c0e0:202:55d8:b484:2:0:402c]:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.22-13.1'  --binlog 'binlog' --gtid 'a64c3cfa-6f91-11ec-a69a-a25d60a8949c:61' )
2022-01-07T09:55:00.714751Z 0 [Note] [MY-000000] [Galera] IST sender 8 -> 61
2022-01-07T09:55:00.722176Z 10 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2022-01-07T09:55:01.683221Z 0 [Note] [MY-000000] [Galera] (bcee6db8-8a1d, 'ssl://[::]:4567') turning message relay requesting off
2022-01-07T09:55:01.771272Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/07 09:55:01 socat[20108] E Failed to set SNI host ""
2022-01-07T09:55:01.772950Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-01-07T09:55:01.773036Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node:  exit codes: 0 1
2022-01-07T09:55:01.773103Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296
2022-01-07T09:55:01.773171Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
2022-01-07T09:55:01.773386Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
2022-01-07T09:55:01.798505Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '[240b:c0e0:202:55d8:b484:2:0:402c]:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.22-13.1'  --binlog 'binlog' --gtid 'a64c3cfa-6f91-11ec-a69a-a25d60a8949c:61' : 32 (Broken pipe)
2022-01-07T09:55:01.800720Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32
2022-01-07T09:55:01.800767Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined
2022-01-07T09:55:01.800804Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-07T09:55:01.800898Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '[240b:c0e0:202:55d8:b484:2:0:402c]:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.22-13.1'  --binlog 'binlog' --gtid 'a64c3cfa-6f91-11ec-a69a-a25d60a8949c:61'
2022-01-07T09:55:01.801271Z 0 [Warning] [MY-000000] [Galera] 0.0 (mysql-0999-pxc-db-pxc-0): State transfer to 1.0 (mysql-0999-pxc-db-pxc-1) failed: -32 (Broken pipe)
2022-01-07T09:55:01.801304Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 61)
2022-01-07T09:55:01.801717Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mysql-0999-pxc-db-pxc-0) synced with group.
2022-01-07T09:55:01.801741Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 61)
2022-01-07T09:55:01.801852Z 10 [Note] [MY-000000] [Galera] Server mysql-0999-pxc-db-pxc-0 synced with group
2022-01-07T09:55:01.801912Z 10 [Note] [MY-000000] [WSREP] Server status change joined -> synced
2022-01-07T09:55:01.801932Z 10 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2022-01-07T09:55:01.801955Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-07T09:55:02.804881Z 0 [Note] [MY-000000] [Galera] forgetting df8fe1e5-9f10 (ssl://[240b:c0e0:202:55d8:b484:2:0:402c]:4567)
2022-01-07T09:55:02.804978Z 0 [Note] [MY-000000] [Galera] Node bcee6db8-8a1d state primary
2022-01-07T09:55:02.805040Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,bcee6db8-8a1d,49)
memb {
	bcee6db8-8a1d,0
	}
joined {
	}
left {
	}
partitioned {
	df8fe1e5-9f10,0
	}
)
2022-01-07T09:55:02.805078Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2022-01-07T09:55:02.807804Z 0 [Note] [MY-000000] [Galera] forgetting df8fe1e5-9f10 (ssl://[240b:c0e0:202:55d8:b484:2:0:402c]:4567)
2022-01-07T09:55:02.807867Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2022-01-07T09:55:02.808097Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: e241a66a-6f9f-11ec-8354-8fa1c0981d53
2022-01-07T09:55:02.808177Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: e241a66a-6f9f-11ec-8354-8fa1c0981d53
2022-01-07T09:55:02.808221Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: e241a66a-6f9f-11ec-8354-8fa1c0981d53 from 0 (mysql-0999-pxc-db-pxc-0)
2022-01-07T09:55:02.808249Z 0 [Note] [MY-000000] [Galera] Quorum results:
	version    = 6,
	component  = PRIMARY,
	conf_id    = 48,
	members    = 1/1 (primary/total),
	act_id     = 61,
	last_appl. = 6,
	protocols  = 2/10/4 (gcs/repl/appl),
	vote policy= 0,
	group UUID = a64c3cfa-6f91-11ec-a69a-a25d60a8949c
2022-01-07T09:55:02.808332Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2022-01-07T09:55:02.808606Z 2 [Note] [MY-000000] [Galera] ####### processing CC 62, local, ordered
2022-01-07T09:55:02.808683Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 61 upto current CC event 62 upto:61
2022-01-07T09:55:02.808705Z 2 [Note] [MY-000000] [Galera] Drain monitors from 61 up to 61
2022-01-07T09:55:02.808732Z 2 [Note] [MY-000000] [Galera] ####### My UUID: bcee6db8-6f91-11ec-8a1d-c7a5a5eb914e
2022-01-07T09:55:02.808753Z 2 [Note] [MY-000000] [Galera] Skipping cert index reset
2022-01-07T09:55:02.808771Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2022-01-07T09:55:02.808792Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 61 -> 62
2022-01-07T09:55:02.808829Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2022-01-07T09:55:02.813986Z 2 [Note] [MY-000000] [Galera] ================================================
View:
  id: a64c3cfa-6f91-11ec-a69a-a25d60a8949c:62
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
	0: bcee6db8-6f91-11ec-8a1d-c7a5a5eb914e, mysql-0999-pxc-db-pxc-0

2022-01-07T09:54:59.306871Z 1 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2022-01-07T09:54:59.307467Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-01-07T09:54:59.308525Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '[240b:c0e0:202:55d8:b484:2:0:402c]' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.22-13.1'  --binlog 'binlog' )
2022-01-07T09:55:00.222061Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2022-01-07T09:55:00.692964Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|[240b:c0e0:202:55d8:b484:2:0:402c]:4444/xtrabackup_sst//1
2022-01-07T09:55:00.693797Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-01-07T09:55:00.694457Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: a64c3cfa-6f91-11ec-a69a-a25d60a8949c
2022-01-07T09:55:00.695077Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 61, STRv: 3
2022-01-07T09:55:00.695861Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://[240b:c0e0:202:55d8:b484:2:0:402c]:4568
2022-01-07T09:55:00.696632Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl
2022-01-07T09:55:00.698006Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-61, listening at: ssl://[240b:c0e0:202:55d8:b484:2:0:402c]:4568
2022-01-07T09:55:00.699453Z 0 [Note] [MY-000000] [Galera] Member 1.0 (mysql-0999-pxc-db-pxc-1) requested state transfer from '*any*'. Selected 0.0 (mysql-0999-pxc-db-pxc-0)(SYNCED) as donor.
2022-01-07T09:55:00.700289Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 61)
2022-01-07T09:55:00.701050Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2022-01-07T09:55:00.701837Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-01-07T09:55:00.702570Z 1 [Note] [MY-000000] [Galera] GCache history reset: a64c3cfa-6f91-11ec-a69a-a25d60a8949c:0 -> a64c3cfa-6f91-11ec-a69a-a25d60a8949c:61
2022-01-07T09:55:01.792943Z 0 [Note] [MY-000000] [Galera] (df8fe1e5-9f10, 'ssl://[::]:4567') turning message relay requesting off
2022-01-07T09:55:01.801662Z 0 [Warning] [MY-000000] [Galera] 0.0 (mysql-0999-pxc-db-pxc-0): State transfer to 1.0 (mysql-0999-pxc-db-pxc-1) failed: -32 (Broken pipe)
2022-01-07T09:55:01.804166Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receive state. Need to abort.
2022-01-07T09:55:01.804238Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2022-01-07T09:55:01.804295Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2022-01-07T09:55:01.804398Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2022-01-07T09:55:02.808495Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,bcee6db8-8a1d,48)
memb {
	df8fe1e5-9f10,0
	}
joined {
	}
left {
	}
partitioned {
	bcee6db8-8a1d,0
	}
)
2022-01-07T09:55:02.808584Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2022-01-07T09:55:02.808607Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2022-01-07T09:55:02.808817Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2022-01-07T09:55:02.808852Z 0 [Note] [MY-000000] [Galera] mysqld: Terminated.
2022-01-07T09:55:02.808872Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2022-01-07T09:55:02.808886Z 0 [Note] [MY-000000] [WSREP] Terminating SST process
09:55:02 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: 23c893872480ac27eda9bb248bcd29e2ceafa7b4
Server Version: 8.0.22-13.1 Percona XtraDB Cluster (GPL), Release rel13, Revision a48e6d5, WSREP version 26.4.3, wsrep_26.4.3

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
2022-01-07T09:55:02.811845Z 0 [ERROR] [MY-000000] [WSREP-SST] Removing /var/lib/mysql//xtrabackup_galera_info file due to signal
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x210e8d1]
/usr/sbin/mysqld(handle_fatal_signal+0x3c3) [0x12c47f3]
/lib64/libpthread.so.0(+0x12b20) [0x7f112c994b20]
/lib64/libc.so.6(abort+0x203) [0x7f112a861e91]
/usr/lib64/galera4/libgalera_smm.so(+0x143e5) [0x7f111e7243e5]
/usr/lib64/galera4/libgalera_smm.so(+0x179d66) [0x7f111e889d66]
/usr/lib64/galera4/libgalera_smm.so(+0x181e6e) [0x7f111e891e6e]
/lib64/libpthread.so.0(+0x814a) [0x7f112c98a14a]
/lib64/libc.so.6(clone+0x43) [0x7f112a93cdc3]
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file using lib coredumper
PATH: (null)

2022-01-07T09:55:03.467737Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-01-07T09:55:03.467829Z 0 [ERROR] [MY-000000] [WSREP-SST] SST script interrupted
2022-01-07T09:55:03.467874Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '[240b:c0e0:202:55d8:b484:2:0:402c]' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.22-13.1'  --binlog 'binlog' : 32 (Broken pipe)
2022-01-07T09:55:03.467966Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
Error writting coredump: -1 Signal: 11

Any insight pls ?

matthewb · January 7, 2022, 3:02pm

2022-01-07T09:55:01.801662Z 0 [Warning] [MY-000000] [Galera] 0.0 (mysql-0999-pxc-db-pxc-0): State transfer to 1.0 (mysql-0999-pxc-db-pxc-1) failed: -32 (Broken pipe)

Perhaps networking issues? You need to have prots 4444, 4567, and 4568 open in all directions.

cmg1986 · January 7, 2022, 3:25pm

@matthewb I suppose PORTs are already open because when i use original image percona-xtradb-cluster:8.0.22-13.1 it went absolutely fine.

So not sure why issue is happening with image which i created from my modified Dockerfile.

Thanks !!

cmg1986 · January 8, 2022, 2:30pm

Any insight pls on mentioned error ?

matthewb · January 8, 2022, 2:55pm

@cmg1986 Not sure. I’m not a Docker expert. This is why we always recommend using our official images. Our images go through a large battery of tests and integration checks to ensure everything works as-is.

A_Niu · May 10, 2022, 3:13pm

The problem is still outstanding with in new image with tag 8.0.27-18.1. Is it planned to fix it in soon future? I prefer to being fixed from Percona side, so it will be easier in the future upgrade.

My environment: OCI k8s
Deployed percona-helm-chart with Terraform: percona-helm-charts/charts at main · percona/percona-helm-charts · GitHub
xtradb.log (6.0 KB)

Topic		Replies	Views
Running Percona XtraDB Cluster in a Docker Container 8.0 Percona XtraDB Cluster 8.x	1	615	September 26, 2023
SSL connection error : certificate verify failed, [Galera] Handshake failed Percona XtraDB Cluster 8.x	6	1694	August 12, 2022
Percona XtraDB Cluster 8.0 CentOS 8 Percona XtraDB Cluster 5.x	2	1793	June 30, 2023
Percona XtraDB MySQL Backup is failing due to SSL certificate expired \| Please let me know the process to renew the SSL certs Percona Operator for MySQL	2	262	April 12, 2024
Kubernetes - [Galera] Handshake failed due to unknown CA Percona Server for MySQL 8.0 kubernetes	3	834	August 19, 2024

Handshake Failure with certificate validation failed

Related topics