Percona XtraDB cluster cross-site replication

Hello,

I am testing Percona XtraDB cluster cross-site replication. I’m following documentation: Set up Percona XtraDB Cluster cross-site replication.

There are two XtraDB clusters involved to this testing: xtradb001-local in pxc-v-1-9-0-local namespace and xtradb-poc-env3 in xtradb-poc-env3 namespace. So both are running within the same Kubernetes cluster. We are using MetalLB for load balancing.

I have configured “xtradb001-local” to be source:
replicationChannels:
- name: pxc1_to_pxc2
isSource: true

And “xtradb-poc-env3” to be the destination for replication:
- name: pxclocal_to_pxcenv3
isSource: false
sourcesList:
- host: 10.183.176.239 #MetalLB external IP address
port: 3306
weight: 100

secrets.yaml contains the default password for “replication” user. This is used on both XtraDB clusters.

This is situation for now:
[root@dbaasjump002 deploy]# kubectl get pods --selector percona.com/replicationPod=true
NAME READY STATUS RESTARTS AGE
xtradb-poc-env3-pxc-0 4/4 Running 1 3h

[root@dbaasjump002 deploy]# kubectl get pods
NAME READY STATUS RESTARTS AGE
centos-mysql 1/1 Running 0 12d
percona-xtradb-cluster-operator-7c4786cb9d-d4cq5 1/1 Running 0 32d
xb-cron-xtradb-poc-env3-fs-pvc-2021928000-372f8-4clch 0/1 Completed 0 10h
xtradb-poc-env3-haproxy-0 2/3 CrashLoopBackOff 40 3h
xtradb-poc-env3-haproxy-1 2/3 Running 40 179m
xtradb-poc-env3-pxc-0 4/4 Running 1 3h
xtradb-poc-env3-pxc-1 3/4 CrashLoopBackOff 33 179m

[root@dbaasjump002 deploy]# kubectl exec -c pxc --stdin --tty xtradb-poc-env3-pxc-0 – /bin/bash
bash-4.4$ mysql -uroot -proot_password
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)

How it would be possible to troubleshoot the issue? I have tested from xtradb-poc-env3-pxc-0 Pod that the connection works to source cluster just fine with MySQL.

bash-4.4$ mysql -h 10.183.176.239 -uroot -proot_password -e ‘SHOW DATABASES;’
mysql: [Warning] Using a password on the command line interface can be insecure.
±-------------------+
| Database |
±-------------------+
| backuptest28092021 |
| information_schema |
| mysql |
| performance_schema |
| sys |
±-------------------+

1 Like

Hi @katajistok,

Please share the logs of operator pod and crashed pods:

$ kubectl logs percona-xtradb-cluster-operator-7c4786cb9d-d4cq5
$ kubectl logs xtradb-poc-env3-pxc-1 pxc
$ kubectl logs xtradb-poc-env3-haproxy-0 haproxy
1 Like

kubectl logs percona-xtradb-cluster-operator-7c4786cb9d-d4cq5: Untitled - Pastebin
kubectl logs xtradb-poc-env3-pxc-1 pxc: Untitled - Pastebin
kubectl logs xtradb-poc-env3-haproxy-0 haproxy: Untitled - Pastebin

1 Like

Hello, I tested the replication just like in the documentation and I got it to work. For some reason the replication did not work at all when there was MetalLB in between. Case closed. Thank you for help.

1 Like

Hmm, so it does not work with MetalLB? I wonder how is it different.
@katajistok thank you for sharing!

1 Like

Yes, exactly. Don’t know why MetalLB didn’t work over there. But if someone encounters this situation, for reference we are using metal Kubernetes cluster with Talos OS on hardware nodes.

1 Like