Percona XtraDB cluster cross-site replication

MySQL & MariaDB Percona Operator for MySQL
1

Hello,

I am testing Percona XtraDB cluster cross-site replication. I’m following documentation: Set up Percona XtraDB Cluster cross-site replication.

There are two XtraDB clusters involved to this testing: xtradb001-local in pxc-v-1-9-0-local namespace and xtradb-poc-env3 in xtradb-poc-env3 namespace. So both are running within the same Kubernetes cluster. We are using MetalLB for load balancing.

I have configured “xtradb001-local” to be source:
replicationChannels:
- name: pxc1_to_pxc2
isSource: true

And “xtradb-poc-env3” to be the destination for replication:
- name: pxclocal_to_pxcenv3
isSource: false
sourcesList:
- host: 10.183.176.239 #MetalLB external IP address
port: 3306
weight: 100

secrets.yaml contains the default password for “replication” user. This is used on both XtraDB clusters.

This is situation for now:
[root@dbaasjump002 deploy]# kubectl get pods --selector percona.com/replicationPod=true
NAME READY STATUS RESTARTS AGE
xtradb-poc-env3-pxc-0 4/4 Running 1 3h

[root@dbaasjump002 deploy]# kubectl get pods
NAME READY STATUS RESTARTS AGE
centos-mysql 1/1 Running 0 12d
percona-xtradb-cluster-operator-7c4786cb9d-d4cq5 1/1 Running 0 32d
xb-cron-xtradb-poc-env3-fs-pvc-2021928000-372f8-4clch 0/1 Completed 0 10h
xtradb-poc-env3-haproxy-0 2/3 CrashLoopBackOff 40 3h
xtradb-poc-env3-haproxy-1 2/3 Running 40 179m
xtradb-poc-env3-pxc-0 4/4 Running 1 3h
xtradb-poc-env3-pxc-1 3/4 CrashLoopBackOff 33 179m

[root@dbaasjump002 deploy]# kubectl exec -c pxc --stdin --tty xtradb-poc-env3-pxc-0 – /bin/bash
bash-4.4$ mysql -uroot -proot_password
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)

How it would be possible to troubleshoot the issue? I have tested from xtradb-poc-env3-pxc-0 Pod that the connection works to source cluster just fine with MySQL.

bash-4.4$ mysql -h 10.183.176.239 -uroot -proot_password -e ‘SHOW DATABASES;’
mysql: [Warning] Using a password on the command line interface can be insecure.
±-------------------+
| Database |
±-------------------+
| backuptest28092021 |
| information_schema |
| mysql |
| performance_schema |
| sys |
±-------------------+

2 Likes
2

Hi @katajistok,

Please share the logs of operator pod and crashed pods:

$ kubectl logs percona-xtradb-cluster-operator-7c4786cb9d-d4cq5
$ kubectl logs xtradb-poc-env3-pxc-1 pxc
$ kubectl logs xtradb-poc-env3-haproxy-0 haproxy
1 Like
3

kubectl logs percona-xtradb-cluster-operator-7c4786cb9d-d4cq5: Untitled - Pastebin
kubectl logs xtradb-poc-env3-pxc-1 pxc: Untitled - Pastebin
kubectl logs xtradb-poc-env3-haproxy-0 haproxy: Untitled - Pastebin

1 Like
4

Hello, I tested the replication just like in the documentation and I got it to work. For some reason the replication did not work at all when there was MetalLB in between. Case closed. Thank you for help.

2 Likes
5

Hmm, so it does not work with MetalLB? I wonder how is it different.
@katajistok thank you for sharing!

1 Like
6

Yes, exactly. Don’t know why MetalLB didn’t work over there. But if someone encounters this situation, for reference we are using metal Kubernetes cluster with Talos OS on hardware nodes.

1 Like