Hello,
I am trying to set up a three-node cluster on Docker on my Apple M1 MacBook. I am using the following link as guidance: Running Percona XtraDB Cluster in a Docker Container - Percona XtraDB Cluster
The second node constantly fails to join the cluster with the following message: Important update: the same procedure works perfectly fine with image of 5.7
2023-07-24 09:58:31 2023-07-24T13:58:31.643302Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2023-07-24 09:58:31 2023-07-24T13:58:31.643677Z 0 [ERROR] [MY-000000] [WSREP-SST] Failed to start the mysql server that checks for async replication.
2023-07-24 09:58:31 2023-07-24T13:58:31.643768Z 0 [ERROR] [MY-000000] [WSREP-SST] Check the parameters and retry
2023-07-24 09:58:31 2023-07-24T13:58:31.644003Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 521 pid:1901
2023-07-24 09:58:31 2023-07-24T13:58:31.665948Z 0 [ERROR] [MY-000000] [WSREP-SST] ------------ mysql error log (START) ------------
2023-07-24 09:58:31 ---- Starting the MySQL server used for post-processing ----
2023-07-24 09:58:31 /usr/bin/wsrep_sst_common: line 833: /usr/bin/qemu-x86_64: No such file or directory
2023-07-24 09:58:31
2023-07-24 09:58:31 2023-07-24T13:58:31.666226Z 0 [ERROR] [MY-000000] [WSREP-SST] ------------ mysql error log (END) ------------
2023-07-24 09:58:31 2023-07-24T13:58:31.666328Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
I am doing it as simply as possible, following the commands from the procedure exactly. Can someone suggest any directions on what I need to do to make it work?
First node starts properly.
Thanks.
P.S. It’s worth mentioning that I am already successfully running 4 databases with GTID replication on the same Docker, using both 5.7 and 8.0 images (different docker network though)
Hello @egel,
Which container version/tag are you using? Would it be possible to try on something other than an ARM system? The error message looks like it is trying to access an x86 emulation binary which isn’t present. I imagine this is only being called because of ARM M1. I don’t believe any of our database servers are yet officially supported on M1/ARM.
Hello,
Thanks for reply,
I’ve think that the issue is with how the wsrep_sst_common and or wsrep_sst_xtrabackup-v2 expect to see the command during runtime.
In my case it running on the mac so it need to run with qemu-x86_64, i.e. from inside of the container the cmd looks like:
[root@73392b1d912e /]# ps -ef|grep mysql
mysql 1 0 3 19:37 ? 00:01:47 /usr/bin/qemu-x86_64 /usr/sbin/mysqld mysqld --wsrep_start_position=a38d34e9-2a54-11ee-82c6-ce501604110f:30
And there is the catch I believe (instead of working on "/usr/sbin/mysqld " it works on /usr/bin/qemu-x86_64 to setup variables.
I am trying to debug these scripts in the percona image that I am using: percona/percona-xtradb-cluster:8.0 but not yet succeed.
The 5.7 running just fine as well standalone 8.
My final goal is to create simulation of :
cluster 5.7 → replica → cluster 8.0 (cluster 8.0 adding GTID to the 5.7 stream and therefore I can turn cluster 8.0 to be read/write)
Do you, by any chance, have access to a Mac and can try to set up the same environment I am attempting to use? I would appreciate it if you could verify whether the issue indeed exists in this setup.
I am not dismissing the possibility that I might be doing something wrong there.
I do have a Mac but it’s not an ARM-based Mac. Since ARM is not fully supported by us yet, I have to recommend that you attempt this on a non-ARM-based machine and see if you experience the same problems.
What I did:
I made changes to the wsrep_sst_common script. There is a function called get_mysql_path, which is intended to find the path to the mysqld executable.
In the code first attempt, it use the “readlink -f” command, but it returned the simulator executable instead of mysqld (there also possible issue with permissions when use readlink approach, so I think worth to review it any way). In the second attempt, which never got executed because the MYSQLD_PATH was already set (although incorrectly), it use the “which mysqld” command. To fix this issue for me locally, I simply set the MYSQLD_PATH to be empty after the first attempt, and then the second attempt picked up the proper location.
There is also an issue with the wait_for_mysql_startup script. It doesn’t accurately detect when the server is down, and as a result, the script failed, and the second node’s join also failed.
Now after the script fixed I am able to startup the cluster successfully.
p.s. I believe the fixes I did is not related to the arm and probably would benefit in general.
Although the way I fixed probably need to be revised,
Sorry, I was not clear enough.
I am using: percona/percona-xtradb-cluster:8.0 percona/percona-xtradb-cluster:latest
The issue with the wsrep_sst_common that in get_mysql_path() the call to readlink -f returns the /usr/bin/qemu-x86_64 and not /usr/sbin/mysqld.
This is happen because the container (in my case) running in simulator.
there is how it looks from bash prompt of container that running un docker :
ps -ef|grep mysql
mysql 1 0 3 19:37 ? 00:01:47 /usr/bin/qemu-x86_64 /usr/sbin/mysqld mysqld --wsrep_start_position=a38d34e9-2a54-11ee-82c6-ce501604110f:30
readlink -f /proc/1/exe
/usr/bin/qemu-x86_64
ps -ef|grep 1
mysql 1 0 52 06:59 ? 04:45:37 /usr/bin/qemu-x86_64 /usr/sbin/mysqld mysqld
which mysqld
/usr/sbin/mysqld
While we do have ARM docker containers for standard Percona MySQL, I don’t believe there are nor will be for PXC. I only see three choices for you moving forward with this: 1) Switch to a non-ARM machine (anyone can get a 12 month free t3.micro on AWS), 2) continue fiddling with it on your own, or 3) Use Group Replication instead of PXC. Group Replication will soon (a year or two) replace PXC as the go-to for HA Clustering with MySQL. GR is a native plugin to MySQL and since we have native ARM builds of MySQL, GR should work there natively.