And in the joiner we are using 18.04 with the Percona version that ships with Ubuntu 18.04: 5.7.20-29.24-0ubuntu2.1
This is the log in the joiner:
2018-10-11T18:09:20.265444Z 2 [Note] WSREP: State gap can’t be serviced using IST. Switching to SST
2018-10-11T18:09:20.265466Z 2 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (7f4a4aab-e3df-11e6-9919-63561272760b): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():538. IST will be unavailable.
2018-10-11T18:09:20.265866Z 0 [Note] WSREP: Member 2.0 (sam) requested state transfer from ‘any’. Selected 1.0 (jon)(SYNCED) as donor.
2018-10-11T18:09:20.265877Z 0 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 20736082206)
2018-10-11T18:09:20.265894Z 2 [Note] WSREP: Requesting state transfer: success, donor: 1
2018-10-11T18:09:20.265926Z 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 → 7f4a4aab-e3df-11e6-9919-63561272760b:20736081608
2018-10-11T18:09:20.604652Z WSREP_SST: [ERROR] Cleanup after exit with status:1
2018-10-11T18:09:20.610732Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘176.9.25.20’ --datadir ‘/var/lib/percona-xtradb-cluster/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘12178’ ‘’ : 1 (Operation not permitted)
2018-10-11T18:09:20.610746Z 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2018-10-11T18:09:20.610754Z 0 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
2018-10-11T18:09:20.610810Z 0 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2018-10-11T18:09:20.610834Z 0 [ERROR] Aborting
Hello,
“Operation not permitted” error suggests some problem with OS restrictions. Do you have Apparmor or SELinux enabled on the new node?
The existing cluster has more then one node already and all running Ubuntu 16?
Can you share also the donor log?
Hi again. I forgot to answer your other question. We have 3 nodes in total: the donor (Ubuntu 16.04 + Percona repo), the joiner (Ubuntu 18.04), and also an arbitrator (Ubuntu 16.04 + Percona repo).
Can you check the datadir has proper permissions on the joiner:
ls -ld /var/lib/percona-xtradb-cluster/
ls -lha /var/lib/percona-xtradb-cluster/
The donor uses default datadir - /var/lib/mysql/, any reason this was different for joiner? And I wonder how many more differences are there in the configuration?
Also, the joiner has lower version 5.7.20 vs 5.7.23.
Ideally, both nodes should be as much identical as possible.
The problem seems to be on the joiner side, and these lines look suspicious:
2018-10-15T11:53:54.384339Z 2 [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (7f4a4aab-e3df-11e6-9919-63561272760b): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():538. IST will be unavailable.
...
2018-10-15T11:53:56.056496Z WSREP_SST: [ERROR] Cleanup after exit with status:1
2018-10-15T11:53:56.062787Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '176.9.25.20' --datadir '/var/lib/percona-xtradb-cluster/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '31523' '' : 1 (Operation not permitted)
...
2018-10-15T11:54:14.988357Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '176.9.25.20' --datadir '/var/lib/percona-xtradb-cluster/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '654' '' : 2 (No such file or directory)
Please attach configuration from both and dmesg log from the joiner.
Btw, what version does the garbd have:
# ls -ld /var/lib/percona-xtradb-cluster/
drwxr-x--- 6 mysql mysql 4096 Oct 16 10:36 /var/lib/percona-xtradb-cluster/
And I empty it before I start the joiner, there are no files in there when the joiner starts.
The reason that the datadir path is different is because it is different in the default configuration of Percona repos and Ubuntu repos. Since I have noticed that Ubuntu defaults to /var/lib/percona-xtradb-cluster/ I have used this datadir in the joiner. The rest of the configuration is mainly the same.
I attach the configuration files from the donor and the joiner.
I know that versions slightly differ but from now on we would like to stick with Ubuntu shipped version which is currently a few minor releases older, but since they are both 5.7 I would expect them to be compatible.
I already attached the joiner log in the previous message.
I attach /var/log/syslog (I understand this is what you mean by “dmesg log”, please correct me if I am wrong), and also the joiner log in case you want to follow timestamps (there is a 2 hour difference due to our timezone).
I have reinstalled the joiner using a clean Ubuntu 18.04 and this time adding Percona repositories. And now the replication process has worked perfectly well.
The joiner version is now the same as the client:
Server version: 5.7.23-23-57-log - Percona XtraDB Cluster (GPL), Release rel23, Revision f5578f0, WSREP version 31.31, wsrep_31.31
Also xtrabackup (is it used for SST?) version is the same:
xtrabackup version 2.4.12 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 170eb8c)
while in the plain Ubuntu setup it was:
xtrabackup version 2.4.9 based on MySQL server 5.7.13 Linux (x86_64) (revision id: a467167cdd4)
So for it looks like that there is a problem with the cross-version compatibility, or maybe the problem is in the configuration of Ubuntu packages that do not handle well the SST process.
We would really like to get Percona from standard Ubuntu repositories so I hope you fix this in the future.