In the production environment, one node in the cluster failed. After several attempts to restart it, we couldn’t restore it to operation. Finally, a full synchronization of the node had to be performed.
Currently, we have synchronization configured with only one specific node. The data transfer itself ends successfully on both the donor and joiner sides:
joiner:
2024-07-09T10:39:53.743907Z 0 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000027 of size 134217728 bytes
2024-07-09T10:40:51.295875Z 0 [Note] WSREP: 0.0 (host.donor): State transfer to 2.0 (host.joiner) complete.
donor:
2024-07-09T10:40:51.298101Z 0 [Note] WSREP: 0.0 (host.donor): State transfer to 2.0 (host.joiner) complete.
However, for some reason, the following things happen on the joiner side afterwards:
2024-07-09T10:40:51.297100Z WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst
2024-07-09T10:40:51.301475Z 0 [Note] WSREP: Member 0.0 (host.donor) synced with group.
2024-07-09T10:43:07.598356Z WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/
2024-07-09T10:43:07.721110Z WSREP_SST: [ERROR] Cleanup after exit with status:1
2024-07-09T10:43:07.744004Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘xxx.xxx.xxx.xxx’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘25937’ --mysqld-version ‘5.7.43-47-57’ ‘’ : 1 (Operation not permitted)
2024-07-09T10:43:07.744042Z 0 [ERROR] WSREP: Failed to read uuid
from joiner script.
2024-07-09T10:43:07.744086Z 0 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted)
2024-07-09T10:43:07.744139Z 0 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2024-07-09T10:43:07.746571Z 0 [ERROR] Aborting
Generally, it looks like a problem with moving the data files from “/var/lib/mysql//.sst” to “/var/lib/mysql/”. We checked the directory permissions, but everything seems correct. After restarting the virtual machine, unfortunately, the database started the synchronization again. Currently, the synchronization is being performed again but this time with wsrep_debug=ON in hopes of learning more about the error. We also plan to manually move the data files to the /var/lib/mysql/ directory if we encounter the same error, although we are not sure if that makes sense at all.
Do you have any ideas about what could be causing the error during the “postprocessing” of the already fetched data? We have a Percona cluster version 5.7.43-47-57 (yes, we know that support for this version has ended, but there is no possibility for a quick upgrade to version 8).
Best regards,
Eryk Michalik