I am trying to add a 4th node to an existing 3 node cluster and am getting the following errors:
2022-08-16T20:48:54.136263Z 0 [Note] [MY-000000] [WSREP-SST] Moving the backup to /data/
2022-08-16T20:48:54.147118Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-08-16T20:48:54.147849Z 0 [ERROR] [MY-000000] [WSREP-SST] Move failed.
2022-08-16T20:48:54.147862Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 2409
Can not move file ibdata1 to /data//ibdata1: No such file or directory
Anyone run into this? All directory permissions are the same on the donor node and the new node as well as config file locations and settings.
2 Likes
HI @jimbo881 , thanks for posting to the Percona forums!
It seems as if PXC can’t find the path /data . Do you have selinux or apparmor in enforcing mode?
1 Like
Actually the problem can be seen on the donor node:
2022-10-14T16:16:41.147509Z 0 [ERROR] [MY-000000] [Galera] async IST sender failed to serve ssl://xxx.xxx.xxx:4568: ist send failed: asio.system:104’, asio error ‘write: Connection reset by peer’: 104 (Connection reset by peer)
at galera/src/ist.cpp:send():1162
The joiner node actually receives some of the files:
221014 16:16:15 [01] Moving undo_001 to /db01/mysql01/data//undo_001
221014 16:16:15 [01] ...done
221014 16:16:15 [01] Moving undo_002 to /db01/mysql01/data//undo_002
221014 16:16:15 [01] ...done
221014 16:16:15 [01] Moving ib_logfile0 to /db01/mysql01/data//ib_logfile0
221014 16:16:15 [01] ...done
221014 16:16:15 [01] Moving ib_logfile1 to /db01/mysql01/data//ib_logfile1
221014 16:16:15 [01] ...done
221014 16:16:15 [01] Moving ibdata1 to /db01/mysql01/data//ibdata1
Can not move file ibdata1 to /db01/mysql01/data//ibdata1: No such file or directory
Cannot find any reason why…
1 Like
Hi @jimbo881 ,
In my opinion, the error you see in the donor’s log is the consequence of the error on the joiner side. Joiner is not able to restore SST (Can not move file ibdata1 to) so cancels IST and gives up.
Could you please provide complete logs from the donor and joiner (for the time of joiner joins the cluster) ?
1 Like
From the Joiner:
2022-10-14T17:45:57.621905Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘xxx.xxx.xxx.xxx’ --datadir ‘/db01/mysql01/data/’ --basedir ‘/db01/release/mysql-8.0.25/Percona-XtraDB-Cluster_8.0.25-15.1_Linux.x86_64.glibc2.17/’ --plugindir ‘/db01/plugin_dir/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘20455’ --mysqld-version ‘8.0.25-15.1’ ‘’ )
2022-10-14T17:45:57.710629Z 0 [Warning] [MY-000000] [WSREP-SST] wsrep_node_address or wsrep_sst_receive_address not set. Consider setting them if SST fails.
2022-10-14T17:45:57.864499Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /db01/mysql01/data//sst_in_progress
2022-10-14T17:45:58.102030Z 0 [Note] [MY-000000] [Galera] (0d4ced4e-881d, ‘ssl://0.0.0.0:4567’) turning message relay requesting off
2022-10-14T17:45:58.162310Z 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|xxx.xxx.xxx.xxx:4444/xtrabackup_sst//1
2022-10-14T17:45:58.162670Z 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2022-10-14T17:45:58.162925Z 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 08a2fa87-fe03-11ec-900b-9a31e63c76bb
2022-10-14T17:45:58.163183Z 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 549, STRv: 3
2022-10-14T17:45:58.163496Z 2 [Note] [MY-000000] [Galera] IST receiver addr using ssl://151.119.201.6:4568
2022-10-14T17:45:58.163782Z 2 [Note] [MY-000000] [Galera] IST receiver using ssl
2022-10-14T17:45:58.164315Z 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-549, listening at: ssl://xxx.xxx.xxx.xxx:4568
2022-10-14T17:45:58.186373Z 0 [Note] [MY-000000] [Galera] Member 1.0 (mp-ddc01) requested state transfer from ‘any’. Selected 0.0 (mp-idc01)(SYNCED) as donor.
2022-10-14T17:45:58.186718Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY → JOINER (TO: 549)
2022-10-14T17:45:58.187052Z 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2022-10-14T17:45:58.187373Z 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2022-10-14T17:45:58.187648Z 2 [Note] [MY-000000] [Galera] GCache history reset: 08a2fa87-fe03-11ec-900b-9a31e63c76bb:0 → 08a2fa87-fe03-11ec-900b-9a31e63c76bb:549
2022-10-14T17:45:58.955957Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST…
2022-10-14T17:45:58.966366Z 0 [Note] [MY-000000] [WSREP-SST] …Waiting for SST streaming to complete!
2022-10-14T17:46:13.980976Z 0 [Note] [MY-000000] [WSREP-SST] Preparing the backup at /db01/mysql01/data//sst-xb-tmpdir
2022-10-14T17:46:13.994207Z 0 [Note] [MY-000000] [Galera] 0.0 (mp-idc01): State transfer to 1.0 (mp-ddc01) complete.
2022-10-14T17:46:14.021445Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mp-idc01) synced with group.
2022-10-14T17:46:16.578979Z 0 [Note] [MY-000000] [WSREP-SST] Moving the backup to /db01/mysql01/data/
2022-10-14T17:46:16.590532Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR **********************
2022-10-14T17:46:16.591247Z 0 [ERROR] [MY-000000] [WSREP-SST] Move failed.
2022-10-14T17:46:16.591259Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 2409
2022-10-14T17:46:16.591618Z 0 [ERROR] [MY-000000] [WSREP-SST] ------------ innobackup.move.log (START) ------------
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --defaults_group=mysqld --datadir=/db01/mysql01/data/
xtrabackup: recognized client arguments: --move-back=1 --force-non-empty-directories=1 --xtrabackup-plugin-dir=/db01/release/mysql-8.0.25/Percona-XtraDB-Cluster_8.0.25-15.1_Linux.x86_64.glibc2.17/bin/pxc_extra/pxb-8.0/lib/plugin --target-dir=/db01/mysql01/data//sst-xb-tmpdir
/db01/release/mysql-8.0.25/Percona-XtraDB-Cluster_8.0.25-15.1_Linux.x86_64.glibc2.17/bin/pxc_extra/pxb-8.0/bin/xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
221014 17:46:16 [01] Moving undo_001 to /db01/mysql01/data//undo_001
221014 17:46:16 [01] …done
221014 17:46:16 [01] Moving undo_002 to /db01/mysql01/data//undo_002
221014 17:46:16 [01] …done
221014 17:46:16 [01] Moving ib_logfile0 to /db01/mysql01/data//ib_logfile0
221014 17:46:16 [01] …done
221014 17:46:16 [01] Moving ib_logfile1 to /db01/mysql01/data//ib_logfile1
221014 17:46:16 [01] …done
221014 17:46:16 [01] Moving ibdata1 to /db01/mysql01/data//ibdata1
Can not move file ibdata1 to /db01/mysql01/data//ibdata1: No such file or directory
2022-10-14T17:46:16.591648Z 0 [ERROR] [MY-000000] [WSREP-SST] ------------ innobackup.move.log (END) ------------
2022-10-14T17:46:16.591662Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************************************************
2022-10-14T17:46:16.591671Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:22
2022-10-14T17:46:16.615171Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘xxx.xxx.xxx.xxx’ --datadir ‘/db01/mysql01/data/’ --basedir ‘/db01/release/mysql-8.0.25/Percona-XtraDB-Cluster_8.0.25-15.1_Linux.x86_64.glibc2.17/’ --plugindir ‘/db01/plugin_dir/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘20455’ --mysqld-version ‘8.0.25-15.1’ ‘’ : 22 (Invalid argument)
2022-10-14T17:46:16.615223Z 0 [ERROR] [MY-000000] [WSREP] Failed to read uuid:seqno from joiner script.
From the Donor:
members(3):
0: 08a2186f-fe03-11ec-9417-8b55156c49a5, mp-idc01
1: 7ad2e74c-4bdb-11ed-91de-f2dfab42a2bc, mp-ddc01
2: f69d9d19-4b51-11ed-933a-d7813f79ba6c, mp-odc01
2022-10-14T16:16:21.150333Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-10-14T16:16:21.154228Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 547
2022-10-14T16:16:21.154271Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 313
2022-10-14T16:16:21.154291Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2022-10-14T16:16:21.737638Z 0 [Note] [MY-000000] [Galera] Member 1.0 (mp-ddc01) requested state transfer from ‘any’. Selected 0.0 (mp-idc01)(SYNCED) as donor.
2022-10-14T16:16:21.737694Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED → DONOR/DESYNCED (TO: 547)
2022-10-14T16:16:21.737802Z 1 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 134, req: STRv1
2022-10-14T16:16:21.737887Z 1 [Note] [MY-000000] [Galera] Cert index preload: 313 → 547
2022-10-14T16:16:21.738166Z 1 [Note] [MY-000000] [Galera] IST sender using ssl
2022-10-14T16:16:21.747423Z 1 [Note] [MY-000000] [WSREP] Server status change synced → donor
2022-10-14T16:16:21.747450Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-10-14T16:16:21.747474Z 0 [Note] [MY-000000] [Galera] async IST sender starting to serve ssl://151.119.201.6:4568 sending 313-547, preload starts from 313
2022-10-14T16:16:21.747666Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘xxx.xxx.xxx.xxx:4444/xtrabackup_sst//1’ --socket ‘/tmp/mysql.sock’ --datadir ‘/db01/mysql01/data/’ --basedir ‘/db01/release/mysql-8.0.25/Percona-XtraDB-Cluster_8.0.25-15.1_Linux.x86_64.glibc2.17/’ --plugindir ‘/db01/plugin_dir/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --mysqld-version ‘8.0.25-15.1’ ‘’ --gtid ‘08a2fa87-fe03-11ec-900b-9a31e63c76bb:547’ )
2022-10-14T16:16:21.759995Z 1 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2022-10-14T16:16:21.791658Z 0 [Note] [MY-000000] [Galera] IST sender 313 → 547
2022-10-14T16:16:21.945245Z 0 [Warning] [MY-000000] [WSREP-SST] wsrep_node_address or wsrep_sst_receive_address not set. Consider setting them if SST fails.
2022-10-14T16:16:22.902102Z 0 [Note] [MY-000000] [Galera] (08a2186f-9417, ‘ssl://0.0.0.0:4567’) turning message relay requesting off
2022-10-14T16:16:32.403624Z 0 [Note] [MY-000000] [WSREP-SST] Streaming the backup to joiner at xxx.xxx.xxx.xxx 4444
2022-10-14T16:16:32.536785Z 63514 [Warning] [MY-013712] [Server] No suitable ‘keyring_component_metadata_query’ service implementation found to fulfill the request.
2022-10-14T16:16:37.509311Z 0 [Note] [MY-000000] [Galera] SST sent: 08a2fa87-fe03-11ec-900b-9a31e63c76bb:547
2022-10-14T16:16:37.509351Z 0 [Note] [MY-000000] [WSREP] Server status change donor → joined
2022-10-14T16:16:37.509373Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-10-14T16:16:37.554428Z 0 [Note] [MY-000000] [Galera] 0.0 (mp-idc01): State transfer to 1.0 (mp-ddc01) complete.
2022-10-14T16:16:37.554482Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED → JOINED (TO: 547)
2022-10-14T16:16:37.598739Z 0 [Note] [MY-000000] [Galera] Member 0.0 (mp-idc01) synced with group.
2022-10-14T16:16:37.598777Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 547)
2022-10-14T16:16:37.599049Z 15 [Note] [MY-000000] [Galera] Server mp-idc01 synced with group
2022-10-14T16:16:37.599096Z 15 [Note] [MY-000000] [WSREP] Server status change joined → synced
2022-10-14T16:16:37.599107Z 15 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2022-10-14T16:16:37.599116Z 15 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2022-10-14T16:16:41.147509Z 0 [ERROR] [MY-000000] [Galera] async IST sender failed to serve ssl://151.119.201.6:4568: ist send failed: asio.system:104’, asio error ‘write: Connection reset by peer’: 104 (Connection reset by peer)
at galera/src/ist.cpp:send():1162
2022-10-14T16:16:41.147737Z 0 [Note] [MY-000000] [Galera] async IST sender served
2022-10-14T16:16:41.156323Z 0 [Note] [MY-000000] [Galera] declaring f69d9d19-933a at ssl://151.117.160.135:4567 stable
2022-10-14T16:16:41.156352Z 0 [Note] [MY-000000] [Galera] forgetting 7ad2e74c-91de (ssl://151.119.201.6:4567)
2022-10-14T16:16:41.184643Z 0 [Note] [MY-000000] [Galera] Node 08a2186f-9417 state primary
2022-10-14T16:16:41.199007Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,08a2186f-9417,143)
memb {
08a2186f-9417,0
f69d9d19-933a,0
}
joined {
}
left {
}
partitioned {
1 Like
In your my.cnf in please add
[sst]
wsrep-debug=1
That should cause printing more diagnostics and the information where the transferred SST is. Once it fails again please try to investigate if the missing file is transferred or not. It should be inside /db01/mysql01/data//sst-xb-tmpdir
BTW, please consider using the newest version of PXC. Sice 8.0.25 there were many bugs fixed around SST
1 Like