We get following errors sometimes during the backup
2025-11-21 00:00:59.857 INFO: (2957505e-8792, ‘ssl://0.0.0.0:4567’) connection established to bd74745f-bcef ssl://10.128.14.196:4567
2025-11-21 00:00:59.861 INFO: (2957505e-8792, ‘ssl://0.0.0.0:4567’) connection established to 8b448633-967d ssl://10.128.16.89:4567
2025-11-21 00:00:59.861 INFO: Current view of cluster as seen by this node
view (view_id(PRIM,2957505e-8792,696)
memb {
2957505e-8792,0
692d6df6-a7bb,0
8b448633-967d,0
bd74745f-bcef,0
}
joined {
}
left {
}
partitioned {
}
)
2025-11-21 00:00:59.861 INFO: Save the discovered primary-component to disk
2025-11-21 00:00:59.861 WARN: open file(./gvwstate.dat.tmp) failed(Permission denied)
EXID_CODE=141
‘[’ -f /tmp/backup-is-completed ‘]’
log ERROR ‘Backup was finished unsuccessfull’
2025-11-21 00:00:59 [ERROR] Backup was finished unsuccessfull
This error code is generated by garbd when it is unable to establish proper connectivity with the cluster. The error can occur if the connection times out or fails for any reason.
You can review the associated error code and logs to obtain more details about the timeout or failure on the donor node as well.
I debugged the script before and observed same line from script. But script is coming with percona distribution and i could not decide what should i do.
Sometimes we are observing following error in same cluster. It is randomly/sporadically and most probably it has same RCA
issue:
2024-05-03 09:39:08.414 INFO: [SST script] Metadata : 2024-05-03 09:39:08.414 INFO: [SST script] Content-Type: application/octet-stream 2024-05-03 09:39:08.414 INFO: [SST script] 2024-05-03 09:39:08.419 INFO: [SST script] ++ mc -C /tmp/mc stat --json dest/percona-xtradb-backups-2024-05-01/percona-xtradb-2024-05-03-09:38:29-full.md5 2024-05-03 09:39:08.419 INFO: [SST script] ++ sed -e 's/.*"size":\([0-9]*\).*/\1/' 2024-05-03 09:39:08.480 INFO: Donor no longer in donor state, interrupting script 2024-05-03 09:39:08.481 INFO: [SST script] + md5_size= 2024-05-03 09:39:08.481 INFO: [SST script] ++ handle_sigterm 2024-05-03 09:39:08.481 INFO: [SST script] ++ (( 1 == 0 )) 2024-05-03 09:39:08.484 INFO: SST script ended 2024-05-03 09:39:08.484 INFO: Closing send monitor... 2024-05-03 09:39:08.484 INFO: Closed send monitor. 2024-05-03 09:39:08.484 INFO: gcomm: terminating thread 2024-05-03 09:39:08.484 INFO: gcomm: joining thread 2024-05-03 09:39:08.484 INFO: gcomm: closing backend 2024-05-03 09:39:08.485 INFO: Current view of cluster as seen by this node view (view_id(NON_PRIM,56563bdb-8188,185) memb { e9a3befc-8de4,0 } joined { } left { } partitioned { 56563bdb-8188,0 a9481be8-8f18,0 fbe969b4-ae67,0 } ) 2024-05-03 09:39:08.485 INFO: PC protocol downgrade 1 -> 0 2024-05-03 09:39:08.485 INFO: Current view of cluster as seen by this node view ((empty)) 2024-05-03 09:39:08.485 INFO: gcomm: closed 2024-05-03 09:39:08.485 INFO: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2024-05-03 09:39:08.485 INFO: Flow-control interval: [9999999, 9999999] 2024-05-03 09:39:08.485 INFO: Received NON-PRIMARY. 2024-05-03 09:39:08.485 INFO: Shifting JOINER -> OPEN (TO: 114621956) 2024-05-03 09:39:08.485 INFO: New SELF-LEAVE. 2024-05-03 09:39:08.485 INFO: Flow-control interval: [9999999, 9999999] 2024-05-03 09:39:08.485 INFO: Received SELF-LEAVE. Closing connection. 2024-05-03 09:39:08.485 INFO: Shifting OPEN -> CLOSED (TO: 114621956) 2024-05-03 09:39:08.485 INFO: RECV thread exiting 0: Success 2024-05-03 09:39:08.485 INFO: SST script already terminated 2024-05-03 09:39:08.486 INFO: recv_thread() joined. 2024-05-03 09:39:08.486 INFO: Closing replication queue. 2024-05-03 09:39:08.486 INFO: Closing slave action queue. 2024-05-03 09:39:08.486 INFO: Exiting main loop + EXID_CODE=4 + '[' -f /tmp/backup-is-completed ']' + log ERROR 'Backup was finished unsuccessfull' Terminating processProcess completed with error: /usr/bin/run_backup.sh: 4 (Interrupted system call)2024-05-03 09:39:08 [ERROR] Backup was finished unsuccessfull + exit 4
The following ticket opened for this case but it was closed
I am trying to understand what can we do solve this issue.