K8s pod for myqsl xtrabackup xbcloud: Upload failed

1

Hello, i have this error in all pods from dev and from prod only some time:
2025-10-19 00:05:55.022 INFO: [SST script] 251019 00:05:55 xbcloud: Upload failed.

2025-10-19 00:05:55.022 INFO: [SST script] 2025/10/19 00:05:55 socat[418] E write(1, 0x563d8917e000, 7397): Broken pipe

2025-10-19 00:05:55.023 INFO: [SST script] ++ handle_sigterm

2025-10-19 00:05:55.023 INFO: [SST script] ++ (( 0 == 0 ))

2025-10-19 00:05:55.023 INFO: [SST script] +++ ps -C socat -o pid=

2025-10-19 00:05:55.026 INFO: [SST script] +++ true

2025-10-19 00:05:55.026 INFO: [SST script] ++ pid_s=

2025-10-19 00:05:55.026 INFO: [SST script] ++ ‘[’ -n ‘’ ‘]’

2025-10-19 00:05:55.026 INFO: [SST script] ++ log INFO ‘SST request was finished’

2025-10-19 00:05:55.028 INFO: [SST script] 2025-10-19 00:05:55 [INFO] SST request was finished

2025-10-19 00:05:55.028 INFO: [SST script] + FIRST_RECEIVED=1

2025-10-19 00:05:55.028 INFO: [SST script] + mc -C /tmp/mc stat --insecure dest/k8s.prod.namespace/ssss-db-l-2025-10-19-00:00:39-full.md5

2025-10-19 00:05:55.089 INFO: [SST script] mc: Unable to stat dest/k8s.prod.namespace/ssss-db-l-2025-10-19-00:00:39-full.md5. Object does not exist.

INFO: PC protocol downgrade 1 → 0
2025-10-19 00:05:55.092 INFO: Current view of cluster as seen by this node
view ((empty))
2025-10-19 00:05:55.092 INFO: gcomm: closed
2025-10-19 00:05:55.092 INFO: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2025-10-19 00:05:55.092 INFO: Flow-control interval: [9999999, 9999999]
2025-10-19 00:05:55.092 INFO: Received NON-PRIMARY.
2025-10-19 00:05:55.092 INFO: Shifting JOINER → OPEN (TO: 12147598)
2025-10-19 00:05:55.092 INFO: New SELF-LEAVE.
2025-10-19 00:05:55.092 INFO: Flow-control interval: [9999999, 9999999]
2025-10-19 00:05:55.092 INFO: Received SELF-LEAVE. Closing connection.
2025-10-19 00:05:55.092 INFO: Shifting OPEN → CLOSED (TO: 12147598)
2025-10-19 00:05:55.092 INFO: RECV thread exiting 0: Success
2025-10-19 00:05:55.092 INFO: SST script already terminated
2025-10-19 00:05:55.092 INFO: recv_thread() joined.
2025-10-19 00:05:55.092 INFO: Closing replication queue.
2025-10-19 00:05:55.092 INFO: Closing slave action queue.
2025-10-19 00:05:55.092 INFO: Exiting main loop
Terminating processProcess completed with error: /usr/bin/run_backup.sh: 1 (Operation not permitted)+ EXID_CODE=1

  • ‘[’ -f /tmp/backup-is-completed ‘]’
  • log ERROR ‘Backup was finished unsuccessfull’
    2025-10-19 00:05:55 [ERROR] Backup was finished unsuccessfull
2

@prdmarian

Can you please specify exactly what PXC Operator version you are using ? In some older operator version we have seen a few reported Jira’s related to PXB backup failures.

Is that some customised K8s setup or using the default Percona Operator for MySQL ?

Did you check inside the failing pod and the associated backup jobs if they show any insight? Is that reproducible or just happening randomly ?

kubectl logs <pxc_pod> > pxc_pod.log
kubectl describe pod <pxc_pod> > pxc_pod_desc.txt

kubectl get pxc-backups 
kubectl get pxc-backups <backup_name> - o yaml
kubectl logs <backup job pod> > backup.logs

If you running any manual backup is that uploaded fine of if there still you see the same problem ?

3

app version 1.14.0 yes all pods what fail have that
2025-10-19 00:05:55.022 INFO: [SST script] 251019 00:05:55 xbcloud: Upload failed.

2025-10-19 00:05:55.022 INFO: [SST script] 2025/10/19 00:05:55 socat[418] E write(1, 0x563d8917e000, 7397): Broken pipe