K8s pod for myqsl xtrabackup xbcloud: Upload failed

1

Hello, i have this error in all pods from dev and from prod only some time:
2025-10-19 00:05:55.022 INFO: [SST script] 251019 00:05:55 xbcloud: Upload failed.

2025-10-19 00:05:55.022 INFO: [SST script] 2025/10/19 00:05:55 socat[418] E write(1, 0x563d8917e000, 7397): Broken pipe

2025-10-19 00:05:55.023 INFO: [SST script] ++ handle_sigterm

2025-10-19 00:05:55.023 INFO: [SST script] ++ (( 0 == 0 ))

2025-10-19 00:05:55.023 INFO: [SST script] +++ ps -C socat -o pid=

2025-10-19 00:05:55.026 INFO: [SST script] +++ true

2025-10-19 00:05:55.026 INFO: [SST script] ++ pid_s=

2025-10-19 00:05:55.026 INFO: [SST script] ++ ‘[’ -n ‘’ ‘]’

2025-10-19 00:05:55.026 INFO: [SST script] ++ log INFO ‘SST request was finished’

2025-10-19 00:05:55.028 INFO: [SST script] 2025-10-19 00:05:55 [INFO] SST request was finished

2025-10-19 00:05:55.028 INFO: [SST script] + FIRST_RECEIVED=1

2025-10-19 00:05:55.028 INFO: [SST script] + mc -C /tmp/mc stat --insecure dest/k8s.prod.namespace/ssss-db-l-2025-10-19-00:00:39-full.md5

2025-10-19 00:05:55.089 INFO: [SST script] mc: Unable to stat dest/k8s.prod.namespace/ssss-db-l-2025-10-19-00:00:39-full.md5. Object does not exist.

INFO: PC protocol downgrade 1 → 0
2025-10-19 00:05:55.092 INFO: Current view of cluster as seen by this node
view ((empty))
2025-10-19 00:05:55.092 INFO: gcomm: closed
2025-10-19 00:05:55.092 INFO: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2025-10-19 00:05:55.092 INFO: Flow-control interval: [9999999, 9999999]
2025-10-19 00:05:55.092 INFO: Received NON-PRIMARY.
2025-10-19 00:05:55.092 INFO: Shifting JOINER → OPEN (TO: 12147598)
2025-10-19 00:05:55.092 INFO: New SELF-LEAVE.
2025-10-19 00:05:55.092 INFO: Flow-control interval: [9999999, 9999999]
2025-10-19 00:05:55.092 INFO: Received SELF-LEAVE. Closing connection.
2025-10-19 00:05:55.092 INFO: Shifting OPEN → CLOSED (TO: 12147598)
2025-10-19 00:05:55.092 INFO: RECV thread exiting 0: Success
2025-10-19 00:05:55.092 INFO: SST script already terminated
2025-10-19 00:05:55.092 INFO: recv_thread() joined.
2025-10-19 00:05:55.092 INFO: Closing replication queue.
2025-10-19 00:05:55.092 INFO: Closing slave action queue.
2025-10-19 00:05:55.092 INFO: Exiting main loop
Terminating processProcess completed with error: /usr/bin/run_backup.sh: 1 (Operation not permitted)+ EXID_CODE=1

  • ‘[’ -f /tmp/backup-is-completed ‘]’
  • log ERROR ‘Backup was finished unsuccessfull’
    2025-10-19 00:05:55 [ERROR] Backup was finished unsuccessfull
2

@prdmarian

Can you please specify exactly what PXC Operator version you are using ? In some older operator version we have seen a few reported Jira’s related to PXB backup failures.

Is that some customised K8s setup or using the default Percona Operator for MySQL ?

Did you check inside the failing pod and the associated backup jobs if they show any insight? Is that reproducible or just happening randomly ?

kubectl logs <pxc_pod> > pxc_pod.log
kubectl describe pod <pxc_pod> > pxc_pod_desc.txt

kubectl get pxc-backups 
kubectl get pxc-backups <backup_name> - o yaml
kubectl logs <backup job pod> > backup.logs

If you running any manual backup is that uploaded fine of if there still you see the same problem ?

3

app version 1.14.0 yes all pods what fail have that
2025-10-19 00:05:55.022 INFO: [SST script] 251019 00:05:55 xbcloud: Upload failed.

2025-10-19 00:05:55.022 INFO: [SST script] 2025/10/19 00:05:55 socat[418] E write(1, 0x563d8917e000, 7397): Broken pipe

4

@prdmarian

Did you try with any manual backups other than the scheduled one if you still face any such issues ? You can try changing the scheduled backup timing if the issue happening frequently.

Please share the requested logs and information.

Have you checked internally to ensure there are no network issues that might disrupt the upload process?