Xbcloud failing an upload but exiting 0?

MySQL & MariaDB Percona XtraBackup
1

We’re using MySQL version 5.7.34 and xbstream/xbcloud 2.4.24.

We have an automated restore tool that is receiving an error from s3 saying a key doesn’t exist when attempting to download. During the research of the original upload, it appears xbcloud is getting a non response from aws and then marking the upload as failed. The problem is our code is seeing a 0 return code and thinking everything went as planned. Here’s the output from xbcloud followed by lines of logging indicating what our automation is seeing:

211208 12:24:41 xbcloud: Failed to parse XML response.
211208 12:24:41 xbcloud: error: failed to upload chunk: xtrabackup-XXXX-2021-12-07_1800/schema/table.ibd.qp.xbcrypt.00000000000000013931, size: 65654
211208 12:24:42 xbcloud: Upload failed.

I, [2021-12-08T12:24:42.523266 #15100]  INFO -- : Child process stderr:
I, [2021-12-08T12:24:42.523287 #15100]  INFO -- : Child process exit code: 0

Digging through the xbcloud code at percona-xtrabackup/s3.cc at 8.0 · percona/percona-xtrabackup · GitHub, it appears the code block in question is:

void S3_client::retry_error(Http_response *resp, bool *retry) {
  S3_response s3_resp;
  if (!s3_resp.parse_http_response(*resp)) {
    msg_ts("%s: Failed to parse XML response.\n", my_progname);
  } else if (s3_resp.error()) {
    msg_ts("%s: S3 error message: %s\n", my_progname,
           s3_resp.error_message().c_str());
    if (s3_resp.error_code() == "RequestTimeout") {
      *retry = true;
    }
  }
}

From my limited experience with C++, should that code block be returning false after the unhandled error from AWS?

1 Like
2

Hi There,

Can you please send the full output of xbcloud log and also the command your script is running to invoke xbcloud ?

One think to check beforehand is if you are not using xbcloud with pipe to another program. When using PIPE you need to adjust the way you check return codes. For example:

marcelo@marce-bld:/work/pxb/src/8.0$ false | true
marcelo@marce-bld:/work/pxb/src/8.0$ echo $?
0
marcelo@marce-bld:/work/pxb/src/8.0$ false | true
marcelo@marce-bld:/work/pxb/src/8.0$ echo "${PIPESTATUS[0]} ${PIPESTATUS[1]}"
1 0
2 Likes
3

Looking it over, that looks to be the problem. We’re piping it to grep -v 'successfully uploaded chunk' in order to keep our logs for an extended period of time without filling up disk. I’m sure we can refactor it in order to get out from this behavior. Thank you so much for this find.

1 Like
4

While it looks like we’re simply not handling the failure case properly, the failure itself appears to be more common than it used to. Do you know what may be causing the invalid response that is keeping xbcloud from retrying the upload?

1 Like
5

Please run xbcloud with --verbose flag and send us the full log to analize.

1 Like