Azure backup fails with /usr/bin/run_backup.sh: 1 (Operation not permitted)

MySQL & MariaDB Percona Operator for MySQL
1

Description:

I’m trying to backup to azure blob storage, and getting the following rather cryptic error in the job logs

+ EXID_CODE=1
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessfull'
Process completed with error: /usr/bin/run_backup.sh: 1 (Operation not permitted)2024-03-27 20:27:26 [ERROR] Backup was finished unsuccessfull
+ exit 1

Backups to the fs-pvc storage work fine.
I triple checked that the “azure-secret” secret exists, is in the correct namespace (pxc), has the correct values for the storage account and shared key, and that those values work fine to upload to azure blob storage (using cloudberry explorer)

Steps to Reproduce:

kubectl apply -f deploy/backup/backup.yaml -n pxc

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterBackup
metadata:
  name: backup-az-1
spec:
  pxcCluster: cluster1
  storageName: azure-blob

Version:

Operator 1.14.0
Cluster 8.0.35

Logs:

backup https://pastebin.com/raw/K8KDJffx
describe pxc https://pastebin.com/raw/Ad4x0Ush
cr.yaml https://pastebin.com/raw/bvfCzFU5

Expected Result:

Backup completes and I can see files uploaded to the azure blob storage container.

Actual Result:

Backup job errors out and no file is created in azure.

2

Hello @starcla.

Thank you for submitting it.
After this line in your backup log:

2024-03-27 20:27:25.918  INFO: [SST script] + xbcloud put --parallel=4 --curl-retriable-errors=7 --storage=azure cluster1-2024-03-27-20:27:19-full.sst_info

I would expect to see:

2024-03-28 12:47:10.158  INFO: [SST script] 240328 12:47:09 xbcloud: Successfully connected.

Where is in your case it goes sideways.

I can’t reproduce it for now.
Is there anything specific in your configuration or networking?

3

Hi @Sergey_Pronin , thanks for looking into this.
I’m not entirely sure of what’s going on with our network, but I tried uploading a file using the azure cli container running in the same cluster and namespace and that worked

% kubectl run -it --rm --restart=Never --image mcr.microsoft.com/azure-cli:latest tempazurecli -n pxc -- sh
If you don't see a command prompt, try pressing enter.
# az storage blob upload --account-name XXXX --account-key 'YYYY' -c pxc-backup -n MyBlob2 -f /etc/udhcpd.conf --verbose
Try to get storage auth_mode value from environment variables or config file.
Try to get storage sas_token value from environment variables or config file.
Request URL: 'https://XXXX.blob.core.windows.net/pxc-backup/MyBlob2'
Request method: 'PUT'
Request headers:
    'Content-Length': '5636'
    'x-ms-blob-type': 'REDACTED'
    'If-None-Match': '*'
    'x-ms-version': 'REDACTED'
    'Content-Type': 'application/octet-stream'
    'Accept': 'application/xml'
    'User-Agent': 'AZURECLI/2.58.0 (DOCKER) azsdk-python-storage-blob/12.16.0 Python/3.11.8 (Linux-4.19.288-2.ph3-x86_64-with)'
    'x-ms-client-request-id': '1a46cfac-ed22-11ee-9cc3-92a0d4c38bfc'
    'CommandName': 'REDACTED'
    'ParameterSetName': 'REDACTED'
    'x-ms-date': 'REDACTED'
    'Authorization': 'REDACTED'
A body is sent with the request
Response status: 201
Response headers:
    'Content-Length': '0'
    'Content-MD5': 'REDACTED'
    'Last-Modified': 'Thu, 28 Mar 2024 16:42:02 GMT'
    'ETag': '"0x8DC4F45FEB1486E"'
    'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
    'x-ms-request-id': '7c10c5c8-001e-00e3-532e-817c4b000000'
    'x-ms-client-request-id': '1a46cfac-ed22-11ee-9cc3-92a0d4c38bfc'
    'x-ms-version': 'REDACTED'
    'x-ms-content-crc64': 'REDACTED'
    'x-ms-request-server-encrypted': 'REDACTED'
    'Date': 'Thu, 28 Mar 2024 16:42:01 GMT'
Finished[#############################################################]  100.0000%
{
  "client_request_id": "1a46cfac-ed22-11ee-9cc3-92a0d4c38bfc",
  "content_md5": "pDTvTt/S6uqWUrYYo7D0hA==",
  "date": "2024-03-28T16:42:01+00:00",
  "encryption_key_sha256": null,
  "encryption_scope": null,
  "etag": "\"0x8DC4F45FEB1486E\"",
  "lastModified": "2024-03-28T16:42:02+00:00",
  "request_id": "7c10c5c8-001e-00e3-532e-817c4b000000",
  "request_server_encrypted": true,
  "version": "2022-11-02",
  "version_id": null
}
Command ran in 0.965 seconds (init: 0.214, invoke: 0.752)
4

Hey @starcla ,

I spent some more time today trying to reproduce it and was able to do it with wrong credentials. For some reason error is not that clear. Will look into that.

I would recommend to try the following (and I understand that you tried with another Pod, but not sure if you used the same Secret resource):

So recommendation is:

  1. when creating base64 string use -n flag in echo. /n is the most common issue for secret generation.
    echo -n YOURKEY | base64
  2. If you are in Linux (not MacOS) than also try adding -w flag to ensure that base64 string is a single string:
    echo -n YOURKEY | base64 -w0
  3. For tests you may try to generate a secret with stringData fields, where you can skip base64 encoding:
apiVersion: v1
kind: Secret
metadata:
  name: azure-secret
type: Opaque
stringData:
  AZURE_STORAGE_ACCOUNT_NAME: %YOUR_STORAGE_ACCOUNT_NAME%
  AZURE_STORAGE_ACCOUNT_KEY: %YOUR_REAL_KEY%

Please let me know if it helps.

5

thanks for pointing me to the wrong credentials. I can backup to azure blob now, it’s just still weird:

  • I tried setting the secret using the plain text stringData and the same key copied directly from the azure portal and it failed.
  • I tried again but using the 2nd key from the portal, and it succeeded.
  • I tried again in exactly the same way using the 1st access key from the portal, and it failed.

I don’t see anything special in the 1st access key
qDI5M73Arf0tMgLnXxecIYspRKAM9OdEs27SSKcAopgRtUZox2BX4gffb5WirWbsds0GmqdSwAra+AStttLbUA== (I’ve rotated it now, it’s no longer valid) but I’m glad it’s resolved for me and that the error message will be more clear in the future.
Thanks again for your help!