Kubernetes: PBM not working with GCS bucket

MongoDB Percona Backup for MongoDB
, ,
1

I have deployed percona-operator and percona server for mongodb using helm on GKE cluster (gke-v1.21.14).
MongoDB(percona-server-mongodb:6.0.3) replicaset is running fine.

But I’m unable to setup the backups as I’m getting this error in backup-agent (pbm sidecar) container.

 2023-01-19T19:16:46.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result                                             │
│ 2023-01-19T19:16:51.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result                                             │
│ 2023-01-19T19:16:56.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result

Output of pbm list command from pbm container.

bash-4.4$ pbm config list
Error: unable to get config key: invalid config key

Output of pbm status

bash-4.4$ pbm status
Error: get status of pitr: check for errors: get current epoch: get config: get: mongo: no documents in result
bash-4.4$ pbm status -s cluster
Cluster:
========
rs0:
  - rs0/psmdb6-of-rs0-1.psmdb6-of-rs0.la5-inf-wyn.svc.cluster.local:27017 [S]: pbm-agent v2.0.3 FAILED status:
      > ERROR with storage: unable to get storage: get config: get: mongo: no documents in result
  - rs0/psmdb6-of-rs0-0.psmdb6-of-rs0.la5-inf-wyn.svc.cluster.local:27017 [P]: pbm-agent v2.0.3 FAILED status:
      > ERROR with storage: unable to get storage: get config: get: mongo: no documents in result
  - rs0/psmdb6-of-rs0-2.psmdb6-of-rs0.la5-inf-wyn.svc.cluster.local:27017 [S]: pbm-agent v2.0.3 FAILED status:
      > ERROR with storage: unable to get storage: get config: get: mongo: no documents in result

Here are the operator logs

│ 2023-01-20T19:28:00.534Z    INFO    controller_psmdb    Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}                                                                                                                                                                                                                                                                 │
│ 2023-01-20T19:28:05.572Z    INFO    controller_psmdb    Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}                                                                                                                                                                                                                                                                 │
│ 2023-01-20T19:28:41.484Z    INFO    controller_psmdb    Cluster state changed    {"previous": "initializing", "current": "ready"}                                                                                                                                                                                                                                                        │
│ 2023-01-20T19:28:41.691Z    INFO    controller_psmdb    Point-in-time recovery will work only with full backup. Please create one manually or wait for scheduled backup to be created (if configured).                                                                                                                                                                                   │
│ 2023-01-20T19:28:46.689Z    INFO    controller_psmdb    Point-in-time recovery will work only with full backup. Please create one manually or wait for scheduled backup to be created (if configured).                                                                                                                                                                                   │
│ 2023-01-20T19:28:51.883Z    INFO    controller_psmdb    Point-in-time recovery will work only with full backup. Please create one manually or wait for scheduled backup to be created (if configured).

Here is the config for backup option in PSMDB custom resource:

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.0.3
    pitr:
      compressionLevel: 6
      compressionType: gzip
      enabled: true
      oplogSpanMin: 10
    serviceAccountName: psmdb-backup-agent
    storages:
      google-cloud-storage-s3:
        s3:
          bucket: percona-backups
          credentialsSecret: google-cloud-storage-s3-backup
          endpointUrl: https://storage.googleapis.com/
          prefix: mongodb-of/
          region: europe-west-1
        type: s3

I read through this doc Backup and restore - Percona Operator for MongoDB to setup the backups.

So, is there something missing here in the storage config to connect to the GCS bucket?
Can someone please take a look and see how can this be resolved?
Thank you.

1 Like
2

Hey @mbihl ,

I don’t see anything wrong with your configuration.
Have you tried to execute a manual backup? Do you see additional hints about issues in backup-agent container logs?
Anything in psmdb-backup?

3

@Sergey_Pronin I did try to execute a manual backup and it didn’t work.
Here are logs from back-agent container.

│ 2023-01-23T18:06:11.000+0000 I got command backup [name: 2023-01-23T18:06:10Z, compression: gzip (level: default)] <ts: 1674497170>                                                       │
│ 2023-01-23T18:06:11.000+0000 I got epoch {1674497158 7}                                                                                                                                   │
│ 2023-01-23T18:06:11.000+0000 D [backup/2023-01-23T18:06:10Z] init backup meta                                                                                                             │
│ 2023-01-23T18:06:11.000+0000 D [backup/2023-01-23T18:06:10Z] nomination list for rs0: []                                                                                                  │
│ 2023-01-23T18:06:13.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated.  │
│     For verbose messaging see aws.Config.CredentialsChainVerboseErrors                                                                                                                    │
│ 2023-01-23T18:06:18.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated.  │
│     For verbose messaging see aws.Config.CredentialsChainVerboseErrors                                                                                                                    │
│ 2023-01-23T18:06:23.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated.  │
│     For verbose messaging see aws.Config.CredentialsChainVerboseErrors                                                                                                                    │
│ 2023-01-23T18:06:26.000+0000 D [backup/2023-01-23T18:06:10Z] nomination timeout                                                                                                           │
│ 2023-01-23T18:06:26.000+0000 D [backup/2023-01-23T18:06:10Z] skip after nomination, probably started by another node

The psmdb-backup has an error status.
image

For the secret to access the storage bucket, I have used key names as GCP_ACCESS_KEY_ID and GCP_SECRET_ACCESS_KEY, so are the name for the keys correct as I’m trying to use GCS bucket?

4

I think the key names should use AWS instead of GCP,so I changed those to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and now the error message is different.

│ 2023-01-23T20:48:56.000+0000 I got epoch {1674506923 2}                                                                                                                                   │
│ 2023-01-23T20:48:56.000+0000 D [backup/2023-01-23T20:48:56Z] init backup meta                                                                                                             │
│ 2023-01-23T20:48:56.000+0000 D [backup/2023-01-23T20:48:56Z] nomination list for rs0: []                                                                                                  │
│ 2023-01-23T20:48:59.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed                                │
│ caused by: Head "https://storage.googleapis.com/percona-backups-mi9-tst/mongodb-of/.pbm.init": net/http: invalid header field value for "Authorization"                                   │
│ 2023-01-23T20:49:04.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed                                │
│ caused by: Head "https://storage.googleapis.com/percona-backups-mi9-tst/mongodb-of/.pbm.init": net/http: invalid header field value for "Authorization"

How can I check the header field value?

5

The issue was that there was a newline in the encoded secret value and was fixed.
This is resolved now and PBM is working with GCS bucket.

Thanks

6

@mbihl glad you figured it out! Let me know if there is anything I can help with.

7

Hi , I want to know if we can use workload identify for the service account operator instead of credentialsSecret on Google platform

1 Like
8

Currently workload identity is not supported in the Operator.

We have it in our roadmap - [K8SPSMDB-410] Add support for Workload Identity Authentication · Issue #26 · percona/roadmap · GitHub

Feel free to upvote it and we’ll see how to prio it better.

9

Thank you . Please let me know what goes in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for Google storage access for backup.

I have the Json similar to below .
{
“auth_provider_x509_cert_url”:“https://www.googleapis.com/oauth2/v1/certs”,
“auth_uri”:“Sign in - Google Accounts”,
“client_email”:“,
“client_id”:”“,
“client_x509_cert_url”:”“,
“private_key”:”“,
“private_key_id”:”“,
“project_id”:”",
“token_uri”:“https://oauth2.googleapis.com/token”,
“type”:“service_account”
}

10

HMAC keys need to be created for the service account to access GCS bucket and the AccessId and SecretKey values generated will go in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values in the k8s secret.

1 Like