# Kubernetes: PBM not working with GCS bucket

**URL:** <https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700>\
**Category:** Percona Backup for MongoDB\
**Tags:** percona, mongodb, pbm\
**Created:** [January 20, 2023, 7:38pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700 "2023-01-20T19:38:07Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mbihl](https://avatars.discourse-cdn.com/v4/letter/m/bc79bd/32.png) [@mbihl](https://forums.percona.com/u/mbihl)\
**Post date:** [January 20, 2023, 7:38pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/1 "2023-01-20T19:38:08Z")

</div>

I have deployed percona-operator and percona server for mongodb using helm on GKE cluster (gke-v1.21.14).  
MongoDB( **percona-server-mongodb:6.0.3** ) replicaset is running fine.

But I’m unable to setup the backups as I’m getting this error in **backup-agent** (pbm sidecar) container.

```auto
 2023-01-19T19:16:46.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result │
│ 2023-01-19T19:16:51.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result │
│ 2023-01-19T19:16:56.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result

```

Output of **pbm list** command from pbm container.

```auto
bash-4.4$ pbm config list
Error: unable to get config key: invalid config key

```

Output of **pbm status**

```auto
bash-4.4$ pbm status
Error: get status of pitr: check for errors: get current epoch: get config: get: mongo: no documents in result
bash-4.4$ pbm status -s cluster
Cluster:
========
rs0:
  - rs0/psmdb6-of-rs0-1.psmdb6-of-rs0.la5-inf-wyn.svc.cluster.local:27017 [S]: pbm-agent v2.0.3 FAILED status:
      > ERROR with storage: unable to get storage: get config: get: mongo: no documents in result
  - rs0/psmdb6-of-rs0-0.psmdb6-of-rs0.la5-inf-wyn.svc.cluster.local:27017 [P]: pbm-agent v2.0.3 FAILED status:
      > ERROR with storage: unable to get storage: get config: get: mongo: no documents in result
  - rs0/psmdb6-of-rs0-2.psmdb6-of-rs0.la5-inf-wyn.svc.cluster.local:27017 [S]: pbm-agent v2.0.3 FAILED status:
      > ERROR with storage: unable to get storage: get config: get: mongo: no documents in result

```

Here are the operator logs

```auto
│ 2023-01-20T19:28:00.534Z INFO controller_psmdb Waiting for the pods {"replset": "rs0", "size": 3, "pods": 2} │
│ 2023-01-20T19:28:05.572Z INFO controller_psmdb Waiting for the pods {"replset": "rs0", "size": 3, "pods": 2} │
│ 2023-01-20T19:28:41.484Z INFO controller_psmdb Cluster state changed {"previous": "initializing", "current": "ready"} │
│ 2023-01-20T19:28:41.691Z INFO controller_psmdb Point-in-time recovery will work only with full backup. Please create one manually or wait for scheduled backup to be created (if configured). │
│ 2023-01-20T19:28:46.689Z INFO controller_psmdb Point-in-time recovery will work only with full backup. Please create one manually or wait for scheduled backup to be created (if configured). │
│ 2023-01-20T19:28:51.883Z INFO controller_psmdb Point-in-time recovery will work only with full backup. Please create one manually or wait for scheduled backup to be created (if configured).                                                                                                                                                                                   

```

Here is the config for backup option in PSMDB custom resource:

```auto
  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.0.3
    pitr:
      compressionLevel: 6
      compressionType: gzip
      enabled: true
      oplogSpanMin: 10
    serviceAccountName: psmdb-backup-agent
    storages:
      google-cloud-storage-s3:
        s3:
          bucket: percona-backups
          credentialsSecret: google-cloud-storage-s3-backup
          endpointUrl: https://storage.googleapis.com/
          prefix: mongodb-of/
          region: europe-west-1
        type: s3

```

I read through this doc [Backup and restore - Percona Operator for MongoDB](https://docs.percona.com/percona-operator-for-mongodb/backups.html#backups-on-amazon-s3-or-s3-compatible-storage) to setup the backups.

So, is there something missing here in the storage config to connect to the GCS bucket?  
Can someone please take a look and see how can this be resolved?  
Thank you.

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [January 23, 2023, 3:00pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/2 "2023-01-23T15:00:04Z")

</div>

Hey @mbihl ,

I don’t see anything wrong with your configuration.  
Have you tried to execute a manual backup? Do you see additional hints about issues in backup-agent container logs?  
Anything in psmdb-backup?

---

<div class="post-metadata">

**Author:** ![mbihl](https://avatars.discourse-cdn.com/v4/letter/m/bc79bd/32.png) [@mbihl](https://forums.percona.com/u/mbihl)\
**Post date:** [January 23, 2023, 6:10pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/3 "2023-01-23T18:10:37Z")

</div>

@Sergey_Pronin I did try to execute a manual backup and it didn’t work.  
Here are logs from back-agent container.

```auto
│ 2023-01-23T18:06:11.000+0000 I got command backup [name: 2023-01-23T18:06:10Z, compression: gzip (level: default)] <ts: 1674497170> │
│ 2023-01-23T18:06:11.000+0000 I got epoch {1674497158 7} │
│ 2023-01-23T18:06:11.000+0000 D [backup/2023-01-23T18:06:10Z] init backup meta │
│ 2023-01-23T18:06:11.000+0000 D [backup/2023-01-23T18:06:10Z] nomination list for rs0: [] │
│ 2023-01-23T18:06:13.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated. │
│ For verbose messaging see aws.Config.CredentialsChainVerboseErrors │
│ 2023-01-23T18:06:18.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated. │
│ For verbose messaging see aws.Config.CredentialsChainVerboseErrors │
│ 2023-01-23T18:06:23.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: NoCredentialProviders: no valid providers in chain. Deprecated. │
│ For verbose messaging see aws.Config.CredentialsChainVerboseErrors │
│ 2023-01-23T18:06:26.000+0000 D [backup/2023-01-23T18:06:10Z] nomination timeout │
│ 2023-01-23T18:06:26.000+0000 D [backup/2023-01-23T18:06:10Z] skip after nomination, probably started by another node    

```

The psmdb-backup has an `error` status.  
 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/9/9631d15d507f55f7cdcc61d43f45dbeaf070ce3c.png)

For the secret to access the storage bucket, I have used key names as `GCP_ACCESS_KEY_ID` and `GCP_SECRET_ACCESS_KEY`, so are the name for the keys correct as I’m trying to use GCS bucket?

---

<div class="post-metadata">

**Author:** ![mbihl](https://avatars.discourse-cdn.com/v4/letter/m/bc79bd/32.png) [@mbihl](https://forums.percona.com/u/mbihl)\
**Post date:** [January 23, 2023, 9:07pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/4 "2023-01-23T21:07:27Z")

</div>

I think the key names should use `AWS` instead of `GCP`,so I changed those to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` and now the error message is different.

```auto
│ 2023-01-23T20:48:56.000+0000 I got epoch {1674506923 2} │
│ 2023-01-23T20:48:56.000+0000 D [backup/2023-01-23T20:48:56Z] init backup meta │
│ 2023-01-23T20:48:56.000+0000 D [backup/2023-01-23T20:48:56Z] nomination list for rs0: [] │
│ 2023-01-23T20:48:59.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed │
│ caused by: Head "https://storage.googleapis.com/percona-backups-mi9-tst/mongodb-of/.pbm.init": net/http: invalid header field value for "Authorization" │
│ 2023-01-23T20:49:04.000+0000 E [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed │
│ caused by: Head "https://storage.googleapis.com/percona-backups-mi9-tst/mongodb-of/.pbm.init": net/http: invalid header field value for "Authorization"

```

How can I check the header field value?

---

<div class="post-metadata">

**Author:** ![mbihl](https://avatars.discourse-cdn.com/v4/letter/m/bc79bd/32.png) [@mbihl](https://forums.percona.com/u/mbihl)\
**Post date:** [January 25, 2023, 2:27pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/5 "2023-01-25T14:27:11Z")

</div>

The issue was that there was a newline in the encoded secret value and was fixed.  
This is resolved now and PBM is working with GCS bucket.

Thanks

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [January 26, 2023, 12:03pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/6 "2023-01-26T12:03:57Z")

</div>

@mbihl glad you figured it out! Let me know if there is anything I can help with.

---

<div class="post-metadata">

**Author:** ![Srinivasulu\_Kota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/srinivasulu_kota/32/9886_2.png) [@Srinivasulu\_Kota](https://forums.percona.com/u/Srinivasulu_Kota)\
**Post date:** [March 27, 2023, 2:59pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/7 "2023-03-27T14:59:10Z")

</div>

Hi , I want to know if we can use workload identify for the service account operator instead of credentialsSecret on Google platform

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [March 28, 2023, 12:28pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/8 "2023-03-28T12:28:20Z")

</div>

Currently workload identity is not supported in the Operator.

We have it in our roadmap - [[K8SPSMDB-410] Add support for Workload Identity Authentication · Issue #26 · percona/roadmap · GitHub](https://github.com/percona/roadmap/issues/26)

Feel free to upvote it and we’ll see how to prio it better.

---

<div class="post-metadata">

**Author:** ![Srinivasulu\_Kota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/srinivasulu_kota/32/9886_2.png) [@Srinivasulu\_Kota](https://forums.percona.com/u/Srinivasulu_Kota)\
**Post date:** [March 31, 2023, 3:42pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/9 "2023-03-31T15:42:30Z")

</div>

Thank you . Please let me know what goes in `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` for Google storage access for backup.

I have the Json similar to below .  
{  
“auth\_provider\_x509\_cert\_url”:“[https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs)”,  
“auth\_uri”:“[Sign in - Google Accounts](https://accounts.google.com/o/oauth2/auth)”,  
“client\_email”:“,  
“client\_id”:”“,  
“client\_x509\_cert\_url”:”“,  
“private\_key”:”“,  
“private\_key\_id”:”“,  
“project\_id”:”",  
“token\_uri”:“[https://oauth2.googleapis.com/token](https://oauth2.googleapis.com/token)”,  
“type”:“service\_account”  
}

---

<div class="post-metadata">

**Author:** ![mbihl](https://avatars.discourse-cdn.com/v4/letter/m/bc79bd/32.png) [@mbihl](https://forums.percona.com/u/mbihl)\
**Post date:** [March 31, 2023, 7:50pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/10 "2023-03-31T19:50:03Z")

</div>

HMAC keys need to be created for the service account to access GCS bucket and the **AccessId** and **SecretKey** values generated will go in `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` values in the k8s secret.

> **[Manage HMAC keys for service accounts  |  Cloud Storage...](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create)**

---

<div class="post-metadata">

**Author:** ![Srinivasulu\_Kota](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/srinivasulu_kota/32/9886_2.png) [@Srinivasulu\_Kota](https://forums.percona.com/u/Srinivasulu_Kota)\
**Post date:** [February 5, 2025, 12:16pm UTC](https://forums.percona.com/t/kubernetes-pbm-not-working-with-gcs-bucket/19700/11 "2025-02-05T12:16:18Z")

</div>

any further update on newer version of PBM using workload identify for google cloud storage hosted on gke .