PBM backup not connectiong to GCS error Forbidden

MongoDB Percona Backup for MongoDB
1

I need to have server:8.0.8-3 (since I need to join an existing replicaset which runs version 8.0.8)

using percona backup 2.10.0

my credentials are like this:

kubectl create secret generic gcs-credentials -n percona \
  --from-literal=AWS_ACCESS_KEY_ID="access_key_from_settings_interopability" \
  --from-literal=AWS_SECRET_ACCESS_KEY="access_key"

the pbm config says this:

storage:
  type: s3
  s3:
    region: auto
    endpointUrl: https://storage.googleapis.com
    forcePathStyle: true
    bucket: backups
    credentials:
      access-key-id: '***'
      secret-access-key: '***'
    maxUploadParts: 10000
    storageClass: STANDARD
    insecureSkipTLSVerify: false
pitr:
  enabled: true
  compression: s2
backup:
  oplogSpanMin: 0
  compression: s2
restore: {}

Cant get the pbm to work on gke and gcs! always get this:

2025-09-06T11:12:18.000+0000 E [agentCheckup] check storage connection: storage check failed with: file stat: get S3 object header: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: , HostID: , api error Forbidden: Forbidden

here is my entire cluster.yaml

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: percona-cluster
  namespace: percona
spec:
  image: percona/percona-server-mongodb:8.0.8-3
  allowUnsafeConfigurations: true
  secrets:
    keyFile: percona-keyfile
  tls:
    mode: disabled
  replsets:
    - name: rs0
      size: 3
      tolerations:
        - key: "role"
          operator: "Equal"
          value: "database"
          effect: "NoSchedule"
      volumeSpec:
        persistentVolumeClaim:
          storageClassName: "percona-storage"
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 300Gi
      expose:
        enabled: true
        exposeType: ClusterIP
  backup:
    image: percona/percona-backup-mongodb:2.10.0
    enabled: true
    pitr:
      compressionLevel: 6
      compressionType: gzip
      enabled: true
      oplogSpanMin: 60
    serviceAccountName: percona-backup
    storages:
      gcs-storage:
        type: s3
        s3:
          bucket: backups
          credentialsSecret: gcs-credentials
          region: europe-west3
          endpointUrl: https://storage.googleapis.com
    tasks:
      - name: "daily-backup"
        enabled: true
        schedule: "0 0 * * *"
        keep: 7
        type: logical
        storageName: gcs-storage

      - name: "weekly-backup"
        enabled: true
        schedule: "0 0 * * 0"
        keep: 4
        type: logical
        storageName: gcs-storage

      - name: "monthly-backup"
        enabled: true
        schedule: "0 0 1 * *"
        keep: 12
        type: logical
        storageName: gcs-storage

      - name: "yearly-backup"
        enabled: true
        schedule: "0 0 1 1 *"
        keep: 10
        type: logical
        storageName: gcs-storage
2

After a lot of tries, it seems newer versions are not alwaysthe best!

I made it work, by selecting pbm 2.9.1.

Now into PMM and getting that to work.

3

Hey Fetah,

A few things to check:

  • Make sure your GCS service account has HMAC keys (not just JSON credentials).
  • Remove forcePathStyle: true—GCS doesn’t support it.
  • Double-check IAM roles on the bucket (Storage Object Admin usually works).
  • If you’re on PBM 2.10.0, try 2.9.1 as you did—there might be a regression.

also try generating HMAC keys or tweaking the config.

4

Hi @Fetah_Jusufi - there’ was a change in PBM 2.10.0 in the configuration of GCS - see: Percona Backup for MongoDB 2.10.0 (2025-06-26) - Percona Backup for MongoDB

Note that PBM configuration for a backup storage now has a dedicated gcs subsection. If you’re upgrading to PBM 2.10.0 or later, you must update your backup configuration accordingly:

  1. Change the storage.type from s3 to gcs.

  2. Change the storage.s3 section to storage.gcs and adjust the parameters accordingly.

Read more about available options in the Google Cloud Storage (GCS) chapter.