# PBM: GCS storage not working as intended in 2.5.0 **URL:** \ **Category:** Percona Backup for MongoDB\ **Created:** [August 7, 2024, 9:21am UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103 "2024-08-07T09:21:03Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [August 7, 2024, 9:21am UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/1 "2024-08-07T09:21:03Z")
Hello, We are using PBM 2.5.0 (upgraded from 1.6.1) for Debian and trying to setup the storage from local to GCS (Google cloud Storage) but we have encountered a few issues. First, the documentation is not clear at all for GCS: [Configure remote backup storage - Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/install/backup-storage.html) doesn’t have any reference for GCS When we tried to setup the remote storage to GCS, we setup a service account with the Storage Admin permissions (highest level possible for GCS) and we got the following errors : from pbm-status : ```auto [S]: pbm-agent v2.5.0 FAILED status: > ERROR with storage: storage check failed with: get S3 object header: Forbidden: Forbidden status code: 403, request id: , host id: ``` From pbm-logs: ```auto [agentCheckup] check storage connection: storage check failed with: get S3 object header: Forbidden: Forbidden status code: 403, request id: , host id: ``` On all nodes of our clusters and on both pbm-agent (27019) and pbm-agent-data (27018) We tried using gcloud cli by using the same account or another one and impersonating the service account and we got no error uploading a file. We also tried to change to another bucket and it doesn’t change anything. The service account also got full admin storage rights on the said bucket. We also tried to change our bucket from multi region part to only one region still the same errors. The bucket is empty and we know that pbm need to create the .pbm.init and it should be able to do so with the permissions of the service account. Here is our config file for pbm : ```auto storage: type: s3 s3: provider: gcs region: eu bucket: bucket_name endpointUrl: https://storage.googleapis.com credentials: access-key-id: " *******" secret-access-key: "-----BEGIN PRIVATE KEY----- *******" ``` I also have a question, why the url shown by `pbm-status` for GCS begin with S3 and is formatted like that ? : `s3://https://storage.googleapis.com/` Note that we didn’t change the config file between 1.6.1 and 2.5.0. The local storage and logical cluster backup are working fine. Can someone help us ? Thank you ---
**Author:** ![radoslaw.szulgo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/radoslaw.szulgo/32/17315_2.png) [@radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)\ **Post date:** [August 7, 2024, 11:32am UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/2 "2024-08-07T11:32:34Z")
Hi @craynaud , Look at the example here: [Configure remote backup storage - Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/install/backup-storage.html) : ```auto storage: type: s3 s3: region: us-east1 bucket: pbm-testing prefix: pbm/test endpointUrl: https://storage.googleapis.com credentials: access-key-id: secret-access-key: ``` The provider is automatically defined depending on the `endpointUrl`. Please remove “provider: gcs” from your configuration. I believe this may cause it cannot be set correctly. ---
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [August 7, 2024, 12:32pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/3 "2024-08-07T12:32:21Z")
Hello @radoslaw.szulgo , Thank you for your answer 🙂 I have tested it but i still got the same error and if I check my config file it’s correctly set to : ```auto type: s3 s3: region: eu bucket: bucket_name endpointUrl: https://storage.googleapis.com credentials: access-key-id: " ******" secret-access-key: "-----BEGIN PRIVATE KEY-----***" ``` But in the running config the provider still appear : ```auto # pbm config pitr: enabled: false oplogSpanMin: 0 compression: gzip compressionLevel: -1 storage: type: s3 s3: provider: gcs region: eu endpointUrl: https://storage.googleapis.com forcePathStyle: true bucket: bucket_name credentials: access-key-id: '***' secret-access-key: '***' maxUploadParts: 10000 storageClass: STANDARD insecureSkipTLSVerify: false backup: oplogSpanMin: 0 priority: [....] ``` Is it ok ? Does the problem can be because of the ` region: eu` ? Since it’s multi region bucket , i’ve set it up to simply eu Also, i’ve put the `provider: gcs` parameter in the config file, because the documentation mention it here : [Remote backup storage options - Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/reference/configuration-options.html#storages3provider) Regards, ---
**Author:** ![radoslaw.szulgo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/radoslaw.szulgo/32/17315_2.png) [@radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)\ **Post date:** [August 7, 2024, 12:41pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/4 "2024-08-07T12:41:02Z")
I think you should use a specific region like “EUROPE-WEST3”. ---
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [August 7, 2024, 12:57pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/5 "2024-08-07T12:57:53Z")
I’ve tried 2 different ways with the same results: 1 - using a specific region in the config file with multi region bucket = same errors 2 - using a specific region in the config file with a regional bucket on the same region as in the config file = same errors And my service account is still Storage Admin on both buckets Regards, ---
**Author:** ![Sandra\_Romanchenko](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sandra_romanchenko/32/84_2.png) [@Sandra\_Romanchenko](https://forums.percona.com/u/Sandra_Romanchenko)\ **Post date:** [August 15, 2024, 9:12pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/6 "2024-08-15T21:12:18Z")
Hi @craynaud, I’ve noticed that your secret-access-key starts with `"-----BEGIN PRIVATE KEY-----"`, it’s not exactly the expected format. Can you please check the docs below? You can find there a sample how access ID and secret should look like and the way to create them: > **[HMAC keys  |  Cloud Storage  |  Google Cloud](https://cloud.google.com/storage/docs/authentication/hmackeys)** > **[Manage HMAC keys for service accounts  |  Cloud Storage...](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#console)** ---
**Author:** ![radoslaw.szulgo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/radoslaw.szulgo/32/17315_2.png) [@radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)\ **Post date:** [September 2, 2024, 10:16am UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/7 "2024-09-02T10:16:08Z")
@craynaud any news from you? Have you resolved the issue? ---
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [September 2, 2024, 12:58pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/8 "2024-09-02T12:58:33Z")
Hello, I’m checking it just now, I was on vacation 🙂 I have a question though, i’ve setup a key on the service account from the IAM interface which is the one i’m using currently and the one that begin with `"-----BEGIN PRIVATE KEY-----"`, why can’t we use that one and have to create a specific HMAC Key on GCS for the service account ? I mean is there any technical reasons behind that ? Also, it seems that the documentation related to GCS is not quite on point because this is not specified in it for the moment neither is any explanation on why. Best regards, ---
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [September 12, 2024, 1:35pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/9 "2024-09-12T13:35:01Z")
Hello, I’ve tested your solution but i still got 403 from GCS and without proper documentation I don’t even know what to put in the “acces-key-id” line , is it the service account email now ? On the secret-acces-key i’ve put the hmac key like you told me. And another question, do you fully support multiregion bucket or only monoregion bucket ? Regards, ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 12, 2024, 1:46pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/10 "2024-09-12T13:46:42Z")
Hi, HMAC keys are typically used to authenticate against a service. The HMAC key has two parts: 1. Access ID 2. Secret That is an example of what you have to put in the section: ```auto credentials: access-key-id: 'GOOGTS7C7FUP3AIRVJTE2BCDKINBTES3HC2GY5CBFJDCQ2SYHV6A6XXVTJFSA' secret-access-key: 'bGoa+V7g/yqDXvKRqq+JTFn4uQZbPiQJo4pf9RzJ' ``` This is explained in more detail in the documentation linked above. Hope that helps ---
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [September 12, 2024, 1:56pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/11 "2024-09-12T13:56:23Z")
Hi, Yes sorry about that i saw this after and using terraform to create the hmac\_key i’m not sure i can retrieve the secret, i’ll check that ! And what about multiregion bucket ? ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 12, 2024, 2:09pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/12 "2024-09-12T14:09:14Z")
You should be able to retrieve the secret using `terraform output -json` Re: multi region bucket try putting `EU` in your case. Let me know how that goes. ---
**Author:** ![craynaud](https://avatars.discourse-cdn.com/v4/letter/c/ecccb3/32.png) [@craynaud](https://forums.percona.com/u/craynaud)\ **Post date:** [September 12, 2024, 3:56pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/13 "2024-09-12T15:56:38Z")
Hello, It works 🙂 also setting EU as the region value works for multiregion bucket ! Maybe please write in the documentation that for gcs we need hmac key and not a iam service account key Thank you ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 12, 2024, 4:49pm UTC](https://forums.percona.com/t/pbm-gcs-storage-not-working-as-intended-in-2-5-0/32103/14 "2024-09-12T16:49:56Z")
glad to hear it works. We are open to contributions to improve our docs. Feel free to submit a PR [Sign in to GitHub · GitHub](https://github.com/percona/pbm-docs/edit/main/docs/install/backup-storage.md)