GCS backup failure: private key should be a PEM or plain PKCS1 or PKCS8

MongoDB Percona Operator for MongoDB
1

Description:

Hi,

I’m trying to perform backups using GCS. I’m using operator 1.21.0 and following the instructions here:

However the backups fail with this message: “private key should be a PEM or plain PKCS1 or PKCS8” (see logs below).

The error comes from Google SDK:

I’m fairly confident my private key was a valid RSA PKCS8 PEM-encoded key. I tried to provide the key with and without PEM headers, with and without newlines, nothing worked.

Steps to Reproduce:

  1. Setup service accounts and create an GKE cluster
  2. Create a secret with GCS_CLIENT_EMAIL and GCS_PRIVATE_KEY as explained in the docs
  3. Create a CR with:
  backup:
    enabled: true
    storages:
      my-backups:
        gcs:
          bucket: my-bucket
          credentialsSecret: my-secret
          prefix: path/to/backups
        main: true
        type: gcs

Version:

MongoDB Operator: 1.21.0

PBM: 2.11.0

Logs:

PBM agent logs:

check failed with: file stat: list with parts for mw file stat op: fetching pbm file parts base: get properties: Get "https://storage.googleapis.com/storage/v1/b/.../.pbm.init?alt=json&prettyPrint=false&projection=full": private key should be a PEM or plain PKCS1 or PKCS8: asn1: structure error: tags don't match (16 vs {class:0 tag:13 length:45 isCompound:true}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2

Expected Result:

Backup success.

Actual Result:

Backup failed.

Additional Information:

None

2

UPDATE: my bad, the private key string was being fetched from a secret and was in single-line form with \n escape sequences, and these where undistinguishable from real line breaks in my text editor. When I replaced the escape sequences with actual line breaks, the key was successfully decoded.

1 Like