# Percona MongoDB operator backup failure

**URL:** <https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366>\
**Category:** Percona Operator for MongoDB\
**Tags:** percona, mongodb\
**Created:** [September 8, 2022, 6:43am UTC](https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366 "2022-09-08T06:43:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiaoguang\_zhang](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/xiaoguang_zhang/32/7695_2.png) [@xiaoguang\_zhang](https://forums.percona.com/u/xiaoguang_zhang)\
**Post date:** [September 8, 2022, 6:43am UTC](https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366/1 "2022-09-08T06:43:51Z")

</div>

Hi all,

I’m using the helm chart psmdb-operator and psmdb-db under [percona-helm-charts/charts at main · percona/percona-helm-charts · GitHub](https://github.com/percona/percona-helm-charts/tree/main/charts) to deploy a sharded MongoDB.  
It was version 1.11.0 and now upgraded to 1.12.0, however, I can’t trigger a successful backup for both of them.

The backup image located on AWS ECR for pulling.  
The secret _mongodb-backup-s3_ is configured with AWS access id&key, the CRD is upgraded using [https://github.com/percona/percona-server-mongodb-operator/blob/v1.12.0/deploy/crd.yaml](https://github.com/percona/percona-server-mongodb-operator/blob/v1.12.0/deploy/crd.yaml).

And the backup section configuration like this:

```auto
backup:
  enabled: true
  image:
    repository: xxxxx.dkr.ecr.cn-northwest-1.amazonaws.com.cn/percona/percona-backup-mongodb
    tag: 1.8.1
  serviceAccountName: xxxx-mongodb-operator-psmdb-operator
  resources:
    limits:
      cpu: "300m"
      memory: "1G"
    requests:
      cpu: "100m"
      memory: "128Mi"
  storages:
    s3-cn-northwest:
      type: s3
      s3:
        bucket: mongodb-backup
        credentialsSecret: mongodb-backup-s3
# endpointUrl: s3.cn-northwest-1.amazonaws.com.cn
        region: cn-northwest-1
        prefix: ""
  pitr:
    enabled: false
    # oplogSpanMin: 10
  tasks:
  - name: daily-s3
    enabled: true
    schedule: "0 0 * * *"
    keep: 30
    storageName: s3-cn-northwest
    compressionType: gzip

```

The operator successfully proceed to create the cronjob and trigger the psmdb-backup accordingly, but there is no verbose event except for:

```auto
$ kubectl describe psmdb-backup xxx
Spec:
  Cluster Name: xxx-mongodb-psmdb-db
  Compression Type: gzip
  Storage Name: s3-cn-northwest
Status:
  Azure:
    Credentials Secret:
  Destination: 2022-09-08T02:58:59Z
  Error: starting deadline exceeded
  Last Transition: 2022-09-08T02:58:59Z
  Pbm Name: 2022-09-08T02:58:59Z
  s3:
    Bucket: mongodb-backup
    Credentials Secret: mongodb-backup-s3
    Endpoint URL: https://s3.cn-northwest-1.amazonaws.com.cn
    Region: cn-northwest-1
  Start: 2022-09-08T02:58:59Z
  State: error
  Storage Name: s3-cn-northwest
Events: <none>

```

The only hint I can see is the **Error: starting deadline exceeded** , what does this mean? I tried to configure the _storage.s3.endpointUrl_ to AWS China specifically yet failed.  
Is there anything I can leverage to troubleshoot?

Thanks!

---

<div class="post-metadata">

**Author:** ![Tomislav\_Plavcic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tomislav_plavcic/32/9717_2.png) [@Tomislav\_Plavcic](https://forums.percona.com/u/Tomislav_Plavcic)\
**Post date:** [September 19, 2022, 8:24am UTC](https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366/2 "2022-09-19T08:24:41Z")

</div>

Hi @xiaoguang_zhang !  
It seems that you are hitting this issue here: [[K8SPSMDB-660] backup error - starting deadline exceeded - Percona JIRA](https://jira.percona.com/browse/K8SPSMDB-660)  
Could you please upgrade to 1.13.0 and see if you are getting the same.  
1.13.0 was released last week and I expect that the helm chart will be available with this version in a week or so - so if you are only using helm chart you will maybe need to wait few days.  
Please follow up if it solves your issue or not.  
Thanks!

---

<div class="post-metadata">

**Author:** ![xiaoguang\_zhang](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/xiaoguang_zhang/32/7695_2.png) [@xiaoguang\_zhang](https://forums.percona.com/u/xiaoguang_zhang)\
**Post date:** [September 28, 2022, 9:39am UTC](https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366/3 "2022-09-28T09:39:26Z")

</div>

Hi @Tomislav_Plavcic !  
Thanks for your reply! I’ve tried the latest chart, the error still exists.  
First I manually upgrade the CRD and the release, it turns out the backup task scheduler fails to update the cronjob. Then I reinstall it from operator and db, no cronjob created…  
Interesting part is the log from operator prints out after I modified the task part, but the cronjob not created or updated.

```auto
2022-09-28T06:43:45.090Z INFO controller_psmdb deleting outdated backup job {"name": "daily"}
2022-09-28T06:47:36.276Z INFO controller_psmdb Creating or updating backup job {"name": "daily-s3", "schedule": "0 3 * * *"}

```

Then I run a on-demand backup with following, and the error shows the same **Error: starting deadline exceeded**

```auto
cat <<EOF | kubectl apply -f-
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  name: test
spec:
  clusterName: xxx-psmdb-db
  storageName: s3-cn-northwest
EOF

```

---

<div class="post-metadata">

**Author:** ![xiaoguang\_zhang](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/xiaoguang_zhang/32/7695_2.png) [@xiaoguang\_zhang](https://forums.percona.com/u/xiaoguang_zhang)\
**Post date:** [September 29, 2022, 8:19am UTC](https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366/4 "2022-09-29T08:19:04Z")

</div>

I see the operator 1.13.0 takes charge of the backup task instead of k8s cronjob, there is a pmsdb-backup created at the correct scheduled time yesterday.  
Then, only timeout error remaining, anything I missed or is there a way to print more verbose logs? I’ve configured the LOG\_LEVEL=DEBUG in env of the operator, but with no informative logs. Thanks!

---

<div class="post-metadata">

**Author:** ![xiaoguang\_zhang](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/xiaoguang_zhang/32/7695_2.png) [@xiaoguang\_zhang](https://forums.percona.com/u/xiaoguang_zhang)\
**Post date:** [October 18, 2022, 9:25am UTC](https://forums.percona.com/t/percona-mongodb-operator-backup-failure/17366/5 "2022-10-18T09:25:58Z")

</div>

New update.

Sorry I misconfigured the s3 secret for the backup, now it’s all good. Thanks!