# Error while creating backup: failed to find CERTIFICATE

**URL:** <https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101>\
**Category:** Percona Operator for MongoDB\
**Tags:** mongodb, kubernetes, pbm\
**Created:** [June 26, 2023, 2:10am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101 "2023-06-26T02:10:27Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Semantic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/semantic/32/9104_2.png) [@Semantic](https://forums.percona.com/u/Semantic)\
**Post date:** [June 26, 2023, 2:10am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/1 "2023-06-26T02:10:27Z")

</div>

While creating a normal logical backup of a 1.3TB replica, its state changes to errored after about 16 hours of run with the following:

```auto
check for concurrent jobs: getting pbm object: create PBM connection to mongo-rs0-2.mongo-rs0.mongo.svc.cluster.local:27017,mongo-rs0-0.mongo-rs0.mongo.svc.cluster.local:27017,mongo-rs0-1.mong-rs0.mongo.svc.cluster.local:27017: create mongo connection: create mongo client: failed to find CERTIFICATE

```

The backup job however still progresses and completes, but I’m unsure as to the validity of the backup it creates (and the fact that I doubt it’s possible to restore from that unless manually changing its status to success)

Operator 1.13.0

---

<div class="post-metadata">

**Author:** ![aporrinali](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/aporrinali/32/8492_2.png) [@aporrinali](https://forums.percona.com/u/aporrinali)\
**Post date:** [July 13, 2023, 10:34am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/2 "2023-07-13T10:34:23Z")

</div>

+1 same issue, but with a much lighter base (1GB)

```auto
create pbm object: create PBM connection to prod-mongodb-rs0-0.prod-mongodb-rs0.mongo.svc.cluster.local:27017,prod-mongodb-rs0-1.prod-mongodb-rs0.mongo.svc.cluster.local:27017,prod-mongodb-rs0-2.prod-mongodb-rs0.mongo.svc.cluster.local:27017: create mongo connection: create mongo client: failed to find CERTIFICATE

```

PSMDB Operator 1.14.0

Would you need any additional info, please ask

---

<div class="post-metadata">

**Author:** ![Semantic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/semantic/32/9104_2.png) [@Semantic](https://forums.percona.com/u/Semantic)\
**Post date:** [July 14, 2023, 1:46am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/3 "2023-07-14T01:46:04Z")

</div>

I think it’s possible for it to be related to [this](https://forums.percona.com/t/intermittent-backup-failures/22362/6), and as @Sergey_Pronin says, the backups are good but indicating a wrong failed status. I haven’t tried restoring from those failed ones though.

---

<div class="post-metadata">

**Author:** ![ki0](https://avatars.discourse-cdn.com/v4/letter/k/5f9b8f/32.png) [@ki0](https://forums.percona.com/u/ki0)\
**Post date:** [July 26, 2023, 11:10am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/4 "2023-07-26T11:10:27Z")

</div>

Hi there,

Same problem here and in my case, it’s not related to [this](https://forums.percona.com/t/intermittent-backup-failures/22362/6). I think it is a problem with deploy certificates, i have two environments and i’ve done an operator upgrade from 1.9.0 to 1.14.0, in development it is working as expected, i got backups in S3 but in production, i got this error. Comparing both environments, i’ve only seen that the certificates doesn’t upgrade properly like in development, i say this because in ArgoCD the objects of each app are different, not the same result. In development, i see two secret objects for certificates but they aren’t in production so i’m guessing something is happening there.

Cheers.

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [August 3, 2023, 8:28am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/5 "2023-08-03T08:28:00Z")

</div>

@Semantic @aporrinali is it happening to any backup resource?

---

<div class="post-metadata">

**Author:** ![Semantic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/semantic/32/9104_2.png) [@Semantic](https://forums.percona.com/u/Semantic)\
**Post date:** [August 5, 2023, 4:31pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/6 "2023-08-05T16:31:44Z")

</div>

Hello @Sergey_Pronin!

What do you mean by a resource in this case?

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [August 7, 2023, 7:25am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/7 "2023-08-07T07:25:39Z")

</div>

@Semantic I’m talking about `psmdb-backup`. In other words - if you try manual backup through creating a psmdb-backup resource - is it always erroring out or there is a chance that it goes through?

---

<div class="post-metadata">

**Author:** ![Semantic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/semantic/32/9104_2.png) [@Semantic](https://forums.percona.com/u/Semantic)\
**Post date:** [August 7, 2023, 5:53pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/8 "2023-08-07T17:53:40Z")

</div>

Correct my backups are only done manually at the moment. Some clusters happen to work, some don’t even though I retried them a couple times.

---

<div class="post-metadata">

**Author:** ![Nils\_Dehn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nils_dehn/32/13629_2.png) [@Nils\_Dehn](https://forums.percona.com/u/Nils_Dehn)\
**Post date:** [November 2, 2023, 9:35pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/9 "2023-11-02T21:35:55Z")

</div>

The problem persists with operator 1.15.0 and pbm 2.3.0 - at least for manual backups.  
The backup container completed the backup but backup object is in failed state.

Trying cron backups next.

---

<div class="post-metadata">

**Author:** ![Claudio\_Bisegni](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/claudio_bisegni/32/13488_2.png) [@Claudio\_Bisegni](https://forums.percona.com/u/Claudio_Bisegni)\
**Post date:** [November 9, 2023, 5:10pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/10 "2023-11-09T17:10:50Z")

</div>

hi all, i also got this error on the PSMDB-backup resource but seeing the mongod backup process seem that it’s ok. I run 1.15 operator and mongodb backup 2.3.0

---

<div class="post-metadata">

**Author:** ![m.vasilenko](https://avatars.discourse-cdn.com/v4/letter/m/a5b964/32.png) [@m.vasilenko](https://forums.percona.com/u/m.vasilenko)\
**Post date:** [April 16, 2024, 6:34am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/11 "2024-04-16T06:34:30Z")

</div>

Hi,  
Same issue in my case, after two hours backup was successfully created and saved to S3 storage but Backup resource was switched to the error state after ~30 minutes.

Operator image: `percona/percona-server-mongodb-operator:1.15.0`  
Backup image: `perconalab/percona-server-mongodb-operator:main-backup`

Backup resource:

```auto
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  creationTimestamp: '2024-04-16T00:00:00Z'
  finalizers:
    - delete-backup
  generateName: cron-mongodb-20240416000000-
  generation: 1
  labels:
    ancestor: daily-minio
    cluster: mongodb
    type: cron
  name: cron-mongodb-20240416000000-kt5d7
  namespace: mongodb
  resourceVersion: '500059694'
  uid: ef16e0a0-91c5-4e3e-a363-267446d37328
spec:
  clusterName: mongodb
  compressionLevel: 6
  compressionType: gzip
  storageName: minio
  type: logical
status:
  destination: s3://mongodb-backups/2024-04-16T00:00:21Z
  error: >-
    create pbm object: create PBM connection to
    mongodb-rs-0.mongodb-rs.mongodb.svc.cluster.local:27017,mongodb-rs-1.mongodb-rs.mongodb.svc.cluster.local:27017,mongodb-rs-2.mongodb-rs.mongodb.svc.cluster.local:27017:
    create mongo connection: create mongo client: failed to find CERTIFICATE
  lastTransition: '2024-04-16T00:00:23Z'
  pbmName: '2024-04-16T00:00:21Z'
  pbmPod: mongodb-rs-2
  replsetNames:
    - rs
  s3:
    bucket: mongodb-backups
    credentialsSecret: mongodb-backup-s3
    endpointUrl: https://s3-endpoint:9000
    serverSideEncryption: {}
  start: '2024-04-16T00:00:21Z'
  state: error
  storageName: minio
  type: logical

```

---

<div class="post-metadata">

**Author:** ![Semantic](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/semantic/32/9104_2.png) [@Semantic](https://forums.percona.com/u/Semantic)\
**Post date:** [May 20, 2024, 5:17pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/12 "2024-05-20T17:17:51Z")

</div>

As mentioned previously, those backups actually succeed but are marked with a false-positive error state. I believe that since a real solution would involve codebase changes to this certificate error from surfacing in the first place, here’s a not-so-ideal one, but one you can use right now:

You should install the [edit-status](https://github.com/ulucinar/kubectl-edit-status) kube plugin and edit the state of whatever backup you’d like to use.

```
kubectl edit-status psmdb-backup NAME

```

Update the `status.state` field from `error` to `ready`, which would allow you to use it for a restore.

---

<div class="post-metadata">

**Author:** ![Bartosz\_Dabrowski](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/bartosz_dabrowski/32/16520_2.png) [@Bartosz\_Dabrowski](https://forums.percona.com/u/Bartosz_Dabrowski)\
**Post date:** [June 27, 2024, 6:09am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/13 "2024-06-27T06:09:09Z")

</div>

Hi, as the certificate issue is still persistent, are there any hopes of having this topic on the roadmap for any of the upcoming releases?

---

<div class="post-metadata">

**Author:** ![Bartosz\_Dabrowski](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/bartosz_dabrowski/32/16520_2.png) [@Bartosz\_Dabrowski](https://forums.percona.com/u/Bartosz_Dabrowski)\
**Post date:** [November 28, 2024, 8:29am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/14 "2024-11-28T08:29:53Z")

</div>

Bump, does anyone look into that? Backups are unusable in the current state 😕

---

<div class="post-metadata">

**Author:** ![m.vasilenko](https://avatars.discourse-cdn.com/v4/letter/m/a5b964/32.png) [@m.vasilenko](https://forums.percona.com/u/m.vasilenko)\
**Post date:** [November 28, 2024, 8:49am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/15 "2024-11-28T08:49:07Z")

</div>

I had to write a python service to extract backup statuses directly from pbm-container via command `pbm status -o json`, and based on that update Backup resources states in Kubernetes. Tricky, but already works for half a year.

---

<div class="post-metadata">

**Author:** ![radoslaw.szulgo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/radoslaw.szulgo/32/17315_2.png) [@radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)\
**Post date:** [January 20, 2025, 9:06am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/16 "2025-01-20T09:06:06Z")

</div>

Hey all,  
thank you for your patience. I’ve submitted a Jira ticket for that issue:  
[https://perconadev.atlassian.net/browse/K8SPSMDB-1263](https://perconadev.atlassian.net/browse/K8SPSMDB-1263)

Hope our team can address that in the next version.

---

<div class="post-metadata">

**Author:** ![Adhiraj\_Singh](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/adhiraj_singh/32/19891_2.png) [@Adhiraj\_Singh](https://forums.percona.com/u/Adhiraj_Singh)\
**Post date:** [January 22, 2025, 10:45am UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/17 "2025-01-22T10:45:15Z")

</div>

Got the same issue, preventing us from rolling Percona MongoDB into production. Is there a temporary workaround? (Using the latest 1.19.0 operator)

> [@Error while creating backup: failed to find CERTIFICATE](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/12):
>
> As mentioned previously, those backups actually succeed but are marked with a false-positive error state. I believe that since a real solution would involve codebase changes to this certificate error from surfacing in the first place, here’s a not-so-ideal one, but one you can use right now: You should install the [edit-status](https://github.com/ulucinar/kubectl-edit-status) kube plugin and edit the state of whatever backup you’d like to use. kubectl edit-status psmdb-backup NAME Update the status.state field from error to ready, which would…

This doesn’t seem to work, because the backup metadata isn’t uploaded to S3

---

<div class="post-metadata">

**Author:** ![radoslaw.szulgo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/radoslaw.szulgo/32/17315_2.png) [@radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)\
**Post date:** [January 29, 2025, 2:37pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/18 "2025-01-29T14:37:51Z")

</div>

We plan to fix it in 1.20 version of operator.

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [February 27, 2025, 8:56pm UTC](https://forums.percona.com/t/error-while-creating-backup-failed-to-find-certificate/23101/19 "2025-02-27T20:56:16Z")

</div>

@Adhiraj_Singh @Semantic @m.vasilenko @Bartosz_Dabrowski @Semantic @Nils_Dehn @ki0

We are working on fixing this issue: [PR](https://github.com/percona/percona-server-mongodb-operator/pull/1838). The main challenge is that we haven’t been able to reproduce it on our end.

Could you help us test our fix? You can use the `main` branch for testing, but please avoid using it for production traffic, only for testing purposes.

Let us know if you encounter any issues or need assistance. Thanks!