# PBM 2.12 on MongoDB 8.0 Sharded Cluster – “storage is not initialized” + “get epoch: lookup: element not found”

**URL:** <https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822>\
**Category:** Percona Backup for MongoDB\
**Created:** [December 2, 2025, 11:38pm UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822 "2025-12-02T23:38:08Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akshayrg09](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Akshayrg09](https://forums.percona.com/u/Akshayrg09)\
**Post date:** [December 2, 2025, 11:38pm UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/1 "2025-12-02T23:38:08Z")

</div>

I have a sharded MongoDB 8.0 cluster with 2 mongos + 3 config nodes and 2 shard replica sets (3 nodes each).  
PBM 2.12 is installed on every mongod node and each PBM agent connects to its local mongod using the correct URI.

I applied PBM storage config separately to config RS, shard1 RS, and shard2 RS.  
`pbm config` is correct everywhere - filesystem storage (also tried with S3 compatible Blackblaze storage) is initialized and identical on all replica sets.

But when I run a backup through mongos:

```auto
pbm backup ...

```

I always get:

```auto
get backup metadata: context deadline exceeded

```

And in every PBM agent log (config + shards) I repeatedly see:

```auto
storage is not initialized
get epoch: lookup: element not found

```

Cluster is healthy, agents authenticate fine, all nodes are reachable, and nothing is blocked.

I can’t understand why PBM thinks “storage is not initialized” when the storage config is already present on all replica sets.

I need help understanding why PBM cannot create the epoch or initialize storage metadata on MongoDB 8.0.

That’s the issue. Any leads will be appreciated!  
Thank you for your time.

---

<div class="post-metadata">

**Author:** ![radoslaw.szulgo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/radoslaw.szulgo/32/17315_2.png) [@radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)\
**Post date:** [December 3, 2025, 10:42am UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/2 "2025-12-03T10:42:03Z")

</div>

> [@Akshayrg09](#):
>
> I applied PBM storage config separately to config RS, shard1 RS, and shard2 RS

Hi and welcome to our forum!

PBM requires a shared storage (all nodes need to see the same volume or bucket) please check the steps to [add a remote storage](https://docs.percona.com/percona-backup-mongodb/install/backup-storage.html#procedure) for more details.

---

<div class="post-metadata">

**Author:** ![Akshayrg09](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Akshayrg09](https://forums.percona.com/u/Akshayrg09)\
**Post date:** [December 9, 2025, 11:16pm UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/3 "2025-12-09T23:16:55Z")

</div>

Thank you for your response.@**[radoslaw.szulgo](https://forums.percona.com/u/radoslaw.szulgo)**

The storage prefix issue is now resolved, and PBM storage is successfully initialized on the remote backend. However, the backup is still failing due to an authorization error that occurs only on the Config Server PRIMARY.

From the Config Primary PBM agent logs, we consistently see the following error during every backup attempt:

```auto
I got command backup [name: 2025-12-09T21:35:22Z, compression: s2 (level: default)] <ts: 1765316122>, opid: 6938961a63b09adf1c82e4e4
 I got epoch {1765315340 7}
 [backup/2025-12-09T21:44:39Z] get balancer status: run mongo command: (Unauthorized) not authorized on admin to execute command { _configsvrBalancerStatus: 1, lsid: { id: UUID(**") }, $clusterTime: { clusterTime: Timestamp(1765316680, 15), signature: { hash: BinData(0,**), keyId: *** } }, $db: "admin" }

```

This error appears only on the Config Server Primary.  
On the Shard replica sets (Shard1 and Shard2), PBM agents authenticate and operate without any authorization errors. The original “storage is not initialized” errors were coming from other nodes earlier and are now resolved after fixing the prefix and re-initializing storage.

The MongoDB user used by PBM is defined as follows:

```auto
{
  _id: 'admin.**',
  userId: **
  user: '**',
  db: 'admin',
  roles: [{ role: 'root', db: 'admin' }],
  mechanisms: ['SCRAM-SHA-1', 'SCRAM-SHA-256']
}

```

Despite the user having the root role on the admin database, PBM is still unable to execute `_configsvrBalancerStatus` on the Config Server Primary, which causes the backup to fail with:

```auto
get backup metadata: context deadline exceeded

```

Other all nodes logs:

![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/6/5/65fc658e31e8c522434aaf4535f89f232fa84a79.png)

Cluster Topology:

- 2 × mongos

- 3 × config server nodes (config replica set)

- 2 × shards (each a replica set)

- 1 × HAProxy

Environment:

- PBM version: 2.12

- MongoDB version: 8.0 (sharded cluster)

At this point:

- Storage is fully initialized

- PBM agents authenticate successfully on all nodes

- Shard nodes work without auth issues

- Only the Config Server Primary throws the unauthorized error for `_configsvrBalancerStatus`, which blocks the backup from progressing

Could you please advise:

- Whether `_configsvrBalancerStatus` requires any additional internal privilege on MongoDB 8.0 config servers, and

- If any new role or privilege is required beyond `root` for PBM to function correctly in this version?

Thank you very much for your time and support.

---

<div class="post-metadata">

**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\
**Post date:** [December 10, 2025, 11:37am UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/4 "2025-12-10T11:37:21Z")

</div>

Hi, the root role is not enough please check the documentation for creating the `pbmAnyAction` role [Configure authentication in MongoDB - Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/install/configure-authentication.html#create-the-pbm-user)

---

<div class="post-metadata">

**Author:** ![Akshayrg09](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Akshayrg09](https://forums.percona.com/u/Akshayrg09)\
**Post date:** [December 15, 2025, 10:31pm UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/5 "2025-12-15T22:31:25Z")

</div>

Hi Team,

Thanks for all the support so far, our PBM setup is now working, storage is initialized, and all agents authenticate correctly. I just need clarification on a few behaviors we are seeing in our MongoDB 8.0 sharded cluster (2 mongos, 3 config nodes, 2 shards) **.**

* * *

## 1. Backups only succeed when run from Config Server Primary

When I run:

```auto
pbm backup --mongodb-uri=configPrimary

```

the backup succeeds on all replica sets (config + shard1 + shard2).

But when I run:

```auto
pbm backup --mongodb-uri=mongos

```

**it always fails with:**

![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/3X/f/e/feb573721ed1a77e7749a3ba5c2c6765de2e64c3.png)

```auto
incompatible: Backup has no data for the config server or sole replicaset

```

Question:  
Is it expected that PBM backups/restores must be initiated from a config primary or shard primary, and should not be run through mongos?

* * *

## 2. Restore performance seems very slow

Even restoring a small backup (~6 GB) takes close to 1 hour.  
For our real dataset (~200 GB), PBM logical restore may take many hours.

Questions:

- Are these restore times expected for PBM logical restores on sharded clusters?

- Does PBM restore collections and indexes in parallel, or is it single-threaded per node?

- Any recommended tuning to speed up restore?

- For ~200GB clusters, is PBM logical backup still recommended for DR, or should we consider other approaches?

* * *

## 3. What is the recommended best practice?

Before finalizing our DR runbook, I want to confirm:

- Should all PBM commands (backup/restore) always be run from config or shard primaries?

- Is mongos officially unsupported for initiating PBM operations?

- What is the realistic expected restore time for 200GB logical data?

- Any Percona recommendations for faster DR on large sharded clusters?

### **4.** Best Practices for Large Clusters (200GB)

- For large sharded clusters, does Percona still recommend PBM logical backups for DR?

- Or should we consider:

* * *

### 5. Partial Restore / Last N Weeks

If I want to restore only the last 10 weeks of data:

- Does PBM support any form of partial/filtered restore?

- Or is the recommended approach to use `mongodump` with `--query` per collection for partial restores?

- Is it safe to mix PBM (full cluster backup) with filtered `mongodump` during DR workflows?

* * *

Everything else is working fine (storage, roles, prefix, agent auth).  
I just want to confirm whether this behavior is expected and understand the best strategy for scalable backups/restores.

Thank you!

---

<div class="post-metadata">

**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\
**Post date:** [December 16, 2025, 11:44am UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/6 "2025-12-16T11:44:04Z")

</div>

Hi, let me see if I can answer your questions:

1. Actually you can run backup from any host that has PBM client installed even if it’s not part of the cluster. The thing to keep in mind is that in a sharded cluster, you have to [specify the mongodb-uri of the config server replica set](https://docs.percona.com/percona-backup-mongodb/details/authentication.html#mongodb-connection-strings).

2. Restore speed depends on many factors but you can play with [parallelism](https://docs.percona.com/percona-backup-mongodb/reference/restore-options.html?h=parallel#restorenumparallelcollections) and number of [download threads](https://docs.percona.com/percona-backup-mongodb/reference/restore-options.html?h=parallel#restorenumdownloadworkers) among other things

3. I answered that mostly in question 1). For a reference of expected times you can check this [post](https://www.percona.com/blog/physical-backup-support-in-percona-backup-for-mongodb/).

4. For large datasets we recommend either [physical](https://docs.percona.com/percona-backup-mongodb/features/physical.html) or [snapshot](https://docs.percona.com/percona-backup-mongodb/features/snapshots.html) backups with PBM

5. [Selective backup/restores](https://docs.percona.com/percona-backup-mongodb/usage/backup-selective.html) are supported but we don’t have the option to do a custom query. Feel free to open a FR for that in our JIRA.

Hope that helps, our [consulting team is available](https://docs.percona.com/percona-backup-mongodb/get-help.html) in case you need help implementing any of this.

---

<div class="post-metadata">

**Author:** ![Akshayrg09](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Akshayrg09](https://forums.percona.com/u/Akshayrg09)\
**Post date:** [December 16, 2025, 10:47pm UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/7 "2025-12-16T22:47:23Z")

</div>

Hi Ivan,

Thanks again to you and the team for all the support — really appreciate it.

Just to confirm my understanding on a couple of points:

1. For sharded clusters, while backups can be triggered from any host with PBM installed, both backup and restore operations should use the config server replica set URI (rather than mongos or any other). Is that correct?

2. In our testing on a sharded cluster, restoring ~6 GB of data with `mongorestore` completed in ~10 minutes on a smaller (4 GB RAM) server.  
The same dataset restored using PBM logical restore took close to ~1 hour on similar resources, but completed much faster (around a few minutes) when tested on a higher-memory server (~16 GB RAM).

Thanks again, this really helps us set the right expectations and sizing guidelines for our DR runbook.

---

<div class="post-metadata">

**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\
**Post date:** [December 17, 2025, 10:37am UTC](https://forums.percona.com/t/pbm-2-12-on-mongodb-8-0-sharded-cluster-storage-is-not-initialized-get-epoch-lookup-element-not-found/39822/8 "2025-12-17T10:37:36Z")

</div>

Hi, you are welcome.

1. Yes
2. mongorestore and pbm should perform roughly similar if configured the same. You can try configuring pbm with 4 parallel workers and 1 worker per collection to mimic mongorestore on a small server.