Restore to new environment with Operator 1.16.0 and mongo v7

MongoDB Percona Operator for MongoDB
,
1

HI All,
I am facing an issue when trying to restore backup of psmdb v7 (with operator 1.16.0) on new kubernetes setup. (with same secret of psmdb users). Same backup works well when restore on same setup where backup is taken.

I see problem on restore - mongos and config servers are continously restarting with below error. Anyone have any idea if there is something missing for restoring backup to new setups?

{“t”:{“$date”:“2024-06-24T05:49:27.619+00:00”},“s”:“E”, “c”:“ASSERT”, “id”:4457000, “ctx”:“CatalogCache-0”,“msg”:“Tripwire assertion”,“attr”:{“error”:{“code”:6493100,“codeName”:“Location6493100”,“errmsg”:“Time monotonicity violation: lookup time { chunkVersion: { e: ObjectId(‘6674055e88af34e862123419’), t: Timestamp(1718879582, 9), v: Timestamp(1, 2) }, forcedRefreshSequenceNum: 27, epochDisambiguatingSequenceNum: 42 } which is less than the earliest expected timeInStore { chunkVersion: { e: ObjectId(‘667901f96cb45948937c6ab4’), t: Timestamp(1719206393, 18), v: Timestamp(1, 1) }, forcedRefreshSequenceNum: 27, epochDisambiguatingSequenceNum: 41 }.”},“location”:“{fileName:"src/mongo/util/read_through_cache.h", line:549, functionName:"operator()"}”}}
{“t”:{“$date”:“2024-06-24T05:49:29.225+00:00”},“s”:“I”, “c”:“ASSERT”, “id”:4457002, “ctx”:“CatalogCache-0”,“msg”:“Detected prior failed tripwire assertions, please check your logs for "Tripwire assertion" entries with log id 4457000.”,“attr”:{“occurrences”:1}}

2

It happening because in first you made a fresh cluster with its own topologyTime which is newer that topologyTime in your backup. Even one second is a big difference.
Percona documentation states that you need to stop mongos (router service) before restoring by yourself. It’s crucial in mongodb 7 because difference in topologyTime causes critical error and mongos service is the one who remembers this new topologyTime.
tl; dr - you need to stop all mongos before restore. Not sure how the operator works but this is the cause.