Zstd sync up issue on new replica for mongoDB

MongoDB Percona Server for MongoDB
1

Hi Team,

I am trying to sync up a huge (9 TB) replica set into Zstd and increased the oplog to source node to 6TB as well

It has been syncing since three weeks and the estimated time to sync has also been done (five days ago).

Now, the below error is looping with error. Can you please help me with it

{"t":{"$date":"2023-05-09T05:12:55.206+00:00"},"s":"E",  "c":"REPL",     "id":21258,   "ctx":"ReplWriterWorker-336","msg":"Failed to apply update","attr":{"op":{"op":"u","ns":"DB.collections","ui":{"$uuid":"b87fb597-8b6b-45f7-bf95-1d513bd7087f"},"o":{"$v":1,"$set":{"Id":"4dgjkius7ktgni5ji47v4u9n0g"}},"o2":{"_id":{"$oid":"618d95c6cff47e0006011a7c"}},"ts":{"$timestamp":{"t":1682709274,"i":1903}},"t":131,"v":2,"wall":{"$date":"2023-04-28T19:14:34.864Z"}}}}
{"t":{"$date":"2023-05-09T05:12:55.206+00:00"},"s":"E",  "c":"REPL",     "id":21258,   "ctx":"ReplWriterWorker-336","msg":"Failed to apply update","attr":{"op":{"op":"u","ns":"DB.collections","ui":{"$uuid":"b87fb597-8b6b-45f7-bf95-1d513bd7087f"},"o":{"$v":1,"$set":{"Id":"4dgjkius7ktgni5ji47v4u9n0g"}},"o2":{"_id":{"$oid":"618e8e8346e0fb000aa58821"}},"ts":{"$timestamp":{"t":1682709274,"i":1958}},"t":131,"v":2,"wall":{"$date":"2023-04-28T19:14:34.883Z"}}}}
{"t":{"$date":"2023-05-09T05:12:55.206+00:00"},"s":"E",  "c":"REPL",     "id":21258,   "ctx":"ReplWriterWorker-336","msg":"Failed to apply update","attr":{"op":{"op":"u","ns":"DB.collections","ui":{"$uuid":"b87fb597-8b6b-45f7-bf95-1d513bd7087f"},"o":{"$v":1,"$set":{"Id":"4dgjkius7ktgni5ji47v4u9n0g"}},"o2":{"_id":{"$oid":"63ebcbbf12e94b2e4688f569"}},"ts":{"$timestamp":{"t":1682709274,"i":2153}},"t":131,"v":2,"wall":{"$date":"2023-04-28T19:14:34.955Z"}}}}```
2

Hi, it is normal during initial sync to see some oplog apply failures. Do you see the data directory still growing? also which mongodb version are you using?

3

Hi @Ivan_Groenewold Thank you for the reply and I can see the data is still growing (is it because of huge oplog ~5TB) and We are using mongodb V4.4

4

You can check the last modification time of files at the operating system level to see if there’s any activity other than the oplog growing. If you have huge index it is possible it is taking very long to build it.

1 Like
5

Thank you @Ivan_Groenewold I can see the folders in data directory are getting updated.

"msg":"Failed to apply update"

This message was bothering me and it has been recurring since four days

6

The sync got into secondary mode after a week

Thanks for the solution @Ivan_Groenewold

7

glad to hear it worked