pbm restore failed:
I’m doing a restore to a new cluster on EC2. restore runs for few mins and it got errored out. and the mongo process is crashed on all the nodes .
2022-08-23T13:56:19.419+0000 ctr-archive-20220225081326.trademarks.usaf 16.0MB
2022-08-23T13:56:19.419+0000 at.tmv-goods 16.0MB
2022-08-23T13:56:19.419+0000 ctr.trademarks.watch 15.9MB
2022-08-23T13:56:19.419+0000 ctr.trademarks.dartsip 16.0MB
2022-08-23T13:56:19.419+0000
2022-08-23T13:56:22.419+0000 ctr-archive-20220225081326.trademarks.usaf 16.0MB
2022-08-23T13:56:22.419+0000 at.tmv-goods 16.0MB
2022-08-23T13:56:22.419+0000 ctr.trademarks.watch 15.9MB
2022-08-23T13:56:22.419+0000 ctr.trademarks.dartsip 16.0MB
2022-08-23T13:56:22.419+0000
2022-08-23T13:56:25.418+0000 ctr-archive-20220225081326.trademarks.usaf 16.0MB
2022-08-23T13:56:25.418+0000 at.tmv-goods 16.0MB
2022-08-23T13:56:25.418+0000 ctr.trademarks.watch 15.9MB
2022-08-23T13:56:25.418+0000 ctr.trademarks.dartsip 16.0MB
2022-08-23T13:56:25.418+0000
2022-08-23T13:56:26.000+0000 D [restore/2022-08-17T14:09:26Z] releasing lock
2022-08-23T13:56:26.000+0000 E [restore/2022-08-17T14:09:26Z] mongorestore: restore mongo dump (successes: 0 / fails: 0): ctr.trademarks.iss: error creating collection ctr.trademarks.iss: applyOps: connection(10.80.11.176:27028[-18]) socket was unexpectedly closed: EOF
2022-08-23T13:56:27.000+0000 D [restore/2022-08-17T14:09:26Z] releasing lock
2022-08-23T13:56:27.000+0000 E [restore/2022-08-17T14:09:26Z] check cluster for restore started: convergeCluster: restore on the shard shard1ReplSet failed with: mongorestore: restore mongo dump (successes: 0 / fails: 0): ctr.trademarks.iss: error creating collection ctr.trademarks.iss: applyOps: connection(10.80.11.176:27028[-18]) socket was unexpectedly closed: EOF
**2022-08-23T13:56:27.000+0000 E [agentCheckup] check node connection: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: 10.80.11.176:27028, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp 10.80.11.176:27028: connect: connection refused }, ] }**
[mongod@ip-10-80-11-176 ~]$ pbm logs --event=restore
2022-08-23T13:55:53Z I [configReplSet/10.80.11.176:27029] [restore/2022-08-17T14:09:26Z] restore started
2022-08-23T13:55:53Z I [shard1ReplSet/10.80.11.176:27028] [restore/2022-08-17T14:09:26Z] restore started
2022-08-23T13:55:54Z I [shard3ReplSet/10.80.11.213:27028] [restore/2022-08-17T14:09:26Z] restore started
2022-08-23T13:55:54Z I [configReplSet/10.80.11.176:27029] [restore/2022-08-17T14:09:26Z] moving to state running
2022-08-23T13:55:54Z I [shard2ReplSet/10.80.11.55:27028] [restore/2022-08-17T14:09:26Z] restore started
2022-08-23T13:55:54Z I [shard1ReplSet/10.80.11.176:27028] [restore/2022-08-17T14:09:26Z] moving to state running
2022-08-23T13:55:55Z I [shard3ReplSet/10.80.11.213:27028] [restore/2022-08-17T14:09:26Z] moving to state running
2022-08-23T13:55:55Z I [shard2ReplSet/10.80.11.55:27028] [restore/2022-08-17T14:09:26Z] moving to state running
2022-08-23T13:56:02Z I [configReplSet/10.80.11.176:27029] [restore/2022-08-17T14:09:26Z] restoring users and roles
2022-08-23T13:56:02Z I [configReplSet/10.80.11.176:27029] [restore/2022-08-17T14:09:26Z] moving to state dumpDone
2022-08-23T13:56:26Z E [shard1ReplSet/10.80.11.176:27028] [restore/2022-08-17T14:09:26Z] **mongorestore: restore mongo dump (successes: 0 / fails: 0): ctr.trademarks.iss: error creating collection ctr.trademarks.iss: applyOps: connection(10.80.11.176:27028[-18]) socket was unexpectedly closed: EOF**
**2022-08-23T13:56:27Z E [configReplSet/10.80.11.176:27029] [restore/2022-08-17T14:09:26Z] check cluster for restore started: convergeCluster: restore on the shard shard1ReplSet failed with: mongorestore: restore mongo dump (successes: 0 / fails: 0): ctr.trademarks.iss: error creating collection ctr.trademarks.iss: applyOps: connection(10.80.11.176:27028[-18]) socket was unexpectedly closed: EOF**
**2022-08-23T14:01:13Z E [shard3ReplSet/10.80.11.213:27028] [restore/2022-08-17T14:09:26Z] mongorestore: restore mongo dump (successes: 5540000 / fails: 0): ca.trademark20220807200002: error restoring from archive on stdin: connection(10.80.11.213:27028[-29]) socket was unexpectedly closed: EOF**