Hello Team,
We are using percona for backup and restore of mongo and it has been working fine for most of cases.
PBM version - 1.6.1-1.el8.x86_64
mongo - 4.2.0
DB - not sharded, running in cluster of 3 - one primary and two secondary
Backup - triggered every mid night LDN
Recently, we have seen scenario where mongoPBMagent was killed after receiving command for backUP.
Backup was started on one of secondary server, and then agent got killed
2022-08-14T16:36:53.000+0000 I pbm-agent:
Version: 1.6.1
Platform: linux/amd64
GitCommit: aebbc24a03f21f9ddee6ebb72f510ff1d1c58fdc
GitBranch: release-1.6.1
BuildTime: 2021-11-03_12:11_UTC
GoVersion: go1.16.9
2022-08-14T16:36:53.000+0000 I node: cat/uklvasapp1001.gdc.standardchartered.com:24242
2022-08-14T16:36:53.000+0000 I listening for the commands
2022-08-14T16:36:53.000+0000 I starting PITR routine
2022-08-15T23:30:00.000+0000 I got command backup [name: 2022-08-15T23:30:00Z, compression: s2] <ts: 1660606200>
2022-08-15T23:30:00.000+0000 I got epoch {1660495028 2}
2022-08-15T23:30:00.000+0000 I [backup/2022-08-15T23:30:00Z] backup started
2022-08-15T23:30:03.000+0000 D [backup/2022-08-15T23:30:00Z] wait for tmp users {1660606203 3501}
2022-08-15T23:30:05.039+0000 Setting num cpus to 32
2022-08-15T23:30:05.062+0000 enqueued collection 'admin.system.version'
2022-08-15T23:30:05.062+0000 enqueued collection 'admin.pbmLog'
2022-08-15T23:30:05.063+0000 enqueued collection 'admin.system.profile'
2022-08-15T23:30:05.063+0000 enqueued collection 'admin.pbmConfig'
2022-08-15T23:30:05.063+0000 enqueued collection 'admin.pbmBackups.old'
2022-08-15T23:30:05.063+0000 enqueued collection 'admin.pbmOpLog'
2022-08-15T23:30:05.064+0000 enqueued collection 'admin.pbmPITRChunks'
2022-08-15T23:30:05.064+0000 enqueued collection 'admin.pbmBackups'
2022-08-15T23:30:05.064+0000 enqueued collection 'admin.pbmRestores'
2022-08-15T23:30:05.064+0000 enqueued collection 'admin.pbmCmd'
2022-08-15T23:30:05.064+0000 enqueued collection 'admin.system.roles'
2022-08-15T23:30:05.065+0000 enqueued collection 'admin.pbmRRoles'
2022-08-15T23:30:05.065+0000 enqueued collection 'admin.pbmLockOp'
2022-08-15T23:30:05.065+0000 enqueued collection 'admin.pbmRUsers'
2022-08-15T23:30:05.065+0000 enqueued collection 'admin.pbmLock'
2022-08-15T23:30:05.065+0000 enqueued collection 'admin.system.users'
2022-08-15T23:30:05.065+0000 enqueued collection 'admin.pbmAgents'
2022-08-15T23:43:54.874+0000 writing cat.Tradecollect1 to archive on stdout
2022/08/15 23:43:54 [############............] cat.ccp55838/109278 (51.1%)
2022-08-15T23:43:54.883+0000 Mux close namespace cat.collection3
panic: send on closed channel
goroutine 784727 [running]:
github.com/mongodb/mongo-tools/common/archive.(*MuxIn).Open(0xc0000d4320, 0x0, 0xc0071e3c78)
/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/common/archive/multiplexer.go:282 +0x31c
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).dumpValidatedQueryToIntent(0xc0004ae370, 0xc0071e3ef8, 0xc0003ee280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:666 +0x89
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).dumpQueryToIntent(...)
/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:633
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).DumpIntent(0xc0004ae370, 0xc0003ee280, 0x0, 0x0, 0x0, 0x0)
/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:616 +0x8fe
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).DumpIntents.func1(0xc0004ae370, 0xc0004a84e0, 0x5)
/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:501 +0x15a
created by github.com/mongodb/mongo-tools/mongodump.(*MongoDump).DumpIntents
/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:490 +0x15b
Could you please let me know what might have triggered these errors.
Thanks