PBm agent killed while taking backup

Hello Team,
We are using percona for backup and restore of mongo and it has been working fine for most of cases.

PBM version - 1.6.1-1.el8.x86_64
mongo - 4.2.0
DB - not sharded, running in cluster of 3 - one primary and two secondary
Backup - triggered every mid night LDN

Recently, we have seen scenario where mongoPBMagent was killed after receiving command for backUP.
Backup was started on one of secondary server, and then agent got killed

2022-08-14T16:36:53.000+0000 I pbm-agent:
Version:   1.6.1
Platform:  linux/amd64
GitCommit: aebbc24a03f21f9ddee6ebb72f510ff1d1c58fdc
GitBranch: release-1.6.1
BuildTime: 2021-11-03_12:11_UTC
GoVersion: go1.16.9
2022-08-14T16:36:53.000+0000 I node: cat/uklvasapp1001.gdc.standardchartered.com:24242
2022-08-14T16:36:53.000+0000 I listening for the commands
2022-08-14T16:36:53.000+0000 I starting PITR routine
2022-08-15T23:30:00.000+0000 I got command backup [name: 2022-08-15T23:30:00Z, compression: s2] <ts: 1660606200>
2022-08-15T23:30:00.000+0000 I got epoch {1660495028 2}
2022-08-15T23:30:00.000+0000 I [backup/2022-08-15T23:30:00Z] backup started
2022-08-15T23:30:03.000+0000 D [backup/2022-08-15T23:30:00Z] wait for tmp users {1660606203 3501}
2022-08-15T23:30:05.039+0000	Setting num cpus to 32
2022-08-15T23:30:05.062+0000	enqueued collection 'admin.system.version'
2022-08-15T23:30:05.062+0000	enqueued collection 'admin.pbmLog'
2022-08-15T23:30:05.063+0000	enqueued collection 'admin.system.profile'
2022-08-15T23:30:05.063+0000	enqueued collection 'admin.pbmConfig'
2022-08-15T23:30:05.063+0000	enqueued collection 'admin.pbmBackups.old'
2022-08-15T23:30:05.063+0000	enqueued collection 'admin.pbmOpLog'
2022-08-15T23:30:05.064+0000	enqueued collection 'admin.pbmPITRChunks'
2022-08-15T23:30:05.064+0000	enqueued collection 'admin.pbmBackups'
2022-08-15T23:30:05.064+0000	enqueued collection 'admin.pbmRestores'
2022-08-15T23:30:05.064+0000	enqueued collection 'admin.pbmCmd'
2022-08-15T23:30:05.064+0000	enqueued collection 'admin.system.roles'
2022-08-15T23:30:05.065+0000	enqueued collection 'admin.pbmRRoles'
2022-08-15T23:30:05.065+0000	enqueued collection 'admin.pbmLockOp'
2022-08-15T23:30:05.065+0000	enqueued collection 'admin.pbmRUsers'
2022-08-15T23:30:05.065+0000	enqueued collection 'admin.pbmLock'
2022-08-15T23:30:05.065+0000	enqueued collection 'admin.system.users'
2022-08-15T23:30:05.065+0000	enqueued collection 'admin.pbmAgents'
2022-08-15T23:43:54.874+0000	writing cat.Tradecollect1 to archive on stdout
2022/08/15 23:43:54 [############............]  cat.ccp55838/109278  (51.1%)
2022-08-15T23:43:54.883+0000	Mux close namespace cat.collection3
panic: send on closed channel

goroutine 784727 [running]:
github.com/mongodb/mongo-tools/common/archive.(*MuxIn).Open(0xc0000d4320, 0x0, 0xc0071e3c78)
	/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/common/archive/multiplexer.go:282 +0x31c
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).dumpValidatedQueryToIntent(0xc0004ae370, 0xc0071e3ef8, 0xc0003ee280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:666 +0x89
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).dumpQueryToIntent(...)
	/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:633
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).DumpIntent(0xc0004ae370, 0xc0003ee280, 0x0, 0x0, 0x0, 0x0)
	/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:616 +0x8fe
github.com/mongodb/mongo-tools/mongodump.(*MongoDump).DumpIntents.func1(0xc0004ae370, 0xc0004a84e0, 0x5)
	/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:501 +0x15a
created by github.com/mongodb/mongo-tools/mongodump.(*MongoDump).DumpIntents
	/mnt/jenkins/workspace/pbm-autobuild-RELEASE/test/rpmbuild/BUILD/src/github.com/percona/percona-backup-mongodb/vendor/github.com/mongodb/mongo-tools/mongodump/mongodump.go:490 +0x15b

Could you please let me know what might have triggered these errors.

Thanks

Hello, did you check the system messages? it might have been the operating system out-of-memory killer.