Mongodb Intialsync failed at specific db collection

Ahmed_Asim · July 26, 2023, 12:32pm

Description:

I’m trying to do cross site replication using percona mongodb operator mongodb version 4.4 and the InitialSync failed 100 times when it comes to specific collection which has over 196 million document as shown in screenshot below and I can’t see any specific logs for why it gets failed but when I do rs.status() at the DR site I see this message, each time it resets when it reaches 100G then it clears all the data and start from scratch !!

{
				"durationMillis" : 12999821,
				"status" : "InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'x.x.x.x.x' :: caused by :: network error while attempting to run command 'collStats' on host 'x.x.x.x:27017' ",
				"syncSource" : "x.x.x.x:27017",
				"rollBackId" : 8,
				"operationsRetried" : 1,
				"totalTimeUnreachableMillis" : 0
			}

I thought the issue with VPN tunnel between the two sites but I tried the sync publicly without going over the tunnel and it failed also at this specific collection. I don’t have any special config in place all are defaults except for the below::

db.adminCommand( { setParameter: 1, initialSyncTransientErrorRetryPeriodSeconds: 864000 } )
db.adminCommand( { setParameter: 1, oplogInitialFindMaxSeconds: 600 } )

what can we do to overcome this please given the fact it’s not networking issue

Ahmed_Asim · July 26, 2023, 1:26pm

replication status and info from the primary

rs0:PRIMARY> rs.status()
{
	"set" : "rs0",
	"date" : ISODate("2023-07-26T13:23:12.256Z"),
	"myState" : 1,
	"term" : NumberLong(68),
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"majorityVoteCount" : 2,
	"writeMajorityCount" : 2,
	"votingMembersCount" : 3,
	"writableVotingMembersCount" : 3,
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"lastCommittedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"readConcernMajorityWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
		"appliedOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
		"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z")
	},
	"lastStableRecoveryTimestamp" : Timestamp(1690377767, 1),
	"electionCandidateMetrics" : {
		"lastElectionReason" : "electionTimeout",
		"lastElectionDate" : ISODate("2023-07-24T15:46:30.194Z"),
		"electionTerm" : NumberLong(68),
		"lastCommittedOpTimeAtElection" : {
			"ts" : Timestamp(1690213573, 2),
			"t" : NumberLong(67)
		},
		"lastSeenOpTimeAtElection" : {
			"ts" : Timestamp(1690213573, 2),
			"t" : NumberLong(67)
		},
		"numVotesNeeded" : 2,
		"priorityAtElection" : 2,
		"electionTimeoutMillis" : NumberLong(10000),
		"priorPrimaryMemberId" : 5,
		"numCatchUpOps" : NumberLong(0),
		"newTermStartDate" : ISODate("2023-07-24T15:46:32.190Z"),
		"wMajorityWriteAvailabilityDate" : ISODate("2023-07-24T15:46:34.435Z")
	},
	"electionParticipantMetrics" : {
		"votedForCandidate" : true,
		"electionTerm" : NumberLong(67),
		"lastVoteDate" : ISODate("2023-07-23T23:08:12.406Z"),
		"electionCandidateMemberId" : 5,
		"voteReason" : "",
		"lastAppliedOpTimeAtElection" : {
			"ts" : Timestamp(1690153681, 1),
			"t" : NumberLong(66)
		},
		"maxAppliedOpTimeInSet" : {
			"ts" : Timestamp(1690153681, 1),
			"t" : NumberLong(66)
		},
		"priorityAtElection" : 2
	},
	"members" : [
		{
			"_id" : 2,
			"name" : "x.x.x.x:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 71287,
			"optime" : {
				"ts" : Timestamp(1690377789, 367),
				"t" : NumberLong(68)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1690377789, 367),
				"t" : NumberLong(68)
			},
			"optimeDate" : ISODate("2023-07-26T13:23:09Z"),
			"optimeDurableDate" : ISODate("2023-07-26T13:23:09Z"),
			"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastHeartbeat" : ISODate("2023-07-26T13:23:10.287Z"),
			"lastHeartbeatRecv" : ISODate("2023-07-26T13:23:10.452Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "x.x.x.x:27017",
			"syncSourceId" : 4,
			"infoMessage" : "",
			"configVersion" : 37,
			"configTerm" : 68
		},
		{
			"_id" : 4,
			"name" : "x.x.x.x27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 238371,
			"optime" : {
				"ts" : Timestamp(1690377791, 138),
				"t" : NumberLong(68)
			},
			"optimeDate" : ISODate("2023-07-26T13:23:11Z"),
			"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1690213590, 1),
			"electionDate" : ISODate("2023-07-24T15:46:30Z"),
			"configVersion" : 37,
			"configTerm" : 68,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 5,
			"name" : "x.x.x.x:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 64601,
			"optime" : {
				"ts" : Timestamp(1690377791, 138),
				"t" : NumberLong(68)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1690377791, 138),
				"t" : NumberLong(68)
			},
			"optimeDate" : ISODate("2023-07-26T13:23:11Z"),
			"optimeDurableDate" : ISODate("2023-07-26T13:23:11Z"),
			"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastHeartbeat" : ISODate("2023-07-26T13:23:12.037Z"),
			"lastHeartbeatRecv" : ISODate("2023-07-26T13:23:11.963Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "x.x.x.x:27017",
			"syncSourceId" : 4,
			"infoMessage" : "",
			"configVersion" : 37,
			"configTerm" : 68
		},
		{
			"_id" : 6,
			"name" : "x.x.x.x:27017",
			"health" : 1,
			"state" : 5,
			"stateStr" : "STARTUP2",
			"uptime" : 70805,
			"optime" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastAppliedWallTime" : ISODate("1970-01-01T00:00:00Z"),
			"lastDurableWallTime" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2023-07-26T13:23:11.627Z"),
			"lastHeartbeatRecv" : ISODate("2023-07-26T13:23:10.911Z"),
			"pingMs" : NumberLong(30),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "x.x.x.x:27017",
			"syncSourceId" : 2,
			"infoMessage" : "",
			"configVersion" : 37,
			"configTerm" : 68
		}
	],
	"ok" : 1,
	"$gleStats" : {
		"lastOpTime" : Timestamp(0, 0),
		"electionId" : ObjectId("7fffffff0000000000000044")
	},
	"lastCommittedOpTime" : Timestamp(1690377791, 138),
	"$configServerState" : {
		"opTime" : {
			"ts" : Timestamp(1690377792, 2),
			"t" : NumberLong(141)
		}
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1690377792, 2),
		"signature" : {
			"hash" : BinData(0,"V5YKHAYqnSzwgzZUUBCbs2jvXZ4="),
			"keyId" : NumberLong("7225960576311099393")
		}
	},
	"operationTime" : Timestamp(1690377791, 138)
}



rs0:PRIMARY> rs.printReplicationInfo()
configured oplog size:   300000MB
log length start to end: 4096578secs (1137.94hrs)
oplog first event time:  Fri Jun 09 2023 03:26:53 GMT+0000 (UTC)
oplog last event time:   Wed Jul 26 2023 13:23:11 GMT+0000 (UTC)
now:                     Wed Jul 26 2023 13:23:22 GMT+0000 (UTC)
rs0:PRIMARY>

Ahmed_Asim · July 26, 2023, 1:29pm

@Sergey_Pronin for your kind support

Santosh_Varma · July 26, 2023, 3:42pm

Hi Ahmed,
Can you check the liveliness check of the pod on which the sync is failing.

Regards

Ahmed_Asim · July 26, 2023, 4:04pm

Hi @Santosh_Varma

you mean the syncSourceHost or the pod on the DR side?

Ahmed_Asim · July 26, 2023, 4:25pm

this is from inside the pod which is always fail with sync if that what you meant

Ahmed_Asim · July 26, 2023, 5:03pm

FYI, we disabled the liveness probe

Santosh_Varma · July 26, 2023, 5:58pm

Okay…have you disabled the liveliness prod of mongod container ?
can you again try resync while the liveliness probe is disabled?

Also, when you encountered the error, did you check from other nodes if there was any connetion timeout error to the node which is failing just before the pod restarts.

btw, does you pod also restarts when it fails to sycn?

Ahmed_Asim · July 26, 2023, 9:51pm

Okay…have you disabled the liveliness prod of mongod container ? YES
can you again try resync while the liveliness probe is disabled? We did that already and the sync resets and starts from the beginning each time

Also, when you encountered the error, did you check from other nodes if there was any connetion timeout error to the node which is failing just before the pod restarts. I didn’t notice any timeout and the pod is not restarting

btw, does you pod also restarts when it fails to sycn? No it’s not, each time we noticed that /data/db size is 103GB and suddenly it goes down to 2GB and syncing the collections starts from the beginning

@Santosh_Varma

Ahmed_Asim · July 26, 2023, 11:57pm

I see this log entry all the time at the pod which the sync fails

"t":{"$date":"2023-07-26T21:05:39.508+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.512+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.584+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.666+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.701+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.726+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.940+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:40.095+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:40.102+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.202:42283","connectionId":9846,"connectionCount":26}}
{"t":{"$date":"2023-07-26T21:05:40.103+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn9846","msg":"Connection ended","attr":{"remote":"10.11.10.202:42283","connectionId":9846,"connectionCount":25}}
{"t":{"$date":"2023-07-26T21:05:40.114+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.200:41982","connectionId":9847,"connectionCount":26}}
{"t":{"$date":"2023-07-26T21:05:40.114+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn9847","msg":"Connection ended","attr":{"remote":"10.11.10.200:41982","connectionId":9847,"connectionCount":25}}
{"t":{"$date":"2023-07-26T21:05:40.115+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.221:7374","connectionId":9848,"connectionCount":26}}
{"t":{"$date":"2023-07-26T21:05:40.115+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn9848","msg":"Connection ended","attr":{"remote":"10.11.10.221:7374","connectionId":9848,"connectionCount":25}}
{"t":{"$date":"2023-07-26T21:05:40.122+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}

Ahmed_Asim · July 27, 2023, 12:32am

I aslo got this at the syncSourceHost, what does internal assertion error means

"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"-",        "id":4892201, "ctx":"conn106","msg":"Internal assertion","attr":{"error":{"code":262,"codeName":"ExceededTimeLimit","errmsg":"operation exceeded time limit"},"location":"{fileName:\"src/mongo/util/interruptible.h\", line:398, functionName:\"operator()\"}"}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"NETWORK",  "id":22934,   "ctx":"conn106","msg":"Starting server-side compression negotiation"}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"NETWORK",  "id":22935,   "ctx":"conn106","msg":"Compression negotiation not requested by client"}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"FTDC",     "id":23905,   "ctx":"conn106","msg":"Using exhaust for isMaster or hello protocol"}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D5", "c":"COMMAND",  "id":21958,   "ctx":"conn106","msg":"Appending operationTime to cmd response: {operationTime}","attr":{"operationTime":{"ts":{"$timestamp":{"t":1690417670,"i":1526}}}}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn106","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"64c19f0cdb8e53afd2267da1"},"counter":13},"maxAwaitTimeMS":10000,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690410798,"i":1}},"signature":{"hash":{"$binary":{"base64":"RU+uo1BhrIMQtRMuEuhKuYmmMpg=","subType":"0"}},"keyId":7225960576311099393}},"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":1079,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D1", "c":"QUERY",    "id":22790,   "ctx":"conn106","msg":"Received interrupt request for unknown op","attr":{"opId":282900}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D2", "c":"QUERY",    "id":22783,   "ctx":"conn106","msg":"Ops known during interrupt","attr":{"ops":[]}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D2", "c":"COMMAND",  "id":21965,   "ctx":"conn106","msg":"About to run the command","attr":{"db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"64c19f0cdb8e53afd2267da1"},"counter":13},"maxAwaitTimeMS":10000,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690410798,"i":1}},"signature":{"hash":{"$binary":{"base64":"RU+uo1BhrIMQtRMuEuhKuYmmMpg=","subType":"0"}},"keyId":7225960576311099393}},"$db":"admin","$readPreference":{"mode":"primaryPreferred"}}}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"FTDC",     "id":23904,   "ctx":"conn106","msg":"Using maxAwaitTimeMS for awaitable isMaster protocol."}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D1", "c":"REPL",     "id":21342,   "ctx":"conn106","msg":"Waiting for an isMaster response from a topology change or until deadline","attr":{"deadline":{"$date":"2023-07-27T00:28:06.338Z"},"currentTopologyVersionCounter":13}}

Ahmed_Asim · July 27, 2023, 12:33am

Also seen this at the syncSourceHost

{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn13255","msg":"Slow query","attr":{"type":"command","ns":"IntegrationANB.$cmd","command":{"listCollections":1,"nameOnly":true,"readConcern":{"provenance":"implicitDefault"},"databaseVersion":{"uuid":{"$uuid":"9389b8db-33cb-4e45-a864-42e643b9b2fb"},"lastMod":1},"lsid":{"id":{"$uuid":"6ff3d40e-5cab-4106-be61-8fd444fc69a3"},"uid":{"$binary":{"base64":"T2UCS2zgdm0zHC75J2H+MUDKJ1MRHWhVTIs5W48URwU=","subType":"0"}}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690417840,"i":2}},"signature":{"hash":{"$binary":{"base64":"wz6nEAAdb1b3lsMLk3EYaK1zjHY=","subType":"0"}},"keyId":7225960576311099393}},"$audit":{"$impersonatedUsers":[{"user":"integration-anb","db":"IntegrationANB"}],"$impersonatedRoles":[{"role":"dbOwner","db":"IntegrationANB"}]},"$client":{"driver":{"name":"mongo-csharp-driver","version":"2.11.6.0"},"os":{"type":"Linux","name":"Linux 5.4.17-2136.315.5.el7uek.x86_64 #2 SMP Wed Dec 21 19:57:57 PST 2022","architecture":"x86_64","version":"5.4.17-2136.315.5.el7uek.x86_64"},"platform":".NET 5.0.17","mongos":{"host":"psmdb-db-mongos-1:27017","client":"10.244.5.12:33158","version":"4.4.16-16"}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1690417840,"i":2}},"t":145}},"$db":"IntegrationANB"},"numYields":0,"reslen":786,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"r":1}},"Database":{"acquireCount":{"r":1}},"Collection":{"acquireCount":{"r":8}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"provenance":"implicitDefault"},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D1", "c":"QUERY",    "id":22790,   "ctx":"conn13255","msg":"Received interrupt request for unknown op","attr":{"opId":293641}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D2", "c":"QUERY",    "id":22783,   "ctx":"conn13255","msg":"Ops known during interrupt","attr":{"ops":[]}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D3", "c":"NETWORK",  "id":22925,   "ctx":"conn13255","msg":"Compressing message","attr":{"compressor":"snappy"}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D3", "c":"NETWORK",  "id":22927,   "ctx":"conn12193","msg":"Decompressing message","attr":{"compressor":"snappy"}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D2", "c":"COMMAND",  "id":21965,   "ctx":"conn12193","msg":"About to run the command","attr":{"db":"IntegrationANB","commandArgs":{"listCollections":1,"nameOnly":true,"readConcern":{"provenance":"implicitDefault"},"databaseVersion":{"uuid":{"$uuid":"9389b8db-33cb-4e45-a864-42e643b9b2fb"},"lastMod":1},"lsid":{"id":{"$uuid":"6ffa4317-97f6-4827-841b-fff2a1a16bf2"},"uid":{"$binary":{"base64":"T2UCS2zgdm0zHC75J2H+MUDKJ1MRHWhVTIs5W48URwU=","subType":"0"}}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690417840,"i":2}},"signature":{"hash":{"$binary":{"base64":"wz6nEAAdb1b3lsMLk3EYaK1zjHY=","subType":"0"}},"keyId":7225960576311099393}},"$audit":{"$impersonatedUsers":[{"user":"integration-anb","db":"IntegrationANB"}],"$impersonatedRoles":[{"role":"dbOwner","db":"IntegrationANB"}]},"$client":{"driver":{"name":"mongo-csharp-driver","version":"2.11.6.0"},"os":{"type":"Linux","name":"Linux 5.4.17-2136.315.5.el7uek.x86_64 #2 SMP Wed Dec 21 19:57:57 PST 2022","architecture":"x86_64","version":"5.4.17-2136.315.5.el7uek.x86_64"},"platform":".NET 5.0.17","mongos":{"host":"psmdb-db-mongos-0:27017","client":"10.244.5.12:60574","version":"4.4.16-16"}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1690417840,"i":2}},"t":145}},"$db":"IntegrationANB"}}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D5", "c":"COMMAND",  "id":21957,   "ctx":"conn12193","msg":"Appending operationTime to cmd response for authorized client: {operationTime}","attr":{"operationTime":{"ts":{"$timestamp":{"t":1690417839,"i":5}}}}}

Ahmed_Asim · July 27, 2023, 1:33am

Initial sync failed again with this status:

	{
				"durationMillis" : 15933213,
				"status" : "CallbackCanceled: error fetching oplog during initial sync :: caused by :: oplog fetcher shutting down",
				"syncSource" : "x.x.x.x:27017",
				"rollBackId" : 8,
				"operationsRetried" : 1,
				"totalTimeUnreachableMillis" : 0
			}

Ahmed_Asim · July 27, 2023, 8:37am

I also can see this:

{"t":{"$date":"2023-07-27T08:34:43.552+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:34:58.144+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:01.217+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:06.336+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:18.881+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:33.472+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:36.544+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}

Ahmed_Asim · July 27, 2023, 9:48am

this is the log when the the Intialsync fails:

{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"E",  "c":"INITSYNC", "id":21149,   "ctx":"ReplCoordExtern-5","msg":"Collection clone failed","attr":{"namespace":"IntegrationANB.TerminalHistory","error":"HostUnreachable: network error while attempting to run command 'collStats' on host 'x.x.x.x:27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"W",  "c":"INITSYNC", "id":21060,   "ctx":"ReplCoordExtern-5","msg":"Database clone failed","attr":{"dbName":"IntegrationANB","dbNumber":4,"totalDbs":23,"error":"InitialSyncFailure: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host 'x.x.x.x27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I",  "c":"INITSYNC", "id":21183,   "ctx":"ReplCoordExtern-5","msg":"Finished cloning data. Beginning oplog replay","attr":{"databaseClonerFinishStatus":"InitialSyncFailure: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host 'x.x.x..x:27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I",  "c":"INITSYNC", "id":21181,   "ctx":"ReplCoordExtern-0","msg":"Finished fetching oplog during initial sync","attr":{"oplogFetcherFinishStatus":"CallbackCanceled: oplog fetcher shutting down","lastFetched":"{ ts: Timestamp(1690450679, 6), t: 78 }"}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I",  "c":"INITSYNC", "id":21191,   "ctx":"ReplCoordExtern-0","msg":"Initial sync attempt finishing up"}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I",  "c":"INITSYNC", "id":21192,   "ctx":"ReplCoordExtern-0","msg":"Initial sync status and statistics","attr":{"status":"in_progress","statistics":{"failedInitialSyncAttempts":4,"maxFailedInitialSyncAttempts":10,"initialSyncStart":{"$date":"2023-07-26T20:55:20.259Z"},"totalInitialSyncElapsedMillis":45760217,"initialSyncAttempts":[{"durationMillis":0,"status":"InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.","syncSource":":27017","operationsRetried":0,"totalTimeUnreachableMillis":0},{"durationMillis":15933213,"status":"CallbackCanceled: error fetching oplog during initial sync :: caused by :: oplog fetcher shutting down","syncSource":"207.127.97.242:27017","rollBackId":8,"operationsRetried":1,"totalTimeUnreachableMillis":0},{"durationMillis":15182353,"status":"InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host '207.127.97.242:27017' ","syncSource":"207.127.97.242:27017","rollBackId":8,"operationsRetried":41,"totalTimeUnreachableMillis":14598721},{"durationMillis":14613951,"status":"InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host '207.127.97.242:27017' ","syncSource":"207.127.97.242:27017","rollBackId":8,"operationsRetried":1,"totalTimeUnreachableMillis":0}],"approxTotalDataSize":294266552446.0,"approxTotalBytesCopied":163515750588.0,"remainingInitialSyncEstimatedMillis":36590879,"appliedOps":0,"initialSyncOplogStart":{"$timestamp":{"t":1690436060,"i":8}},"totalTimeUnreachableMillis":0,"databases":{"databasesToClone":20,"databasesCloned":3,"admin":{"collections":5,"clonedCollections":5,"start":{"$date":"2023-07-27T05:34:27.587Z"},"end":{"$date":"2023-07-27T05:34:28.482Z"},"elapsedMillis":895,"admin.pbmRUsers":{"documentsToCopy":0,"documentsCopied":0,"indexes":1,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2023-07-27T05:34:27.622Z"},"end":{"$date":"2023-07-27T05:34:27.819Z"},"elapsedMillis":197,"receivedBatches":0},"admin.system.roles":{"documentsToCopy":2,"documentsCopied":2,"indexes":2,"fetchedBatches":1,"bytesToCopy":441,"approxBytesCopied":440,"start":{"$date":"2023-07-27T05:34:27.819Z"},"end":{"$date":"2023-07-27T05:34:27.994Z"},"elapsedMillis":175,"receivedBatches":1},"admin.system.users":{"documentsToCopy":5,"documentsCopied":5,"indexes":2,"fetchedBatches":1,"bytesToCopy":3210,"approxBytesCopied":3210,"start":{"$date":"2023-07-27T05:34:27.994Z"},"end":{"$date":"2023-07-27T05:34:28.168Z"},"elapsedMillis":174,"receivedBatches":1},"admin.pbmRRoles":{"documentsToCopy":0,"documentsCopied":0,"indexes":1,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2023-07-27T05:34:28.168Z"},"end":{"$date":"2023-07-27T05:34:28.325Z"},"elapsedMillis":157,"receivedBatches":0},"admin.system.version":{"documentsToCopy":3,"documentsCopied":3,"indexes":1,"fetchedBatches":1,"bytesToCopy":279,"approxBytesCopied":279,"start":{"$date":"2023-07-27T05:34:28.325Z"},"end":{"$date":"2023-07-27T05:34:28.482Z"},"elapsedMillis":157,"receivedBatches":1}},"ElmInfoDb":{"collections":5,"clonedCollections":5,"start":{"$date":"2023-07-27T05:34:28.482Z"},"end":{"$date":"2023-07-27T05:36:17.462Z"},"elapsedMillis":108980,"ElmInfoDb.ElmInfos":{"documentsToCopy":412212,"documentsCopied":412212,"indexes":3,"fetchedBatches":15,"bytesToCopy":216465224,"approxBytesCopied":216411300,"start":{"$date":"2023-07-27T05:34:28.518Z"},"end":{"$date":"2023-07-27T05:34:38.815Z"},"elapsedMillis":10297,"receivedBatches":15},"ElmInfoDb.IndividualNationalAddresses":{"documentsToCopy":239201,"documentsCopied":239201,"indexes":1,"fetchedBatches":8,"bytesToCopy":102303029,"approxBytesCopied":102138827,"start":{"$date":"2023-07-27T05:34:38.815Z"},"end":{"$date":"2023-07-27T05:34:43.428Z"},"elapsedMillis":4613,"receivedBatches":8},"ElmInfoDb.MobileOwnership":{"documentsToCopy":5428575,"documentsCopied":5428575,"indexes":4,"fetchedBatches":53,"bytesToCopy":825143400,"approxBytesCopied":825143400,"start":{"$date":"2023-07-27T05:34:43.428Z"},"end":{"$date":"2023-07-27T05:36:12.826Z"},"elapsedMillis":89398,"receivedBatches":53},"ElmInfoDb.ElmErrors":{"documentsToCopy":483983,"documentsCopied":483983,"indexes":2,"fetchedBatches":6,"bytesToCopy":76383206,"approxBytesCopied":75985331,"start":{"$date":"2023-07-27T05:36:12.826Z"},"end":{"$date":"2023-07-27T05:36:16.308Z"},"elapsedMillis":3482,"receivedBatches":6},"ElmInfoDb.CompanyNationalAddresses":{"documentsToCopy":52075,"documentsCopied":52075,"indexes":1,"fetchedBatches":3,"bytesToCopy":20433337,"approxBytesCopied":20413400,"start":{"$date":"2023-07-27T05:36:16.308Z"},"end":{"$date":"2023-07-27T05:36:17.462Z"},"elapsedMillis":1154,"receivedBatches":3}},"HyperPay":{"collections":4,"clonedCollections":4,"start":{"$date":"2023-07-27T05:36:17.462Z"},"end":{"$date":"2023-07-27T05:36:37.345Z"},"elapsedMillis":19883,"HyperPay.Tokens":{"documentsToCopy":83298,"documentsCopied":83298,"indexes":1,"fetchedBatches":3,"bytesToCopy":31565284,"approxBytesCopied":31486644,"start":{"$date":"2023-07-27T05:36:17.499Z"},"end":{"$date":"2023-07-27T05:36:19.451Z"},"elapsedMillis":1952,"receivedBatches":3},"HyperPay.WebHook":{"documentsToCopy":10,"documentsCopied":10,"indexes":1,"fetchedBatches":1,"bytesToCopy":15105,"approxBytesCopied":15100,"start":{"$date":"2023-07-27T05:36:19.451Z"},"end":{"$date":"2023-07-27T05:36:19.612Z"},"elapsedMillis":161,"receivedBatches":1},"HyperPay.ProxyLogs":{"documentsToCopy":633651,"documentsCopied":633651,"indexes":1,"fetchedBatches":24,"bytesToCopy":380724871,"approxBytesCopied":380190600,"start":{"$date":"2023-07-27T05:36:19.612Z"},"end":{"$date":"2023-07-27T05:36:31.170Z"},"elapsedMillis":11558,"receivedBatches":24},"HyperPay.TopupSessions":{"documentsToCopy":633582,"documentsCopied":633582,"indexes":2,"fetchedBatches":6,"bytesToCopy":69068365,"approxBytesCopied":69060438,"start":{"$date":"2023-07-27T05:36:31.170Z"},"end":{"$date":"2023-07-27T05:36:37.345Z"},"elapsedMillis":6175,"receivedBatches":6}},"IntegrationANB":{"collections":7,"clonedCollections":6,"start":{"$date":"2023-07-27T05:36:37.345Z"},"IntegrationANB.BinRangeLookups":{"documentsToCopy":465,"documentsCopied":465,"indexes":1,"fetchedBatches":2,"bytesToCopy":27423,"approxBytesCopied":26970,"start":{"$date":"2023-07-27T05:36:37.379Z"},"end":{"$date":"2023-07-27T05:36:37.573Z"},"elapsedMillis":194,"receivedBatches":2},"IntegrationANB.TerminalsV2":{"documentsToCopy":6,"documentsCopied":6,"indexes":4,"fetchedBatches":1,"bytesToCopy":923,"approxBytesCopied":918,"start":{"$date":"2023-07-27T05:36:37.573Z"},"end":{"$date":"2023-07-27T05:36:37.770Z"},"elapsedMillis":197,"receivedBatches":1},"IntegrationANB.TerminalActionLogs":{"documentsToCopy":252452,"documentsCopied":252452,"indexes":3,"fetchedBatches":3,"bytesToCopy":26502604,"approxBytesCopied":26255008,"start":{"$date":"2023-07-27T05:36:37.770Z"},"end":{"$date":"2023-07-27T05:36:40.080Z"},"elapsedMillis":2310,"receivedBatches":3},"IntegrationANB.Terminals":{"documentsToCopy":299328,"documentsCopied":299328,"indexes":3,"fetchedBatches":4,"bytesToCopy":32138542,"approxBytesCopied":32028096,"start":{"$date":"2023-07-27T05:36:40.080Z"},"end":{"$date":"2023-07-27T05:36:42.808Z"},"elapsedMillis":2728,"receivedBatches":4},"IntegrationANB.CardFees":{"documentsToCopy":1,"documentsCopied":1,"indexes":1,"fetchedBatches":1,"bytesToCopy":147,"approxBytesCopied":147,"start":{"$date":"2023-07-27T05:36:42.808Z"},"end":{"$date":"2023-07-27T05:36:42.965Z"},"elapsedMillis":157,"receivedBatches":1},"IntegrationANB.POSReceipts":{"documentsToCopy":196282270,"documentsCopied":196282270,"indexes":16,"fetchedBatches":9728,"bytesToCopy":161910101965.0,"approxBytesCopied":161736590480.0,"start":{"$date":"2023-07-27T05:36:42.965Z"},"end":{"$date":"2023-07-27T09:38:00.475Z"},"elapsedMillis":14477510,"receivedBatches":9728},"IntegrationANB.TerminalHistory":{"documentsToCopy":0,"documentsCopied":0,"indexes":0,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2023-07-27T09:38:00.475Z"},"receivedBatches":0}},"IntegrationCommodity":{"collections":0,"clonedCollections":0},"IntegrationEmdha":{"collections":0,"clonedCollections":0},"IntegrationNafath":{"collections":0,"clonedCollections":0},"IntegrationNafith":{"collections":0,"clonedCollections":0},"IntegrationSP":{"collections":0,"clonedCollections":0},"IntegrationSadad":{"collections":0,"clonedCollections":0},"IntegrationSimah":{"collections":0,"clonedCollections":0},"IntegrationSoftPOS":{"collections":0,"clonedCollections":0},"NatheerDB":{"collections":0,"clonedCollections":0},"RayaDB":{"collections":0,"clonedCollections":0},"SAQR":{"collections":0,"clonedCollections":0},"WathqDB":{"collections":0,"clonedCollections":0},"BanksB2BProxy":{"collections":0,"clonedCollections":0},"config":{"collections":0,"clonedCollections":0},"debezium":{"collections":0,"clonedCollections":0},"like4cards":{"collections":0,"clonedCollections":0},"mpgs":{"collections":0,"clonedCollections":0},"one_card_db":{"collections":0,"clonedCollections":0},"test":{"collections":0,"clonedCollections":0}}}}}
{"t":{"$date":"2023-07-27T09:38:00.477+00:00"},"s":"E",  "c":"INITSYNC", "id":21200,   "ctx":"ReplCoordExtern-0","msg":"Initial sync attempt failed","attr":{"attemptsLeft":6,"error":"InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host '207.127.97.242:27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.528+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.200:14831","connectionId":705872,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:00.529+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705872","msg":"Connection ended","attr":{"remote":"10.11.10.200:14831","connectionId":705872,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:00.534+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"CONNPOOL", "id":22572,   "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.10.0.49:27017","error":"HostUnreachable: Connection reset by peer"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"cfg","host":"10.10.0.49:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"10.10.0.41:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"10.10.0.49:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"cfg","host":"10.10.0.49:27017","delayMillis":0}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"8.213.16.23:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"10.10.0.170:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"10.10.0.49:27017"}}
{"t":{"$date":"2023-07-27T09:38:00.612+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.695+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.199:28889","connectionId":705876,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:00.695+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705876","msg":"Connection ended","attr":{"remote":"10.11.10.199:28889","connectionId":705876,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:00.739+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.881+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.982+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.158:21005","connectionId":705879,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:00.982+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705879","msg":"Connection ended","attr":{"remote":"10.11.10.158:21005","connectionId":705879,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:01.131+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.153+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.237+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.261+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.331+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.477+00:00"},"s":"I",  "c":"INITSYNC", "id":21164,   "ctx":"ReplCoordExtern-6","msg":"Starting initial sync attempt","attr":{"initialSyncAttempt":5,"initialSyncMaxAttempts":10}}
{"t":{"$date":"2023-07-27T09:38:01.477+00:00"},"s":"I",  "c":"STORAGE",  "id":20318,   "ctx":"ReplCoordExtern-6","msg":"Finishing collection drop","attr":{"namespace":"local.temp_oplog_buffer","uuid":{"uuid":{"$uuid":"791f7513-12aa-4e4c-b73b-ec2c6948945a"}}}}
{"t":{"$date":"2023-07-27T09:38:01.480+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"ReplCoordExtern-6","msg":"createCollection","attr":{"namespace":"local.temp_oplog_buffer","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"863eadb5-cdac-417a-bc7d-b333c76d8a10"}},"options":{"temp":true}}}
{"t":{"$date":"2023-07-27T09:38:01.487+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"ReplCoordExtern-6","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.temp_oplog_buffer","index":"_id_","commitTimestamp":{"$timestamp":{"t":1690450679,"i":7}}}}
{"t":{"$date":"2023-07-27T09:38:01.487+00:00"},"s":"I",  "c":"REPL",     "id":21799,   "ctx":"ReplCoordExtern-6","msg":"Sync source candidate chosen","attr":{"syncSource":"207.127.97.123:27017"}}
{"t":{"$date":"2023-07-27T09:38:01.488+00:00"},"s":"I",  "c":"INITSYNC", "id":21173,   "ctx":"ReplCoordExtern-6","msg":"Initial syncer oplog truncation finished","attr":{"durationMillis":0}}
{"t":{"$date":"2023-07-27T09:38:01.488+00:00"},"s":"I",  "c":"STORAGE",  "id":4833200, "ctx":"ReplCoordExtern-6","msg":"About to abort all index builders running","attr":{"reason":"Aborting index builds for initial sync"}}
{"t":{"$date":"2023-07-27T09:38:01.488+00:00"},"s":"I",  "c":"REPL",     "id":21754,   "ctx":"ReplCoordExtern-6","msg":"dropReplicatedDatabases - dropping databases","attr":{"numDatabases":5}}
{"t":{"$date":"2023-07-27T09:38:01.514+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.524+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.627+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.692+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.719+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.746+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.221:17643","connectionId":705890,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:01.746+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn705890","msg":"client metadata","attr":{"remote":"10.11.10.221:17643","client":"conn705890","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.19.6"}}}
{"t":{"$date":"2023-07-27T09:38:01.749+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.203:48911","connectionId":705891,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:01.749+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn705891","msg":"client metadata","attr":{"remote":"10.11.10.203:48911","client":"conn705891","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.19.6"}}}
{"t":{"$date":"2023-07-27T09:38:01.818+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.821+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.982+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}

{"t":{"$date":"2023-07-27T09:38:02.121+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.136+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.136+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.195+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.267+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.297+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.334+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.335+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.446+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.528+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.544+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.567+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.604+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.783+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.817+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.953+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.055+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.182+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.229+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.204:16691","connectionId":705913,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.229+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705913","msg":"Connection ended","attr":{"remote":"10.11.10.204:16691","connectionId":705913,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.250+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.261+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.16:11367","connectionId":705915,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.261+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705915","msg":"Connection ended","attr":{"remote":"10.11.10.16:11367","connectionId":705915,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.367+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.30.20.1:41822","connectionId":705916,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.367+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705916","msg":"Connection ended","attr":{"remote":"172.30.20.1:41822","connectionId":705916,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.528+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.540+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.541+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.30.20.1:61331","connectionId":705919,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.541+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705919","msg":"Connection ended","attr":{"remote":"172.30.20.1:61331","connectionId":705919,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.633+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":

Ahmed_Asim · July 27, 2023, 9:48am

{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.656+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.739+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.745+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.203:58496","connectionId":705923,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.745+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705923","msg":"Connection ended","attr":{"remote":"10.11.10.203:58496","connectionId":705923,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.896+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.945+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.041+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.147+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.196+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.201+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.24:42210","connectionId":705929,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:04.201+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705929","msg":"Connection ended","attr":{"remote":"10.11.10.24:42210","connectionId":705929,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:04.328+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.352+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.358+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.200:59054","connectionId":705932,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:04.358+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705932","msg":"Connection ended","attr":{"remote":"10.11.10.200:59054","connectionId":705932,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:04.523+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.623+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.628+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.644+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.647+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.668+00:00"},"s":"I",  "c":"REPL",     "id":21756,   "ctx":"ReplCoordExtern-6","msg":"dropReplicatedDatabases - dropped databases","attr":{"numDatabases":5}}
{"t":{"$date":"2023-07-27T09:38:04.668+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplCoordExternNetwork","msg":"Connecting","attr":{"hostAndPort":"207.127.97.123:27017"}}
{"t":{"$date":"2023-07-27T09:38:04.669+00:00"},"s":"I",  "c":"COMMAND",  "id":20499,   "ctx":"ftdc","msg":"serverStatus was very slow","attr":{"timeStats":{"after basic":0,"after asserts":0,"after connections":0,"after electionMetrics":0,"after extra_info":0,"after flowControl":0,"after globalLock":0,"after indexBulkBuilder":0,"after locks":0,"after logicalSessionRecordCache":0,"after mirroredReads":0,"after network":0,"after opLatencies":0,"after opReadConcernCounters":0,"after opcounters":0,"after opcountersRepl":0,"after oplog":2668,"after oplogTruncation":2668,"after repl":2668,"after scramCache":2668,"after security":2668,"after shardingStatistics":2668,"after storageEngine":2668,"after tcmalloc":2668,"after trafficRecording":2668,"after transactions":2668,"after transportSecurity":2668,"after twoPhaseCommitCoordinator":2668,"after wiredTiger":2668,"at end":2668}}}
{"t":{"$date":"2023-07-27T09:38:04.756+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705890","msg":"Connection ended","attr":{"remote":"10.11.10.221:17643","connectionId":705890,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:04.757+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn705891","msg":"Interrupted operation as its client disconnected","attr":{"opId":1201798}}
{"t":{"$date":"2023-07-27T09:38:04.757+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705891","msg":"Connection ended","attr":{"remote":"10.11.10.203:48911","connectionId":705891,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:04.821+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.879+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.972+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.980+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.200+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.236+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.254+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.338+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.446+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.488+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.500+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.626+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.675+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.686+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.203:29068","connectionId":705954,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:05.686+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn705954","msg":"Connection ended","attr":{"remote":"10.11.10.203:29068","connectionId":705954,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:05.798+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.949+00:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}

Ahmed_Asim · July 27, 2023, 1:32pm

this is also from the pod which the intialsync fails:

{"t":{"$date":"2023-07-27T12:24:28.693+00:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-195","msg":"Heartbeat failed after max retries","attr":{"target":"x.x.x.x:27017","maxHeartbeatRetries":2,"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"}}}

“status” : "InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection ‘Integration.History’ :: caused by :: network error while attempting to run command ‘collStats’ on host ‘x.x.x.x:27017’ ",

Ahmed_Asim · July 27, 2023, 1:33pm

Hi @Santosh_Varma I really appreciate your help!

Santosh_Varma · July 27, 2023, 2:39pm

Hi Ahmed,

Going by the logs,
looks like the destination pod is not able to detect primary (its sync source )

{“t”:{“$date”:“2023-07-27T08:35:06.336+00:00”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“x.x.x.x:27017”,“replicaSet”:“rs0”}}

is existing replicaset outside k8? are there anyother pods which are member of the set rs0 and secondary?

How are you adding the new node? with the endpoint ip and port or with cluster ip and port into existing replicaset?

when intialy sycn fails doe the primary change?

Ahmed_Asim · July 31, 2023, 12:32am

Hi @Santosh_Varma

I can say finally I’m able to identify the issue which is a slow query on that particular collection each time it reports the same error and I’m sure this is issue because I compare the connectionID on both sites and it’s the same and after some time the connection gets ended, please advice how can we fix this

Thanks
Ahmed Asim

Topic		Replies	Views
Initial sync fails Percona Server for MongoDB	1	883	October 7, 2020
MongoDB Cluster IPv6, Replicaset initialization is failing Percona Operator for MongoDB percona	7	1188	July 13, 2022
MongoServerSelectionError: ReplicaSetNoPrimary Percona Operator for MongoDB	11	4168	October 5, 2023
MongoDB Backups failing intermitently - cluster with status initializing Percona Operator for MongoDB percona	10	1601	August 18, 2022
When I turn on the backup option, operator reports an error：error occured during connection handshake: x509 Percona Operator for MongoDB	10	1318	February 3, 2023

Mongodb Intialsync failed at specific db collection

Description:

Related Topics