# Mongodb Intialsync failed at specific db collection

**URL:** <https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922>\
**Category:** Percona Operator for MongoDB\
**Created:** [July 26, 2023, 12:32pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922 "2023-07-26T12:32:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 12:32pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/1 "2023-07-26T12:32:54Z")

</div>

## Description:

I’m trying to do cross site replication using percona mongodb operator mongodb version 4.4 and the InitialSync failed 100 times when it comes to specific collection which has over 196 million document as shown in screenshot below and I can’t see any specific logs for why it gets failed but when I do rs.status() at the DR site I see this message, each time it resets when it reaches 100G then it clears all the data and start from scratch !! ☹

```auto
{
				"durationMillis" : 12999821,
				"status" : "InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'x.x.x.x.x' :: caused by :: network error while attempting to run command 'collStats' on host 'x.x.x.x:27017' ",
				"syncSource" : "x.x.x.x:27017",
				"rollBackId" : 8,
				"operationsRetried" : 1,
				"totalTimeUnreachableMillis" : 0
			}

```

 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/6/61d949b852d5d9355220bf1a0c728cebdc45163f.png)

I thought the issue with VPN tunnel between the two sites but I tried the sync publicly without going over the tunnel and it failed also at this specific collection. I don’t have any special config in place all are defaults except for the below::

```auto
db.adminCommand( { setParameter: 1, initialSyncTransientErrorRetryPeriodSeconds: 864000 } )
db.adminCommand( { setParameter: 1, oplogInitialFindMaxSeconds: 600 } )

```

what can we do to overcome this please given the fact it’s not networking issue

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 1:26pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/2 "2023-07-26T13:26:30Z")

</div>

replication status and info from the primary

```auto
rs0:PRIMARY> rs.status()
{
	"set" : "rs0",
	"date" : ISODate("2023-07-26T13:23:12.256Z"),
	"myState" : 1,
	"term" : NumberLong(68),
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"majorityVoteCount" : 2,
	"writeMajorityCount" : 2,
	"votingMembersCount" : 3,
	"writableVotingMembersCount" : 3,
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"lastCommittedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"readConcernMajorityWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
		"appliedOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1690377791, 138),
			"t" : NumberLong(68)
		},
		"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
		"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z")
	},
	"lastStableRecoveryTimestamp" : Timestamp(1690377767, 1),
	"electionCandidateMetrics" : {
		"lastElectionReason" : "electionTimeout",
		"lastElectionDate" : ISODate("2023-07-24T15:46:30.194Z"),
		"electionTerm" : NumberLong(68),
		"lastCommittedOpTimeAtElection" : {
			"ts" : Timestamp(1690213573, 2),
			"t" : NumberLong(67)
		},
		"lastSeenOpTimeAtElection" : {
			"ts" : Timestamp(1690213573, 2),
			"t" : NumberLong(67)
		},
		"numVotesNeeded" : 2,
		"priorityAtElection" : 2,
		"electionTimeoutMillis" : NumberLong(10000),
		"priorPrimaryMemberId" : 5,
		"numCatchUpOps" : NumberLong(0),
		"newTermStartDate" : ISODate("2023-07-24T15:46:32.190Z"),
		"wMajorityWriteAvailabilityDate" : ISODate("2023-07-24T15:46:34.435Z")
	},
	"electionParticipantMetrics" : {
		"votedForCandidate" : true,
		"electionTerm" : NumberLong(67),
		"lastVoteDate" : ISODate("2023-07-23T23:08:12.406Z"),
		"electionCandidateMemberId" : 5,
		"voteReason" : "",
		"lastAppliedOpTimeAtElection" : {
			"ts" : Timestamp(1690153681, 1),
			"t" : NumberLong(66)
		},
		"maxAppliedOpTimeInSet" : {
			"ts" : Timestamp(1690153681, 1),
			"t" : NumberLong(66)
		},
		"priorityAtElection" : 2
	},
	"members" : [
		{
			"_id" : 2,
			"name" : "x.x.x.x:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 71287,
			"optime" : {
				"ts" : Timestamp(1690377789, 367),
				"t" : NumberLong(68)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1690377789, 367),
				"t" : NumberLong(68)
			},
			"optimeDate" : ISODate("2023-07-26T13:23:09Z"),
			"optimeDurableDate" : ISODate("2023-07-26T13:23:09Z"),
			"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastHeartbeat" : ISODate("2023-07-26T13:23:10.287Z"),
			"lastHeartbeatRecv" : ISODate("2023-07-26T13:23:10.452Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "x.x.x.x:27017",
			"syncSourceId" : 4,
			"infoMessage" : "",
			"configVersion" : 37,
			"configTerm" : 68
		},
		{
			"_id" : 4,
			"name" : "x.x.x.x27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 238371,
			"optime" : {
				"ts" : Timestamp(1690377791, 138),
				"t" : NumberLong(68)
			},
			"optimeDate" : ISODate("2023-07-26T13:23:11Z"),
			"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1690213590, 1),
			"electionDate" : ISODate("2023-07-24T15:46:30Z"),
			"configVersion" : 37,
			"configTerm" : 68,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 5,
			"name" : "x.x.x.x:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 64601,
			"optime" : {
				"ts" : Timestamp(1690377791, 138),
				"t" : NumberLong(68)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1690377791, 138),
				"t" : NumberLong(68)
			},
			"optimeDate" : ISODate("2023-07-26T13:23:11Z"),
			"optimeDurableDate" : ISODate("2023-07-26T13:23:11Z"),
			"lastAppliedWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastDurableWallTime" : ISODate("2023-07-26T13:23:11.443Z"),
			"lastHeartbeat" : ISODate("2023-07-26T13:23:12.037Z"),
			"lastHeartbeatRecv" : ISODate("2023-07-26T13:23:11.963Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "x.x.x.x:27017",
			"syncSourceId" : 4,
			"infoMessage" : "",
			"configVersion" : 37,
			"configTerm" : 68
		},
		{
			"_id" : 6,
			"name" : "x.x.x.x:27017",
			"health" : 1,
			"state" : 5,
			"stateStr" : "STARTUP2",
			"uptime" : 70805,
			"optime" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(0, 0),
				"t" : NumberLong(-1)
			},
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastAppliedWallTime" : ISODate("1970-01-01T00:00:00Z"),
			"lastDurableWallTime" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2023-07-26T13:23:11.627Z"),
			"lastHeartbeatRecv" : ISODate("2023-07-26T13:23:10.911Z"),
			"pingMs" : NumberLong(30),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "x.x.x.x:27017",
			"syncSourceId" : 2,
			"infoMessage" : "",
			"configVersion" : 37,
			"configTerm" : 68
		}
	],
	"ok" : 1,
	"$gleStats" : {
		"lastOpTime" : Timestamp(0, 0),
		"electionId" : ObjectId("7fffffff0000000000000044")
	},
	"lastCommittedOpTime" : Timestamp(1690377791, 138),
	"$configServerState" : {
		"opTime" : {
			"ts" : Timestamp(1690377792, 2),
			"t" : NumberLong(141)
		}
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1690377792, 2),
		"signature" : {
			"hash" : BinData(0,"V5YKHAYqnSzwgzZUUBCbs2jvXZ4="),
			"keyId" : NumberLong("7225960576311099393")
		}
	},
	"operationTime" : Timestamp(1690377791, 138)
}

rs0:PRIMARY> rs.printReplicationInfo()
configured oplog size: 300000MB
log length start to end: 4096578secs (1137.94hrs)
oplog first event time: Fri Jun 09 2023 03:26:53 GMT+0000 (UTC)
oplog last event time: Wed Jul 26 2023 13:23:11 GMT+0000 (UTC)
now: Wed Jul 26 2023 13:23:22 GMT+0000 (UTC)
rs0:PRIMARY> 

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 1:29pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/3 "2023-07-26T13:29:06Z")

</div>

@Sergey_Pronin for your kind support

---

<div class="post-metadata">

**Author:** ![Santosh\_Varma](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/santosh_varma/32/6924_2.png) [@Santosh\_Varma](https://forums.percona.com/u/Santosh_Varma)\
**Post date:** [July 26, 2023, 3:42pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/4 "2023-07-26T15:42:55Z")

</div>

Hi Ahmed,  
Can you check the liveliness check of the pod on which the sync is failing.

Regards

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 4:04pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/5 "2023-07-26T16:04:39Z")

</div>

Hi @Santosh_Varma

you mean the `syncSourceHost` or the pod on the DR side?

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 4:25pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/6 "2023-07-26T16:25:10Z")

</div>

this is from inside the pod which is always fail with sync if that what you meant

 ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/b/b1d740333a0131cf9cc811ba596eb5dd2b95b34a.jpeg)

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 5:03pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/7 "2023-07-26T17:03:41Z")

</div>

FYI, we disabled the liveness probe

---

<div class="post-metadata">

**Author:** ![Santosh\_Varma](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/santosh_varma/32/6924_2.png) [@Santosh\_Varma](https://forums.percona.com/u/Santosh_Varma)\
**Post date:** [July 26, 2023, 5:58pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/8 "2023-07-26T17:58:42Z")

</div>

Okay…have you disabled the liveliness prod of mongod container ?  
can you again try resync while the liveliness probe is disabled?

Also, when you encountered the error, did you check from other nodes if there was any connetion timeout error to the node which is failing just before the pod restarts.

btw, does you pod also restarts when it fails to sycn?

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 9:51pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/9 "2023-07-26T21:51:02Z")

</div>

Okay…have you disabled the liveliness prod of mongod container ? **YES**  
can you again try resync while the liveliness probe is disabled? **We did that already and the sync resets and starts from the beginning each time**

Also, when you encountered the error, did you check from other nodes if there was any connetion timeout error to the node which is failing just before the pod restarts. **I didn’t notice any timeout and the pod is not restarting**

btw, does you pod also restarts when it fails to sycn? **No it’s not, each time we noticed that /data/db size is 103GB and suddenly it goes down to 2GB and syncing the collections starts from the beginning**

@Santosh_Varma

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 26, 2023, 11:57pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/10 "2023-07-26T23:57:08Z")

</div>

I see this log entry all the time at the pod which the sync fails

```auto
"t":{"$date":"2023-07-26T21:05:39.508+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.512+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.584+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.666+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.701+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.726+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:39.940+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:40.095+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-26T21:05:40.102+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.202:42283","connectionId":9846,"connectionCount":26}}
{"t":{"$date":"2023-07-26T21:05:40.103+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn9846","msg":"Connection ended","attr":{"remote":"10.11.10.202:42283","connectionId":9846,"connectionCount":25}}
{"t":{"$date":"2023-07-26T21:05:40.114+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.200:41982","connectionId":9847,"connectionCount":26}}
{"t":{"$date":"2023-07-26T21:05:40.114+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn9847","msg":"Connection ended","attr":{"remote":"10.11.10.200:41982","connectionId":9847,"connectionCount":25}}
{"t":{"$date":"2023-07-26T21:05:40.115+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.221:7374","connectionId":9848,"connectionCount":26}}
{"t":{"$date":"2023-07-26T21:05:40.115+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn9848","msg":"Connection ended","attr":{"remote":"10.11.10.221:7374","connectionId":9848,"connectionCount":25}}
{"t":{"$date":"2023-07-26T21:05:40.122+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 12:32am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/11 "2023-07-27T00:32:24Z")

</div>

I aslo got this at the syncSourceHost, what does internal assertion error means

```auto
"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"-", "id":4892201, "ctx":"conn106","msg":"Internal assertion","attr":{"error":{"code":262,"codeName":"ExceededTimeLimit","errmsg":"operation exceeded time limit"},"location":"{fileName:\"src/mongo/util/interruptible.h\", line:398, functionName:\"operator()\"}"}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"NETWORK", "id":22934, "ctx":"conn106","msg":"Starting server-side compression negotiation"}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"NETWORK", "id":22935, "ctx":"conn106","msg":"Compression negotiation not requested by client"}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"FTDC", "id":23905, "ctx":"conn106","msg":"Using exhaust for isMaster or hello protocol"}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D5", "c":"COMMAND", "id":21958, "ctx":"conn106","msg":"Appending operationTime to cmd response: {operationTime}","attr":{"operationTime":{"ts":{"$timestamp":{"t":1690417670,"i":1526}}}}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn106","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"64c19f0cdb8e53afd2267da1"},"counter":13},"maxAwaitTimeMS":10000,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690410798,"i":1}},"signature":{"hash":{"$binary":{"base64":"RU+uo1BhrIMQtRMuEuhKuYmmMpg=","subType":"0"}},"keyId":7225960576311099393}},"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":1079,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D1", "c":"QUERY", "id":22790, "ctx":"conn106","msg":"Received interrupt request for unknown op","attr":{"opId":282900}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D2", "c":"QUERY", "id":22783, "ctx":"conn106","msg":"Ops known during interrupt","attr":{"ops":[]}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D2", "c":"COMMAND", "id":21965, "ctx":"conn106","msg":"About to run the command","attr":{"db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"64c19f0cdb8e53afd2267da1"},"counter":13},"maxAwaitTimeMS":10000,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690410798,"i":1}},"signature":{"hash":{"$binary":{"base64":"RU+uo1BhrIMQtRMuEuhKuYmmMpg=","subType":"0"}},"keyId":7225960576311099393}},"$db":"admin","$readPreference":{"mode":"primaryPreferred"}}}}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D3", "c":"FTDC", "id":23904, "ctx":"conn106","msg":"Using maxAwaitTimeMS for awaitable isMaster protocol."}
{"t":{"$date":"2023-07-27T00:27:56.338+00:00"},"s":"D1", "c":"REPL", "id":21342, "ctx":"conn106","msg":"Waiting for an isMaster response from a topology change or until deadline","attr":{"deadline":{"$date":"2023-07-27T00:28:06.338Z"},"currentTopologyVersionCounter":13}}

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 12:33am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/12 "2023-07-27T00:33:18Z")

</div>

Also seen this at the syncSourceHost

```auto
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn13255","msg":"Slow query","attr":{"type":"command","ns":"IntegrationANB.$cmd","command":{"listCollections":1,"nameOnly":true,"readConcern":{"provenance":"implicitDefault"},"databaseVersion":{"uuid":{"$uuid":"9389b8db-33cb-4e45-a864-42e643b9b2fb"},"lastMod":1},"lsid":{"id":{"$uuid":"6ff3d40e-5cab-4106-be61-8fd444fc69a3"},"uid":{"$binary":{"base64":"T2UCS2zgdm0zHC75J2H+MUDKJ1MRHWhVTIs5W48URwU=","subType":"0"}}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690417840,"i":2}},"signature":{"hash":{"$binary":{"base64":"wz6nEAAdb1b3lsMLk3EYaK1zjHY=","subType":"0"}},"keyId":7225960576311099393}},"$audit":{"$impersonatedUsers":[{"user":"integration-anb","db":"IntegrationANB"}],"$impersonatedRoles":[{"role":"dbOwner","db":"IntegrationANB"}]},"$client":{"driver":{"name":"mongo-csharp-driver","version":"2.11.6.0"},"os":{"type":"Linux","name":"Linux 5.4.17-2136.315.5.el7uek.x86_64 #2 SMP Wed Dec 21 19:57:57 PST 2022","architecture":"x86_64","version":"5.4.17-2136.315.5.el7uek.x86_64"},"platform":".NET 5.0.17","mongos":{"host":"psmdb-db-mongos-1:27017","client":"10.244.5.12:33158","version":"4.4.16-16"}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1690417840,"i":2}},"t":145}},"$db":"IntegrationANB"},"numYields":0,"reslen":786,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1}},"ReplicationStateTransition":{"acquireCount":{"w":1}},"Global":{"acquireCount":{"r":1}},"Database":{"acquireCount":{"r":1}},"Collection":{"acquireCount":{"r":8}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"provenance":"implicitDefault"},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D1", "c":"QUERY", "id":22790, "ctx":"conn13255","msg":"Received interrupt request for unknown op","attr":{"opId":293641}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D2", "c":"QUERY", "id":22783, "ctx":"conn13255","msg":"Ops known during interrupt","attr":{"ops":[]}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D3", "c":"NETWORK", "id":22925, "ctx":"conn13255","msg":"Compressing message","attr":{"compressor":"snappy"}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D3", "c":"NETWORK", "id":22927, "ctx":"conn12193","msg":"Decompressing message","attr":{"compressor":"snappy"}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D2", "c":"COMMAND", "id":21965, "ctx":"conn12193","msg":"About to run the command","attr":{"db":"IntegrationANB","commandArgs":{"listCollections":1,"nameOnly":true,"readConcern":{"provenance":"implicitDefault"},"databaseVersion":{"uuid":{"$uuid":"9389b8db-33cb-4e45-a864-42e643b9b2fb"},"lastMod":1},"lsid":{"id":{"$uuid":"6ffa4317-97f6-4827-841b-fff2a1a16bf2"},"uid":{"$binary":{"base64":"T2UCS2zgdm0zHC75J2H+MUDKJ1MRHWhVTIs5W48URwU=","subType":"0"}}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1690417840,"i":2}},"signature":{"hash":{"$binary":{"base64":"wz6nEAAdb1b3lsMLk3EYaK1zjHY=","subType":"0"}},"keyId":7225960576311099393}},"$audit":{"$impersonatedUsers":[{"user":"integration-anb","db":"IntegrationANB"}],"$impersonatedRoles":[{"role":"dbOwner","db":"IntegrationANB"}]},"$client":{"driver":{"name":"mongo-csharp-driver","version":"2.11.6.0"},"os":{"type":"Linux","name":"Linux 5.4.17-2136.315.5.el7uek.x86_64 #2 SMP Wed Dec 21 19:57:57 PST 2022","architecture":"x86_64","version":"5.4.17-2136.315.5.el7uek.x86_64"},"platform":".NET 5.0.17","mongos":{"host":"psmdb-db-mongos-0:27017","client":"10.244.5.12:60574","version":"4.4.16-16"}},"$configServerState":{"opTime":{"ts":{"$timestamp":{"t":1690417840,"i":2}},"t":145}},"$db":"IntegrationANB"}}}
{"t":{"$date":"2023-07-27T00:30:41.173+00:00"},"s":"D5", "c":"COMMAND", "id":21957, "ctx":"conn12193","msg":"Appending operationTime to cmd response for authorized client: {operationTime}","attr":{"operationTime":{"ts":{"$timestamp":{"t":1690417839,"i":5}}}}}

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 1:33am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/13 "2023-07-27T01:33:00Z")

</div>

Initial sync failed again with this status:

```auto
	{
				"durationMillis" : 15933213,
				"status" : "CallbackCanceled: error fetching oplog during initial sync :: caused by :: oplog fetcher shutting down",
				"syncSource" : "x.x.x.x:27017",
				"rollBackId" : 8,
				"operationsRetried" : 1,
				"totalTimeUnreachableMillis" : 0
			}

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 8:37am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/14 "2023-07-27T08:37:19Z")

</div>

I also can see this:

```auto
{"t":{"$date":"2023-07-27T08:34:43.552+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:34:58.144+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:01.217+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:06.336+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:18.881+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:33.472+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}
{"t":{"$date":"2023-07-27T08:35:36.544+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"x.x.x.x:27017","replicaSet":"rs0"}}

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 9:48am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/15 "2023-07-27T09:48:00Z")

</div>

this is the log when the the Intialsync fails:

```auto
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"E", "c":"INITSYNC", "id":21149, "ctx":"ReplCoordExtern-5","msg":"Collection clone failed","attr":{"namespace":"IntegrationANB.TerminalHistory","error":"HostUnreachable: network error while attempting to run command 'collStats' on host 'x.x.x.x:27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"W", "c":"INITSYNC", "id":21060, "ctx":"ReplCoordExtern-5","msg":"Database clone failed","attr":{"dbName":"IntegrationANB","dbNumber":4,"totalDbs":23,"error":"InitialSyncFailure: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host 'x.x.x.x27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I", "c":"INITSYNC", "id":21183, "ctx":"ReplCoordExtern-5","msg":"Finished cloning data. Beginning oplog replay","attr":{"databaseClonerFinishStatus":"InitialSyncFailure: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host 'x.x.x..x:27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I", "c":"INITSYNC", "id":21181, "ctx":"ReplCoordExtern-0","msg":"Finished fetching oplog during initial sync","attr":{"oplogFetcherFinishStatus":"CallbackCanceled: oplog fetcher shutting down","lastFetched":"{ ts: Timestamp(1690450679, 6), t: 78 }"}}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I", "c":"INITSYNC", "id":21191, "ctx":"ReplCoordExtern-0","msg":"Initial sync attempt finishing up"}
{"t":{"$date":"2023-07-27T09:38:00.476+00:00"},"s":"I", "c":"INITSYNC", "id":21192, "ctx":"ReplCoordExtern-0","msg":"Initial sync status and statistics","attr":{"status":"in_progress","statistics":{"failedInitialSyncAttempts":4,"maxFailedInitialSyncAttempts":10,"initialSyncStart":{"$date":"2023-07-26T20:55:20.259Z"},"totalInitialSyncElapsedMillis":45760217,"initialSyncAttempts":[{"durationMillis":0,"status":"InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.","syncSource":":27017","operationsRetried":0,"totalTimeUnreachableMillis":0},{"durationMillis":15933213,"status":"CallbackCanceled: error fetching oplog during initial sync :: caused by :: oplog fetcher shutting down","syncSource":"207.127.97.242:27017","rollBackId":8,"operationsRetried":1,"totalTimeUnreachableMillis":0},{"durationMillis":15182353,"status":"InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host '207.127.97.242:27017' ","syncSource":"207.127.97.242:27017","rollBackId":8,"operationsRetried":41,"totalTimeUnreachableMillis":14598721},{"durationMillis":14613951,"status":"InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host '207.127.97.242:27017' ","syncSource":"207.127.97.242:27017","rollBackId":8,"operationsRetried":1,"totalTimeUnreachableMillis":0}],"approxTotalDataSize":294266552446.0,"approxTotalBytesCopied":163515750588.0,"remainingInitialSyncEstimatedMillis":36590879,"appliedOps":0,"initialSyncOplogStart":{"$timestamp":{"t":1690436060,"i":8}},"totalTimeUnreachableMillis":0,"databases":{"databasesToClone":20,"databasesCloned":3,"admin":{"collections":5,"clonedCollections":5,"start":{"$date":"2023-07-27T05:34:27.587Z"},"end":{"$date":"2023-07-27T05:34:28.482Z"},"elapsedMillis":895,"admin.pbmRUsers":{"documentsToCopy":0,"documentsCopied":0,"indexes":1,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2023-07-27T05:34:27.622Z"},"end":{"$date":"2023-07-27T05:34:27.819Z"},"elapsedMillis":197,"receivedBatches":0},"admin.system.roles":{"documentsToCopy":2,"documentsCopied":2,"indexes":2,"fetchedBatches":1,"bytesToCopy":441,"approxBytesCopied":440,"start":{"$date":"2023-07-27T05:34:27.819Z"},"end":{"$date":"2023-07-27T05:34:27.994Z"},"elapsedMillis":175,"receivedBatches":1},"admin.system.users":{"documentsToCopy":5,"documentsCopied":5,"indexes":2,"fetchedBatches":1,"bytesToCopy":3210,"approxBytesCopied":3210,"start":{"$date":"2023-07-27T05:34:27.994Z"},"end":{"$date":"2023-07-27T05:34:28.168Z"},"elapsedMillis":174,"receivedBatches":1},"admin.pbmRRoles":{"documentsToCopy":0,"documentsCopied":0,"indexes":1,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2023-07-27T05:34:28.168Z"},"end":{"$date":"2023-07-27T05:34:28.325Z"},"elapsedMillis":157,"receivedBatches":0},"admin.system.version":{"documentsToCopy":3,"documentsCopied":3,"indexes":1,"fetchedBatches":1,"bytesToCopy":279,"approxBytesCopied":279,"start":{"$date":"2023-07-27T05:34:28.325Z"},"end":{"$date":"2023-07-27T05:34:28.482Z"},"elapsedMillis":157,"receivedBatches":1}},"ElmInfoDb":{"collections":5,"clonedCollections":5,"start":{"$date":"2023-07-27T05:34:28.482Z"},"end":{"$date":"2023-07-27T05:36:17.462Z"},"elapsedMillis":108980,"ElmInfoDb.ElmInfos":{"documentsToCopy":412212,"documentsCopied":412212,"indexes":3,"fetchedBatches":15,"bytesToCopy":216465224,"approxBytesCopied":216411300,"start":{"$date":"2023-07-27T05:34:28.518Z"},"end":{"$date":"2023-07-27T05:34:38.815Z"},"elapsedMillis":10297,"receivedBatches":15},"ElmInfoDb.IndividualNationalAddresses":{"documentsToCopy":239201,"documentsCopied":239201,"indexes":1,"fetchedBatches":8,"bytesToCopy":102303029,"approxBytesCopied":102138827,"start":{"$date":"2023-07-27T05:34:38.815Z"},"end":{"$date":"2023-07-27T05:34:43.428Z"},"elapsedMillis":4613,"receivedBatches":8},"ElmInfoDb.MobileOwnership":{"documentsToCopy":5428575,"documentsCopied":5428575,"indexes":4,"fetchedBatches":53,"bytesToCopy":825143400,"approxBytesCopied":825143400,"start":{"$date":"2023-07-27T05:34:43.428Z"},"end":{"$date":"2023-07-27T05:36:12.826Z"},"elapsedMillis":89398,"receivedBatches":53},"ElmInfoDb.ElmErrors":{"documentsToCopy":483983,"documentsCopied":483983,"indexes":2,"fetchedBatches":6,"bytesToCopy":76383206,"approxBytesCopied":75985331,"start":{"$date":"2023-07-27T05:36:12.826Z"},"end":{"$date":"2023-07-27T05:36:16.308Z"},"elapsedMillis":3482,"receivedBatches":6},"ElmInfoDb.CompanyNationalAddresses":{"documentsToCopy":52075,"documentsCopied":52075,"indexes":1,"fetchedBatches":3,"bytesToCopy":20433337,"approxBytesCopied":20413400,"start":{"$date":"2023-07-27T05:36:16.308Z"},"end":{"$date":"2023-07-27T05:36:17.462Z"},"elapsedMillis":1154,"receivedBatches":3}},"HyperPay":{"collections":4,"clonedCollections":4,"start":{"$date":"2023-07-27T05:36:17.462Z"},"end":{"$date":"2023-07-27T05:36:37.345Z"},"elapsedMillis":19883,"HyperPay.Tokens":{"documentsToCopy":83298,"documentsCopied":83298,"indexes":1,"fetchedBatches":3,"bytesToCopy":31565284,"approxBytesCopied":31486644,"start":{"$date":"2023-07-27T05:36:17.499Z"},"end":{"$date":"2023-07-27T05:36:19.451Z"},"elapsedMillis":1952,"receivedBatches":3},"HyperPay.WebHook":{"documentsToCopy":10,"documentsCopied":10,"indexes":1,"fetchedBatches":1,"bytesToCopy":15105,"approxBytesCopied":15100,"start":{"$date":"2023-07-27T05:36:19.451Z"},"end":{"$date":"2023-07-27T05:36:19.612Z"},"elapsedMillis":161,"receivedBatches":1},"HyperPay.ProxyLogs":{"documentsToCopy":633651,"documentsCopied":633651,"indexes":1,"fetchedBatches":24,"bytesToCopy":380724871,"approxBytesCopied":380190600,"start":{"$date":"2023-07-27T05:36:19.612Z"},"end":{"$date":"2023-07-27T05:36:31.170Z"},"elapsedMillis":11558,"receivedBatches":24},"HyperPay.TopupSessions":{"documentsToCopy":633582,"documentsCopied":633582,"indexes":2,"fetchedBatches":6,"bytesToCopy":69068365,"approxBytesCopied":69060438,"start":{"$date":"2023-07-27T05:36:31.170Z"},"end":{"$date":"2023-07-27T05:36:37.345Z"},"elapsedMillis":6175,"receivedBatches":6}},"IntegrationANB":{"collections":7,"clonedCollections":6,"start":{"$date":"2023-07-27T05:36:37.345Z"},"IntegrationANB.BinRangeLookups":{"documentsToCopy":465,"documentsCopied":465,"indexes":1,"fetchedBatches":2,"bytesToCopy":27423,"approxBytesCopied":26970,"start":{"$date":"2023-07-27T05:36:37.379Z"},"end":{"$date":"2023-07-27T05:36:37.573Z"},"elapsedMillis":194,"receivedBatches":2},"IntegrationANB.TerminalsV2":{"documentsToCopy":6,"documentsCopied":6,"indexes":4,"fetchedBatches":1,"bytesToCopy":923,"approxBytesCopied":918,"start":{"$date":"2023-07-27T05:36:37.573Z"},"end":{"$date":"2023-07-27T05:36:37.770Z"},"elapsedMillis":197,"receivedBatches":1},"IntegrationANB.TerminalActionLogs":{"documentsToCopy":252452,"documentsCopied":252452,"indexes":3,"fetchedBatches":3,"bytesToCopy":26502604,"approxBytesCopied":26255008,"start":{"$date":"2023-07-27T05:36:37.770Z"},"end":{"$date":"2023-07-27T05:36:40.080Z"},"elapsedMillis":2310,"receivedBatches":3},"IntegrationANB.Terminals":{"documentsToCopy":299328,"documentsCopied":299328,"indexes":3,"fetchedBatches":4,"bytesToCopy":32138542,"approxBytesCopied":32028096,"start":{"$date":"2023-07-27T05:36:40.080Z"},"end":{"$date":"2023-07-27T05:36:42.808Z"},"elapsedMillis":2728,"receivedBatches":4},"IntegrationANB.CardFees":{"documentsToCopy":1,"documentsCopied":1,"indexes":1,"fetchedBatches":1,"bytesToCopy":147,"approxBytesCopied":147,"start":{"$date":"2023-07-27T05:36:42.808Z"},"end":{"$date":"2023-07-27T05:36:42.965Z"},"elapsedMillis":157,"receivedBatches":1},"IntegrationANB.POSReceipts":{"documentsToCopy":196282270,"documentsCopied":196282270,"indexes":16,"fetchedBatches":9728,"bytesToCopy":161910101965.0,"approxBytesCopied":161736590480.0,"start":{"$date":"2023-07-27T05:36:42.965Z"},"end":{"$date":"2023-07-27T09:38:00.475Z"},"elapsedMillis":14477510,"receivedBatches":9728},"IntegrationANB.TerminalHistory":{"documentsToCopy":0,"documentsCopied":0,"indexes":0,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2023-07-27T09:38:00.475Z"},"receivedBatches":0}},"IntegrationCommodity":{"collections":0,"clonedCollections":0},"IntegrationEmdha":{"collections":0,"clonedCollections":0},"IntegrationNafath":{"collections":0,"clonedCollections":0},"IntegrationNafith":{"collections":0,"clonedCollections":0},"IntegrationSP":{"collections":0,"clonedCollections":0},"IntegrationSadad":{"collections":0,"clonedCollections":0},"IntegrationSimah":{"collections":0,"clonedCollections":0},"IntegrationSoftPOS":{"collections":0,"clonedCollections":0},"NatheerDB":{"collections":0,"clonedCollections":0},"RayaDB":{"collections":0,"clonedCollections":0},"SAQR":{"collections":0,"clonedCollections":0},"WathqDB":{"collections":0,"clonedCollections":0},"BanksB2BProxy":{"collections":0,"clonedCollections":0},"config":{"collections":0,"clonedCollections":0},"debezium":{"collections":0,"clonedCollections":0},"like4cards":{"collections":0,"clonedCollections":0},"mpgs":{"collections":0,"clonedCollections":0},"one_card_db":{"collections":0,"clonedCollections":0},"test":{"collections":0,"clonedCollections":0}}}}}
{"t":{"$date":"2023-07-27T09:38:00.477+00:00"},"s":"E", "c":"INITSYNC", "id":21200, "ctx":"ReplCoordExtern-0","msg":"Initial sync attempt failed","attr":{"attemptsLeft":6,"error":"InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection 'IntegrationANB.TerminalHistory' :: caused by :: network error while attempting to run command 'collStats' on host '207.127.97.242:27017' "}}
{"t":{"$date":"2023-07-27T09:38:00.528+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.200:14831","connectionId":705872,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:00.529+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705872","msg":"Connection ended","attr":{"remote":"10.11.10.200:14831","connectionId":705872,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:00.534+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"CONNPOOL", "id":22572, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.10.0.49:27017","error":"HostUnreachable: Connection reset by peer"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"cfg","host":"10.10.0.49:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"10.10.0.41:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"10.10.0.49:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"cfg","host":"10.10.0.49:27017","delayMillis":0}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"8.213.16.23:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"10.10.0.170:27017","replicaSet":"cfg"}}
{"t":{"$date":"2023-07-27T09:38:00.599+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"10.10.0.49:27017"}}
{"t":{"$date":"2023-07-27T09:38:00.612+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.695+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.199:28889","connectionId":705876,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:00.695+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705876","msg":"Connection ended","attr":{"remote":"10.11.10.199:28889","connectionId":705876,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:00.739+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.881+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:00.982+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.158:21005","connectionId":705879,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:00.982+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705879","msg":"Connection ended","attr":{"remote":"10.11.10.158:21005","connectionId":705879,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:01.131+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.153+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.237+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.261+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.331+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.477+00:00"},"s":"I", "c":"INITSYNC", "id":21164, "ctx":"ReplCoordExtern-6","msg":"Starting initial sync attempt","attr":{"initialSyncAttempt":5,"initialSyncMaxAttempts":10}}
{"t":{"$date":"2023-07-27T09:38:01.477+00:00"},"s":"I", "c":"STORAGE", "id":20318, "ctx":"ReplCoordExtern-6","msg":"Finishing collection drop","attr":{"namespace":"local.temp_oplog_buffer","uuid":{"uuid":{"$uuid":"791f7513-12aa-4e4c-b73b-ec2c6948945a"}}}}
{"t":{"$date":"2023-07-27T09:38:01.480+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"ReplCoordExtern-6","msg":"createCollection","attr":{"namespace":"local.temp_oplog_buffer","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"863eadb5-cdac-417a-bc7d-b333c76d8a10"}},"options":{"temp":true}}}
{"t":{"$date":"2023-07-27T09:38:01.487+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"ReplCoordExtern-6","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.temp_oplog_buffer","index":"_id_","commitTimestamp":{"$timestamp":{"t":1690450679,"i":7}}}}
{"t":{"$date":"2023-07-27T09:38:01.487+00:00"},"s":"I", "c":"REPL", "id":21799, "ctx":"ReplCoordExtern-6","msg":"Sync source candidate chosen","attr":{"syncSource":"207.127.97.123:27017"}}
{"t":{"$date":"2023-07-27T09:38:01.488+00:00"},"s":"I", "c":"INITSYNC", "id":21173, "ctx":"ReplCoordExtern-6","msg":"Initial syncer oplog truncation finished","attr":{"durationMillis":0}}
{"t":{"$date":"2023-07-27T09:38:01.488+00:00"},"s":"I", "c":"STORAGE", "id":4833200, "ctx":"ReplCoordExtern-6","msg":"About to abort all index builders running","attr":{"reason":"Aborting index builds for initial sync"}}
{"t":{"$date":"2023-07-27T09:38:01.488+00:00"},"s":"I", "c":"REPL", "id":21754, "ctx":"ReplCoordExtern-6","msg":"dropReplicatedDatabases - dropping databases","attr":{"numDatabases":5}}
{"t":{"$date":"2023-07-27T09:38:01.514+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.524+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.627+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.692+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.719+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.746+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.221:17643","connectionId":705890,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:01.746+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn705890","msg":"client metadata","attr":{"remote":"10.11.10.221:17643","client":"conn705890","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.19.6"}}}
{"t":{"$date":"2023-07-27T09:38:01.749+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.203:48911","connectionId":705891,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:01.749+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn705891","msg":"client metadata","attr":{"remote":"10.11.10.203:48911","client":"conn705891","doc":{"driver":{"name":"mongo-go-driver","version":"v1.11.2"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.19.6"}}}
{"t":{"$date":"2023-07-27T09:38:01.818+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.821+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:01.982+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}

{"t":{"$date":"2023-07-27T09:38:02.121+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.136+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.136+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.195+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.267+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.297+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.334+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.335+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.446+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.528+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.544+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.567+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.604+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.783+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.817+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:02.953+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.055+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.182+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.229+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.204:16691","connectionId":705913,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.229+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705913","msg":"Connection ended","attr":{"remote":"10.11.10.204:16691","connectionId":705913,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.250+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.261+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.16:11367","connectionId":705915,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.261+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705915","msg":"Connection ended","attr":{"remote":"10.11.10.16:11367","connectionId":705915,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.367+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.30.20.1:41822","connectionId":705916,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.367+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705916","msg":"Connection ended","attr":{"remote":"172.30.20.1:41822","connectionId":705916,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.528+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.540+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.541+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.30.20.1:61331","connectionId":705919,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.541+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705919","msg":"Connection ended","attr":{"remote":"172.30.20.1:61331","connectionId":705919,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.633+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 9:48am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/16 "2023-07-27T09:48:07Z")

</div>

```auto
{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.656+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.739+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.745+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.203:58496","connectionId":705923,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:03.745+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705923","msg":"Connection ended","attr":{"remote":"10.11.10.203:58496","connectionId":705923,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:03.896+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:03.945+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.041+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.147+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.196+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.201+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.24:42210","connectionId":705929,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:04.201+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705929","msg":"Connection ended","attr":{"remote":"10.11.10.24:42210","connectionId":705929,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:04.328+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.352+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.358+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.200:59054","connectionId":705932,"connectionCount":36}}
{"t":{"$date":"2023-07-27T09:38:04.358+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705932","msg":"Connection ended","attr":{"remote":"10.11.10.200:59054","connectionId":705932,"connectionCount":35}}
{"t":{"$date":"2023-07-27T09:38:04.523+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.623+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.628+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.644+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.647+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.668+00:00"},"s":"I", "c":"REPL", "id":21756, "ctx":"ReplCoordExtern-6","msg":"dropReplicatedDatabases - dropped databases","attr":{"numDatabases":5}}
{"t":{"$date":"2023-07-27T09:38:04.668+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplCoordExternNetwork","msg":"Connecting","attr":{"hostAndPort":"207.127.97.123:27017"}}
{"t":{"$date":"2023-07-27T09:38:04.669+00:00"},"s":"I", "c":"COMMAND", "id":20499, "ctx":"ftdc","msg":"serverStatus was very slow","attr":{"timeStats":{"after basic":0,"after asserts":0,"after connections":0,"after electionMetrics":0,"after extra_info":0,"after flowControl":0,"after globalLock":0,"after indexBulkBuilder":0,"after locks":0,"after logicalSessionRecordCache":0,"after mirroredReads":0,"after network":0,"after opLatencies":0,"after opReadConcernCounters":0,"after opcounters":0,"after opcountersRepl":0,"after oplog":2668,"after oplogTruncation":2668,"after repl":2668,"after scramCache":2668,"after security":2668,"after shardingStatistics":2668,"after storageEngine":2668,"after tcmalloc":2668,"after trafficRecording":2668,"after transactions":2668,"after transportSecurity":2668,"after twoPhaseCommitCoordinator":2668,"after wiredTiger":2668,"at end":2668}}}
{"t":{"$date":"2023-07-27T09:38:04.756+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705890","msg":"Connection ended","attr":{"remote":"10.11.10.221:17643","connectionId":705890,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:04.757+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn705891","msg":"Interrupted operation as its client disconnected","attr":{"opId":1201798}}
{"t":{"$date":"2023-07-27T09:38:04.757+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705891","msg":"Connection ended","attr":{"remote":"10.11.10.203:48911","connectionId":705891,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:04.821+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.879+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.972+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:04.980+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.200+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.236+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.254+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.338+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.446+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.488+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.500+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.626+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.675+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.686+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.11.10.203:29068","connectionId":705954,"connectionCount":34}}
{"t":{"$date":"2023-07-27T09:38:05.686+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn705954","msg":"Connection ended","attr":{"remote":"10.11.10.203:29068","connectionId":705954,"connectionCount":33}}
{"t":{"$date":"2023-07-27T09:38:05.798+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}
{"t":{"$date":"2023-07-27T09:38:05.949+00:00"},"s":"W", "c":"NETWORK", "id":23023, "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Transport endpoint is not connected"}}}

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 1:32pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/17 "2023-07-27T13:32:37Z")

</div>

this is also from the pod which the intialsync fails:

```auto
{"t":{"$date":"2023-07-27T12:24:28.693+00:00"},"s":"I", "c":"REPL_HB", "id":23974, "ctx":"ReplCoord-195","msg":"Heartbeat failed after max retries","attr":{"target":"x.x.x.x:27017","maxHeartbeatRetries":2,"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"}}}

```

“status” : "InitialSyncFailure: error cloning databases :: caused by :: HostUnreachable: Error cloning collection ‘Integration.History’ :: caused by :: network error while attempting to run command ‘collStats’ on host ‘x.x.x.x:27017’ ",

```auto

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 27, 2023, 1:33pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/18 "2023-07-27T13:33:31Z")

</div>

Hi @Santosh_Varma I really appreciate your help!

---

<div class="post-metadata">

**Author:** ![Santosh\_Varma](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/santosh_varma/32/6924_2.png) [@Santosh\_Varma](https://forums.percona.com/u/Santosh_Varma)\
**Post date:** [July 27, 2023, 2:39pm UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/19 "2023-07-27T14:39:37Z")

</div>

Hi Ahmed,

Going by the logs,  
looks like the destination pod is not able to detect primary (its sync source )

{“t”:{“$date”:“2023-07-27T08:35:06.336+00:00”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“x.x.x.x:27017”,“replicaSet”:“rs0”}}

is existing replicaset outside k8? are there anyother pods which are member of the set rs0 and secondary?

How are you adding the new node? with the endpoint ip and port or with cluster ip and port into existing replicaset?

when intialy sycn fails doe the primary change?

---

<div class="post-metadata">

**Author:** ![Ahmed\_Asim](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ahmed_asim/32/11499_2.png) [@Ahmed\_Asim](https://forums.percona.com/u/Ahmed_Asim)\
**Post date:** [July 31, 2023, 12:32am UTC](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922/20 "2023-07-31T00:32:40Z")

</div>

Hi @Santosh_Varma

I can say finally I’m able to identify the issue which is a slow query on that particular collection each time it reports the same error and I’m sure this is issue because I compare the connectionID on both sites and it’s the same and after some time the connection gets ended, please advice how can we fix this

Thanks  
Ahmed Asim

[Next page](https://forums.percona.com/t/mongodb-intialsync-failed-at-specific-db-collection/23922.md?page=2)