Let the PCSM initial clone read from a secondary or hidden member

Hi all,

We use PCSM to move a large collection (several TB) from one replica set to another. The tool works well, but we’ve hit a limitation that keeps the migration from completing safely.

The problem

The initial clone always reads from the source primary. With a multi-TB collection, the clone runs for days and pushes a lot of cold data through the primary’s WiredTiger cache. On a busy, memory-bound primary we saw:

  • cache usage staying above the eviction target for the whole clone,
  • and, at normal daily traffic peaks, application threads pulled into eviction, which showed up as stalled writes and timeouts for the applications using that cluster.

Fewer read workers helped, but didn’t fix it. A clone of several days still crosses several traffic peaks, it can’t be paused, and an aborted clone starts from zero. So for us, migrating a large collection off a busy primary isn’t possible today without risking production impact.

What would help

An optional, separate connection string used only for clone reads, for example --source-clone-uri / PCSM_SOURCE_CLONE_URI. It would accept directConnection=true or a readPreference, so the clone can read from a secondary or a hidden member, such as a dedicated backup node that serves no application traffic.

Everything that needs the primary or is cheap there would stay as it is: the appendOplogNote start anchor, the change stream, and metadata reads.

One correctness detail

The clone start point is taken on the primary. A slightly lagging secondary may not have applied the last writes before that point yet. To avoid missing them, clone reads on the secondary would need a causally consistent session with readConcern: majority and afterClusterTime set to the start timestamp. Delayed members should be rejected.

Related

A rate limit, or pause/resume during the clone phase, would also help with the same problem.

Is this something you’d consider? We’d be glad to discuss the design, and we’re open to contributing a PR if the approach makes sense to you.

Thanks!

Hi @Abhishek_Kadam1,

Thanks for sharing this use case. You are right, PCSM’s initial clone always reads from the primary. A readPreference or directConnection in the source URI is ignored, so reading from a secondary or hidden member isn’t possible today (see Limitations). The clone phase also can’t be paused or resumed: pause/resume only apply once real-time replication has started. For now, lowering --clone-num-read-workers is the main way to reduce read load on the source, though as you saw, it reduces the impact rather than removing it.

The approach sounds reasonable, including the afterClusterTime guard. One suggestion, use a hidden member with votes: 0 for the clone reads. If a voting secondary that the majority depends on falls behind, it can trigger flow control on the primary and bring back the impact you’re trying to avoid.

I have opened a feature request for this so the team can track it: https://perconadev.atlassian.net/browse/PCSM-420. Feel free to watch it or add details there.

PRs are welcome too: percona-clustersync-mongodb/CONTRIBUTING.md at main · percona/percona-clustersync-mongodb · GitHub