Hi all,
We use PCSM to move a large collection (several TB) from one replica set to another. The tool works well, but we’ve hit a limitation that keeps the migration from completing safely.
The problem
The initial clone always reads from the source primary. With a multi-TB collection, the clone runs for days and pushes a lot of cold data through the primary’s WiredTiger cache. On a busy, memory-bound primary we saw:
- cache usage staying above the eviction target for the whole clone,
- and, at normal daily traffic peaks, application threads pulled into eviction, which showed up as stalled writes and timeouts for the applications using that cluster.
Fewer read workers helped, but didn’t fix it. A clone of several days still crosses several traffic peaks, it can’t be paused, and an aborted clone starts from zero. So for us, migrating a large collection off a busy primary isn’t possible today without risking production impact.
What would help
An optional, separate connection string used only for clone reads, for example --source-clone-uri / PCSM_SOURCE_CLONE_URI. It would accept directConnection=true or a readPreference, so the clone can read from a secondary or a hidden member, such as a dedicated backup node that serves no application traffic.
Everything that needs the primary or is cheap there would stay as it is: the appendOplogNote start anchor, the change stream, and metadata reads.
One correctness detail
The clone start point is taken on the primary. A slightly lagging secondary may not have applied the last writes before that point yet. To avoid missing them, clone reads on the secondary would need a causally consistent session with readConcern: majority and afterClusterTime set to the start timestamp. Delayed members should be rejected.
Related
A rate limit, or pause/resume during the clone phase, would also help with the same problem.
Is this something you’d consider? We’d be glad to discuss the design, and we’re open to contributing a PR if the approach makes sense to you.
Thanks!