Replicaset configuration error for in memory configuration

MongoDB Percona Backup for MongoDB
1

Hello,

I am trying to implement the mongodb replicaset configuration here - [URL=“https://docs.mongodb.com/manual/core/inmemory/#replica-set”]https://docs.mongodb.com/manual/core...y/#replica-set[/URL]

With 3 replicas. The primary and second replica are both in memory. The 3rd is using WiredTiger.

However, when I restart the cluster the data does not persist, and I get this error on startup. I am deploying with kubernetes using the official mongodb replicaset helm chart ([URL=“charts/stable/mongodb-replicaset/templates at master · helm/charts · GitHub”]https://github.com/helm/charts/tree/...aset/templates[/URL]) with the percona mongodb image.

Server has startup warnings: 2020-03-19T12:55:06.993+0000 I REPL [conn6] 2020-03-19T12:55:06.993+0000 I REPL [conn6] ** WARNING: This replica set node is running without journaling enabled but the 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** is set to true. The writeConcernMajorityJournalDefault 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** option to the replica set config must be set to false 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete. 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node’s memory consumption may increase until all 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted. 2020-03-19T12:55:06.994+0000 I REPL [conn6] 2020-03-19T12:55:06.994+0000 I REPL [conn6] 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** WARNING: This replica set node is using in-memory (ephemeral) storage with the 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** set to true. The writeConcernMajorityJournalDefault option to the 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** replica set config must be set to false 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete. 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node’s memory consumption may increase until all 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted. 2020-03-19T12:55:06.994+0000 I REPL [conn6]
rs0:PRIMARY> cfg = rs.conf() { “_id” : “rs0”, “version” : 102744, “protocolVersion” : NumberLong(1), “writeConcernMajorityJournalDefault” : false, “members” : [ { “_id” : 0, “host” : “mongodb-mongodb-replicaset-0.mongodb-mongodb-replicaset.default.svc.cluster.local:27017”, “arbiterOnly” : false, “buildIndexes” : true, “hidden” : false, “priority” : 1, “tags” : { }, “slaveDelay” : NumberLong(0), “votes” : 1 }, { “_id” : 1, “host” : “mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local:27017”, “arbiterOnly” : false, “buildIndexes” : true, “hidden” : false, “priority” : 1, “tags” : { }, “slaveDelay” : NumberLong(0), “votes” : 1 }, { “_id” : 2, “host” : “mongodb-mongodb-replicaset-2.mongodb-mongodb-replicaset.default.svc.cluster.local:27017”, “arbiterOnly” : false, “buildIndexes” : true, “hidden” : true, “priority” : 0, “tags” : { }, “slaveDelay” : NumberLong(0), “votes” : 1 } ], “settings” : { “chainingAllowed” : true, “heartbeatIntervalMillis” : 2000, “heartbeatTimeoutSecs” : 10, “electionTimeoutMillis” : 10000, “catchUpTimeoutMillis” : -1, “catchUpTakeoverDelayMillis” : 30000, “getLastErrorModes” : { }, “getLastErrorDefaults” : { “w” : 1, “wtimeout” : 0 }, “replicaSetId” : ObjectId(“5e736baa5c51ff3e198dd880”) } }

The writeConcernMajorityJournalDefault IS set to false.

I would very much appreciate any help, and am happy to contribute back the helm chart for anyone else who wants this configuration!

2

Repasting for readability.
2020-03-19T12:55:06.993+0000 I REPL [conn6]
2020-03-19T12:55:06.993+0000 I REPL [conn6] ** WARNING: This replica set node is running without journaling enabled but the
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** is set to true. The writeConcernMajorityJournalDefault
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** option to the replica set config must be set to false
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete.
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node’s memory consumption may increase until all
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted.
2020-03-19T12:55:06.994+0000 I REPL [conn6]
2020-03-19T12:55:06.994+0000 I REPL [conn6]
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** WARNING: This replica set node is using in-memory (ephemeral) storage with the
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** set to true. The writeConcernMajorityJournalDefault option to the
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** replica set config must be set to false
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete.
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node’s memory consumption may increase until all
2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted.
2020-03-19T12:55:06.994+0000 I REPL [conn6]

3

//Repasting again for readability.
{
“_id” : “rs0”,
“version” : 102744,
“protocolVersion” : NumberLong(1),
“writeConcernMajorityJournalDefault” : false,
“members” : [
{
“_id” : 0,
“host” : “mongodb-mongodb-replicaset-0.mongodb-mongodb-replicaset.default.svc.cluster.local:27017”,
“arbiterOnly” : false,
“buildIndexes” : true,
“hidden” : false,
“priority” : 1,
“tags” : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	},
	{
		"_id" : 1,
		"host" : "mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local:27017",
		"arbiterOnly" : false,
		"buildIndexes" : true,
		"hidden" : false,
		"priority" : 1,
		"tags" : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	},
	{
		"_id" : 2,
		"host" : "mongodb-mongodb-replicaset-2.mongodb-mongodb-replicaset.default.svc.cluster.local:27017",
		"arbiterOnly" : false,
		"buildIndexes" : true,
		"hidden" : true,
		"priority" : 0,
		"tags" : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	}
],
"settings" : {
	"chainingAllowed" : true,
	"heartbeatIntervalMillis" : 2000,
	"heartbeatTimeoutSecs" : 10,
	"electionTimeoutMillis" : 10000,
	"catchUpTimeoutMillis" : -1,
	"catchUpTakeoverDelayMillis" : 30000,
	"getLastErrorModes" : {

	},
	"getLastErrorDefaults" : {
		"w" : 1,
		"wtimeout" : 0
	},
	"replicaSetId" : ObjectId("5e736baa5c51ff3e198dd880")
}

}

4

Hi Jerowe.

I can reproduce this.

Step 1. Start with a normal 3-node replicaset. I’m using v4.2.0-1 of Percona Server for MongoDB but I believe it will be exactly the same for v4.0, and still very likely the same for v3.6 too. (inMemory is supported from v3.2 but tunable write consistency is a factor probably and that was only added in v3.6.)

Step 2. Change the config file as you did, i.e. make the “storage” something like this:

…
storage:
dbPath: /data/plain_rs/n1/data
engine: inMemory
inMemory:
engineConfig:
inMemorySizeGB: 2
…

(N.b. you will have to clear the dbpath directory to restart - the mongod process will object and quit if it sees another storage engine’s files in there. What goes there instead with the inMemory engine in Percona Server for MongoDB’s case is “storage.bson”, “mongod.lock”, and the FTDC data in the “diagnostic.data/metrics.*” files.)

Note: no changing the rs.conf() yet. The writeConcernMajorityJournalDefault value is still the normal default, true.

Step 3. Restart a node. In early stages of doing initial sync it will have the “This replica set is using in-memory (ephemeral) storage with the writeConcernMajorityJournalDefault option to the replica set config set to true …” informational message in the log. The mongo shell will also display this same message you connect to the node.

(A second node can be restarted in inMemory engine mode; same result for it.)

Step 4. Connect to the primary, reconfig the rs config:

testrs:PRIMARY> var rsc = rs.conf()
testrs:PRIMARY> rsc.writeConcernMajorityJournalDefault
true
testrs:PRIMARY> rsc.writeConcernMajorityJournalDefault = false
false
testrs:PRIMARY> rs.reconfig(rsc)

Then log out, reconnect.

The same startup warnings with the “This replica set is using in-memory (ephemeral) storage with the writeConcernMajorityJournalDefault option to the replica set config set to true …” message appear in the mongo shell. (N.b. these are startup log warnings.)

— (End of reproduction) —

Resolution: Restart the mongod node again (after writeConcernMajorityJournalDefault was set to false). The warning message will then be gone from the log.

You were doing what was necessary with writeConcernMajorityJournalDefault. It’s just a matter of the mongo shell using the { getLog : “startupWarnings” } command, and that’s a cache of the situation at the time of the last restart.

Akira