Replicaset configuration error for in memory configuration

jerowejerowe EntrantActive Member Participant
Hello,

I am trying to implement the mongodb replicaset configuration here - https://docs.mongodb.com/manual/core...y/#replica-set

With 3 replicas. The primary and second replica are both in memory. The 3rd is using WiredTiger.

However, when I restart the cluster the data does not persist, and I get this error on startup. I am deploying with kubernetes using the official mongodb replicaset helm chart (https://github.com/helm/charts/tree/...aset/templates) with the percona mongodb image.


Server has startup warnings: 2020-03-19T12:55:06.993+0000 I REPL [conn6] 2020-03-19T12:55:06.993+0000 I REPL [conn6] ** WARNING: This replica set node is running without journaling enabled but the 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** is set to true. The writeConcernMajorityJournalDefault 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** option to the replica set config must be set to false 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete. 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node's memory consumption may increase until all 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted. 2020-03-19T12:55:06.994+0000 I REPL [conn6] 2020-03-19T12:55:06.994+0000 I REPL [conn6] 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** WARNING: This replica set node is using in-memory (ephemeral) storage with the 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** set to true. The writeConcernMajorityJournalDefault option to the 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** replica set config must be set to false 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete. 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node's memory consumption may increase until all 2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted. 2020-03-19T12:55:06.994+0000 I REPL [conn6]
rs0:PRIMARY> cfg = rs.conf() { "_id" : "rs0", "version" : 102744, "protocolVersion" : NumberLong(1), "writeConcernMajorityJournalDefault" : false, "members" : [ { "_id" : 0, "host" : "mongodb-mongodb-replicaset-0.mongodb-mongodb-replicaset.default.svc.cluster.local:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 1, "host" : "mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 2, "host" : "mongodb-mongodb-replicaset-2.mongodb-mongodb-replicaset.default.svc.cluster.local:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : true, "priority" : 0, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatIntervalMillis" : 2000, "heartbeatTimeoutSecs" : 10, "electionTimeoutMillis" : 10000, "catchUpTimeoutMillis" : -1, "catchUpTakeoverDelayMillis" : 30000, "getLastErrorModes" : { }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 }, "replicaSetId" : ObjectId("5e736baa5c51ff3e198dd880") } }

The writeConcernMajorityJournalDefault IS set to false.

I would very much appreciate any help, and am happy to contribute back the helm chart for anyone else who wants this configuration!

Comments

  • Akira KuroganeAkira Kurogane Percona Percona Advisor
    Repasting for readability.
    2020-03-19T12:55:06.993+0000 I REPL [conn6]
    2020-03-19T12:55:06.993+0000 I REPL [conn6] ** WARNING: This replica set node is running without journaling enabled but the
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** is set to true. The writeConcernMajorityJournalDefault
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** option to the replica set config must be set to false
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete.
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node's memory consumption may increase until all
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted.
    2020-03-19T12:55:06.994+0000 I REPL [conn6]
    2020-03-19T12:55:06.994+0000 I REPL [conn6]
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** WARNING: This replica set node is using in-memory (ephemeral) storage with the
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** writeConcernMajorityJournalDefault option to the replica set config
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** set to true. The writeConcernMajorityJournalDefault option to the
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** replica set config must be set to false
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** or w:majority write concerns will never complete.
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** In addition, this node's memory consumption may increase until all
    2020-03-19T12:55:06.994+0000 I REPL [conn6] ** available free RAM is exhausted.
    2020-03-19T12:55:06.994+0000 I REPL [conn6]
  • Akira KuroganeAkira Kurogane Percona Percona Advisor
    //Repasting again for readability.
    {
    "_id" : "rs0",
    "version" : 102744,
    "protocolVersion" : NumberLong(1),
    "writeConcernMajorityJournalDefault" : false,
    "members" : [
    {
    "_id" : 0,
    "host" : "mongodb-mongodb-replicaset-0.mongodb-mongodb-replicaset.default.svc.cluster.local:27017",
    "arbiterOnly" : false,
    "buildIndexes" : true,
    "hidden" : false,
    "priority" : 1,
    "tags" : {

    },
    "slaveDelay" : NumberLong(0),
    "votes" : 1
    },
    {
    "_id" : 1,
    "host" : "mongodb-mongodb-replicaset-1.mongodb-mongodb-replicaset.default.svc.cluster.local:27017",
    "arbiterOnly" : false,
    "buildIndexes" : true,
    "hidden" : false,
    "priority" : 1,
    "tags" : {

    },
    "slaveDelay" : NumberLong(0),
    "votes" : 1
    },
    {
    "_id" : 2,
    "host" : "mongodb-mongodb-replicaset-2.mongodb-mongodb-replicaset.default.svc.cluster.local:27017",
    "arbiterOnly" : false,
    "buildIndexes" : true,
    "hidden" : true,
    "priority" : 0,
    "tags" : {

    },
    "slaveDelay" : NumberLong(0),
    "votes" : 1
    }
    ],
    "settings" : {
    "chainingAllowed" : true,
    "heartbeatIntervalMillis" : 2000,
    "heartbeatTimeoutSecs" : 10,
    "electionTimeoutMillis" : 10000,
    "catchUpTimeoutMillis" : -1,
    "catchUpTakeoverDelayMillis" : 30000,
    "getLastErrorModes" : {

    },
    "getLastErrorDefaults" : {
    "w" : 1,
    "wtimeout" : 0
    },
    "replicaSetId" : ObjectId("5e736baa5c51ff3e198dd880")
    }
    }
  • Akira KuroganeAkira Kurogane Percona Percona Advisor
    Hi Jerowe.

    I can reproduce this.

    Step 1. Start with a normal 3-node replicaset. I'm using v4.2.0-1 of Percona Server for MongoDB but I believe it will be exactly the same for v4.0, and still very likely the same for v3.6 too. (inMemory is supported from v3.2 but tunable write consistency is a factor probably and that was only added in v3.6.)

    Step 2. Change the config file as you did, i.e. make the "storage" something like this:

    ...
    storage:
    dbPath: /data/plain_rs/n1/data
    engine: inMemory
    inMemory:
    engineConfig:
    inMemorySizeGB: 2
    ...

    (N.b. you will have to clear the dbpath directory to restart - the mongod process will object and quit if it sees another storage engine's files in there. What goes there instead with the inMemory engine in Percona Server for MongoDB's case is "storage.bson", "mongod.lock", and the FTDC data in the "diagnostic.data/metrics.*" files.)

    Note: no changing the rs.conf() yet. The writeConcernMajorityJournalDefault value is still the normal default, true.

    Step 3. Restart a node. In early stages of doing initial sync it will have the "This replica set is using in-memory (ephemeral) storage with the writeConcernMajorityJournalDefault option to the replica set config set to true ..." informational message in the log. The mongo shell will also display this same message you connect to the node.

    (A second node can be restarted in inMemory engine mode; same result for it.)

    Step 4. Connect to the primary, reconfig the rs config:

    testrs:PRIMARY> var rsc = rs.conf()
    testrs:PRIMARY> rsc.writeConcernMajorityJournalDefault
    true
    testrs:PRIMARY> rsc.writeConcernMajorityJournalDefault = false
    false
    testrs:PRIMARY> rs.reconfig(rsc)

    Then log out, reconnect.

    The same startup warnings with the "This replica set is using in-memory (ephemeral) storage with the writeConcernMajorityJournalDefault option to the replica set config set to true ..." message appear in the mongo shell. (N.b. these are *startup log* warnings.)

    --- (End of reproduction) ---

    Resolution: Restart the mongod node again (after writeConcernMajorityJournalDefault was set to false). The warning message will then be gone from the log.

    You were doing what was necessary with writeConcernMajorityJournalDefault. It's just a matter of the mongo shell using the { getLog : "startupWarnings" } command, and that's a cache of the situation at the time of the last restart.

    Akira
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.