As far as I know that an arbiter does not contain user data so that it does not require much memory for operational. Here are my questions.
Hi, on my test env running mongo 8 the arbiter consumes around 150 MB so I guess your value is reasonable. Be aware Mongo 4.4 is very old and unsupported, you should be migrating to Mongo 6.0 by now at least.
Thanks for your quick response. 
But seems minimal value of wiredTiger engine’s cache size is about 256MB. So, I guess it would be problematic. I’m running the MongoDB in Kubernetes environment with 300MB memory limit. Could you guide me what is optimal value of memory limit for arbiter?
keep in mind arbiter nodes don’t allocate wiredtiger cache since they don’t store any data
Yes. An arbiter node never holds user data but I have some doubts.
Even thought an arbiter doesn’t hold user data, I can see below values from db.serverStatus().wiredTiger.cache command result. Please take a look at below.
rs2:ARBITER> db.serverStatus().wiredTiger.cache
{
"application threads page read from disk to cache count" : 0,
"application threads page read from disk to cache time (usecs)" : 0,
"application threads page write from cache to disk count" : 5708,
"application threads page write from cache to disk time (usecs)" : 311221,
"bytes allocated for updates" : 76891,
"bytes belonging to page images in the cache" : 0,
"bytes belonging to the history store table in the cache" : 182,
"bytes currently in the cache" : 84866,
"bytes dirty in the cache cumulative" : 156232619,
"bytes not belonging to page images in the cache" : 84866,
"bytes read into cache" : 0,
"bytes written from cache" : 80667531,
"cache overflow score" : 0,
"checkpoint blocked page eviction" : 0,
"checkpoint of history store file blocked non-history store page eviction" : 0,
"eviction calls to get a page" : 10,
"eviction calls to get a page found queue empty" : 10,
"eviction calls to get a page found queue empty after locking" : 0,
"eviction currently operating in aggressive mode" : 0,
"eviction empty score" : 0,
"eviction gave up due to detecting an out of order on disk value behind the last update on the chain" : 0,
"eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update" : 0,
"eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update after validating the update chain" : 0,
"eviction gave up due to detecting out of order timestamps on the update chain after the selected on disk update" : 0,
"eviction passes of a file" : 0,
"eviction server candidate queue empty when topping up" : 0,
"eviction server candidate queue not empty when topping up" : 0,
"eviction server evicting pages" : 0,
"eviction server slept, because we did not make progress with eviction" : 0,
"eviction server unable to reach eviction goal" : 0,
"eviction server waiting for a leaf page" : 19,
"eviction state" : 64,
"eviction walk most recent sleeps for checkpoint handle gathering" : 0,
"eviction walk target pages histogram - 0-9" : 0,
"eviction walk target pages histogram - 10-31" : 0,
"eviction walk target pages histogram - 128 and higher" : 0,
"eviction walk target pages histogram - 32-63" : 0,
"eviction walk target pages histogram - 64-128" : 0,
"eviction walk target pages reduced due to history store cache pressure" : 0,
"eviction walk target strategy both clean and dirty pages" : 0,
"eviction walk target strategy only clean pages" : 0,
"eviction walk target strategy only dirty pages" : 0,
"eviction walks abandoned" : 0,
"eviction walks gave up because they restarted their walk twice" : 0,
"eviction walks gave up because they saw too many pages and found no candidates" : 0,
"eviction walks gave up because they saw too many pages and found too few candidates" : 0,
"eviction walks reached end of tree" : 0,
"eviction walks restarted" : 0,
"eviction walks started from root of tree" : 0,
"eviction walks started from saved location in tree" : 0,
"eviction worker thread active" : 4,
"eviction worker thread created" : 0,
"eviction worker thread evicting pages" : 0,
"eviction worker thread removed" : 0,
"eviction worker thread stable number" : 0,
"files with active eviction walks" : 0,
"files with new eviction walks started" : 0,
"force re-tuning of eviction workers once in a while" : 0,
"forced eviction - history store pages failed to evict while session has history store cursor open" : 0,
"forced eviction - history store pages selected while session has history store cursor open" : 0,
"forced eviction - history store pages successfully evicted while session has history store cursor open" : 0,
"forced eviction - pages evicted that were clean count" : 0,
"forced eviction - pages evicted that were clean time (usecs)" : 0,
"forced eviction - pages evicted that were dirty count" : 0,
"forced eviction - pages evicted that were dirty time (usecs)" : 0,
"forced eviction - pages selected because of a large number of updates to a single item" : 0,
"forced eviction - pages selected because of too many deleted items count" : 0,
"forced eviction - pages selected count" : 0,
"forced eviction - pages selected unable to be evicted count" : 0,
"forced eviction - pages selected unable to be evicted time" : 0,
"hazard pointer blocked page eviction" : 0,
"hazard pointer check calls" : 0,
"hazard pointer check entries walked" : 0,
"hazard pointer maximum array length" : 0,
"history store score" : 0,
"history store table insert calls" : 0,
"history store table insert calls that returned restart" : 0,
"history store table max on-disk size" : 0,
"history store table on-disk size" : 4096,
"history store table out-of-order resolved updates that lose their durable timestamp" : 0,
"history store table out-of-order updates that were fixed up by reinserting with the fixed timestamp" : 0,
"history store table reads" : 0,
"history store table reads missed" : 0,
"history store table reads requiring squashed modifies" : 0,
"history store table truncation by rollback to stable to remove an unstable update" : 0,
"history store table truncation by rollback to stable to remove an update" : 0,
"history store table truncation to remove an update" : 0,
"history store table truncation to remove range of updates due to key being removed from the data page during reconciliation" : 0,
"history store table truncation to remove range of updates due to out-of-order timestamp update on data page" : 0,
"history store table writes requiring squashed modifies" : 0,
"in-memory page passed criteria to be split" : 0,
"in-memory page splits" : 0,
"internal pages evicted" : 0,
"internal pages queued for eviction" : 0,
"internal pages seen by eviction walk" : 0,
"internal pages seen by eviction walk that are already queued" : 0,
"internal pages split during eviction" : 0,
"leaf pages split during eviction" : 0,
"maximum bytes configured" : 268435456,
"maximum page size at eviction" : 0,
"modified pages evicted" : 0,
"modified pages evicted by application threads" : 0,
"operations timed out waiting for space in cache" : 0,
"overflow pages read into cache" : 0,
"page split during eviction deepened the tree" : 0,
"page written requiring history store records" : 0,
"pages currently held in the cache" : 29,
"pages evicted by application threads" : 0,
"pages evicted in parallel with checkpoint" : 0,
"pages queued for eviction" : 0,
"pages queued for eviction post lru sorting" : 0,
"pages queued for urgent eviction" : 0,
"pages queued for urgent eviction during walk" : 0,
"pages queued for urgent eviction from history store due to high dirty content" : 0,
"pages read into cache" : 0,
"pages read into cache after truncate" : 13,
"pages read into cache after truncate in prepare state" : 0,
"pages requested from the cache" : 60022,
"pages seen by eviction walk" : 0,
"pages seen by eviction walk that are already queued" : 0,
"pages selected for eviction unable to be evicted" : 0,
"pages selected for eviction unable to be evicted as the parent page has overflow items" : 0,
"pages selected for eviction unable to be evicted because of active children on an internal page" : 0,
"pages selected for eviction unable to be evicted because of failure in reconciliation" : 0,
"pages selected for eviction unable to be evicted because of race between checkpoint and out of order timestamps handling" : 0,
"pages walked for eviction" : 0,
"pages written from cache" : 5708,
"pages written requiring in-memory restoration" : 0,
"percentage overhead" : 8,
"the number of times full update inserted to history store" : 0,
"the number of times reverse modify inserted to history store" : 0,
"tracked bytes belonging to internal pages in the cache" : 6739,
"tracked bytes belonging to leaf pages in the cache" : 78127,
"tracked dirty bytes in the cache" : 0,
"tracked dirty pages in the cache" : 0,
"unmodified pages evicted" : 0
}
I guess that if an arbiter never interact with wiredTiger’s cache, most of above values should be zero but not, especially below items.
"bytes allocated for updates" : 76891,
"bytes dirty in the cache cumulative" : 156232619,
...
"bytes written from cache" : 80667531,
...
"maximum bytes configured" : 268435456,
Could you give your opinions why the items are having non-zero values?
That is interesting, so based on your observations it seems some memory is indeed allocated for WT cache then. I think in the case of arbiters, the minimum of 256 Mb does not apply though.
Then, do you think the cache of wiredTiger for arbiter can allocate memory up to 256MB and keep it?
In fact, the arbiter which running in my production environment is using about 204.8MB memory.
Please refer to below result.
Percona Server for MongoDB shell version v4.4.17-17
connecting to: mongodb://localhost:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("100dec9e-b5d6-4958-bb4c-f175ec928d7c") }
Percona Server for MongoDB server version: v4.4.17-17
------------------------------------------------
MALLOC: 214746992 ( 204.8 MiB) Bytes in use by application
MALLOC: + 7553024 ( 7.2 MiB) Bytes in page heap freelist
MALLOC: + 904888 ( 0.9 MiB) Bytes in central cache freelist
MALLOC: + 2001472 ( 1.9 MiB) Bytes in transfer cache freelist
MALLOC: + 4870040 ( 4.6 MiB) Bytes in thread cache freelists
MALLOC: + 3932160 ( 3.8 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 234008576 ( 223.2 MiB) Actual memory used (physical + swap)
MALLOC: + 4472832 ( 4.3 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 238481408 ( 227.4 MiB) Virtual address space used
MALLOC:
MALLOC: 13913 Spans in use
MALLOC: 163 Thread heaps in use
MALLOC: 4096 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
Also, I found a document that describing some behavior of wiredTiger cache. See below.
The eviction_target configuration value (default 80%) is the level at which WiredTiger attempts to keep the overall cache usage. Eviction worker threads are active when the cache contains at least this much content, expressed as a percentage of the total cache size.
So, I can calculate a value with above data: 256MB*0.8=204.8MB.
So, I guess that the cache of wiredTiger for arbiter is keeping memory for some reason up to 204.8MB.
What do you think above that? Do you think it makes sense?
@Ivan_Groenewold Hello. Do you happen to have any opinions regarding above my assumption?
Hi, I think your assumptions make sense.
Unanswered | Unsolved | Solved
MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2024 Percona LLC. All rights reserved.