Memory usage of InMemory Storage Engine

MongoDB Percona Server for MongoDB
1

I have tested MongoDB InMemory storage engine.

Each document is about 50~100 bytes and there’s only 50000 document in the collection (Just one collection in mongodb instance).
And there’s 3 indexes( PK, TTL index, and composite 2dsphere index).

Made 4K connections and load generator ran 45K/second update (there’s only update which change TTL and 2dsphere index)

$ ./bin/mongod --storageEngine=inMemory --inMemorySizeGB=3

After 3 day’s running, MongoDB instance consume about 18.5GB memory. But db.serverStatus().inMemory.cache.“bytes currently in the cache” metric is only below 3GB.
I don’t know this is normal that MongoDB instance use 15GB only for client connection handler threads.
But after stop load generator and close all connection, mongodb instance memory usage doesn’t show any changes.

So I tried google perf tools with rebuilt mongodb(–allocator=system)

$ env HEAPPROFILE=/data/profile/hprof HEAP_PROFILE_ALLOCATION_INTERVAL=10737418240 HEAP_PROFILE_INUSE_INTERVAL=1073741824 HEAP_PROFILE_TIME_INTERVAL=600 ./bin/mongod -f ./mongod.conf

After 12 hours running, MongoDB instance is using 5.0GB and wired tiger cache is using only 650MB(678575506 bytes).
And generate memory usage graph with pprof. But more weird things is that pprof graph shows only 777MB memory usage.

Does mongodb allocate memory out side of tcmalloc api ?
Are there any missing point in my profiling procedure ?

Regards,
Matt.

pprof_0177.pdf (14.4 KB)

pprof_0177.txt (10.3 KB)

2

There should not be places that allocates memory outside of tcmalloc.

How do you measure 18GB memory usage?

3

Hi Vadim.

I just check that with “top” and “ps” and “free” command in linux.
Of course there could be some gap with real memory usage, but not supposed to be this big.

Have you profiled mongodb memory usage with google-perf tools ?

4

Matt,

What column do you check in “top” and “ps” ?

5

Resident size of top command. (Virtual size is much greater than RSS in top result)

6

running into the same “issue”. any news on that?

mongodb says (using only one collection):

“count” : 283180000,
“size” : 27978383355,
“storageSize” : 27978383355,
“totalIndexSize” : 23503940000,
“indexSizes” : {
“id” : 5380420000,
“_id_hashed” : 4530880000,
“date_1” : 3114980000,
“fileid_1_date_1” : 4530880000,
“ip_1_date_1_fileid_1” : 5946780000
},
“avgObjSize” : 98.80070398686348,

but top says 140GB RES memory allocated.