Not the answer you need?
Register and ask your own question!

ZSTD compression support for Percona Server for MongoDB 3.4 (working patch)

mipamipa ContributorInactive User Role Beginner
Hi,

I've been able to enable support for facebooks (amazing) Zstandard compression in Percona MongoDB, by simply modifying the wiredtiger SConscript file (using apt-get source percona-server-mongodb-34).

I did various benchmarks (with that patch) that have proven that ZSTD offers a way better compression/performance ratio, than snappy or zlib (see https://github.com/facebook/zstd for more details)

​​​​​​Looking forward to see it default supported in Percona Server for MongoDB :-)

--- percona-server-mongodb-34-3.4.4.orig/src/third_party/wiredtiger/SConstruct
+++ percona-server-mongodb-34-3.4.4/src/third_party/wiredtiger/SConstruct
@@ -114,6 +114,7 @@ env['STATIC_AND_SHARED_OBJECTS_ARE_THE_S

 useZlib = GetOption("zlib")
 useSnappy = GetOption("snappy")
+useZstd = true
 useLz4 = GetOption("lz4")
 useBdb = GetOption("bdb")
 useTcmalloc = GetOption("tcmalloc")
@@ -144,6 +145,15 @@ if useSnappy:
         print 'snappy-c.h must be installed!'
         Exit(1)

+if useZstd:
+    if conf.CheckCHeader('zstd.h'):
+       env.Append(CPPDEFINES=['HAVE_BUILTIN_EXTENSION_ZSTD'])
+        wtlibs.append("zstd")
+    else:
+        print 'zstd.h must be installed!'
+        Exit(1)
+
+
 if useLz4:
     conf.env.Append(CPPPATH=[useLz4 + "/include"])
     conf.env.Append(LIBPATH=[useLz4 + "/lib"])
@@ -270,6 +280,9 @@ if useZlib:
 if useSnappy:
     wtsources.append("ext/compressors/snappy/snappy_compress.c")

+if useZstd:
+    wtsources.append("ext/compressors/zstd/zstd_compress.c")
+
 if useLz4:
     wtsources.append("ext/compressors/lz4/lz4_compress.c")

To compile (tested on Ubuntu) you need to install: libzstd-dev libzstd1


Since zstd is not native supported in MongoDB, only directly via wiredtiger you need to use:
db.createCollection( "myNewCol", { storageEngine: { wiredTiger: { configString: "block_compressor=zstd" }}})

or
wiredTiger:
  engineConfig:
    cacheSizeGB: 1
    checkpointSizeMB: 1000
    statisticsLogDelaySecs: 0
    journalCompressor: none
    directoryForIndexes: false
  collectionConfig:
    blockCompressor: zlib
    configString: "block_compressor=zstd"
  indexConfig:
    prefixCompression: true
    configString: "block_compressor=zstd"

optional you can also set "compression_level" which is defautl set to "3" (I've not yet tested)

Comments

  • david.murphydavid.murphy Contributor Inactive User Role Beginner
    Hi Mipa,

    Is there any way I could convince you to make a pull request on our GITHUB (https://github.com/percona/percona-server-mongodb) and file a PSMDB-JIRA (https://jira.percona.com/projects/PSMDB/issues) ticket? If the performance and simplicity prove out, this might be a good option for both WT & Mongo Rocks engines.


    Cheers
    David Murphy
  • mipamipa Contributor Inactive User Role Beginner
    Hi David,

    thanks for the fast reply.

    There are already two other pending pull requests since January, and one is also regarding ZSTD. I think you guys will find a way better/cleaner way to enable zstd for wiredtiger than my patch, and maybe even support using it as native blockCompressor parameter from mongodb.conf, instead of wiredtiger hack ;-)

    If you first merge the pending ZSTD static pull request, you can even include it static :-)

    Thx,
    Michael
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.