ZSTD compression support for Percona Server for MongoDB 3.4 (working patch)

Hi,

I’ve been able to enable support for facebooks (amazing) Zstandard compression in Percona MongoDB, by simply modifying the wiredtiger SConscript file (using apt-get source percona-server-mongodb-34).

I did various benchmarks (with that patch) that have proven that ZSTD offers a way better compression/performance ratio, than snappy or zlib (see [URL]https://github.com/facebook/zstd[/URL] for more details)

​​​​​​Looking forward to see it default supported in Percona Server for MongoDB :slight_smile:


--- percona-server-mongodb-34-3.4.4.orig/src/third_party/wiredtiger/SConstruct
+++ percona-server-mongodb-34-3.4.4/src/third_party/wiredtiger/SConstruct
@@ -114,6 +114,7 @@ env['STATIC_AND_SHARED_OBJECTS_ARE_THE_S

useZlib = GetOption("zlib")
useSnappy = GetOption("snappy")
+useZstd = true
useLz4 = GetOption("lz4")
useBdb = GetOption("bdb")
useTcmalloc = GetOption("tcmalloc")
@@ -144,6 +145,15 @@ if useSnappy:
print 'snappy-c.h must be installed!'
Exit(1)

+if useZstd:
+ if conf.CheckCHeader('zstd.h'):
+ env.Append(CPPDEFINES=['HAVE_BUILTIN_EXTENSION_ZSTD'])
+ wtlibs.append("zstd")
+ else:
+ print 'zstd.h must be installed!'
+ Exit(1)
+
+
if useLz4:
conf.env.Append(CPPPATH=[useLz4 + "/include"])
conf.env.Append(LIBPATH=[useLz4 + "/lib"])
@@ -270,6 +280,9 @@ if useZlib:
if useSnappy:
wtsources.append("ext/compressors/snappy/snappy_compress.c")

+if useZstd:
+ wtsources.append("ext/compressors/zstd/zstd_compress.c")
+
if useLz4:
wtsources.append("ext/compressors/lz4/lz4_compress.c")

To compile (tested on Ubuntu) you need to install: libzstd-dev libzstd1

Since zstd is not native supported in MongoDB, only directly via wiredtiger you need to use:


db.createCollection( "myNewCol", { storageEngine: { wiredTiger: { configString: "block_compressor=zstd" }}})

or


wiredTiger:
engineConfig:
cacheSizeGB: 1
checkpointSizeMB: 1000
statisticsLogDelaySecs: 0
journalCompressor: none
directoryForIndexes: false
collectionConfig:
blockCompressor: zlib
configString: "block_compressor=zstd"
indexConfig:
prefixCompression: true
configString: "block_compressor=zstd"

optional you can also set “compression_level” which is defautl set to “3” (I’ve not yet tested)

Hi Mipa,

Is there any way I could convince you to make a pull request on our GITHUB ([url]https://github.com/percona/percona-server-mongodb[/url]) and file a PSMDB-JIRA ([url]https://jira.percona.com/projects/PSMDB/issues[/url]) ticket? If the performance and simplicity prove out, this might be a good option for both WT & Mongo Rocks engines.

Cheers
David Murphy

Hi David,

thanks for the fast reply.

There are already two other pending pull requests since January, and one is also regarding ZSTD. I think you guys will find a way better/cleaner way to enable zstd for wiredtiger than my patch, and maybe even support using it as native blockCompressor parameter from mongodb.conf, instead of wiredtiger hack :wink:

If you first merge the pending ZSTD static pull request, you can even include it static :slight_smile:

Thx,
Michael