Enabling encryption does not work in Docker container

Hi there,
We’re using the Docker container version 6.0.4 version of Percona Mongo and we’re running into what appears to be a bug in the container initialization. The problem we’re having is that the Percona instance in the container is ignoring our encryption directive in our config file on initial startup. What’s happening is that the container starts up in a minimal mode to initialize the database and create the initial user. Then it restarts but this time it uses the config file and any other options we pass it. This is a problem because the initial startup creates unencrypted data files and the second startup (with encryption enabled) tries to read those data files and thinks they are corrupted because they are not encrypted as expected.

A workaround was to create the encrypted data files in a different folder. The problem with that is that the initial user now does not exist in the encrypted database. If anyone has suggestions as to how to handle this I’d appreciate it. I’ve spent many hours on this now.

Thanks,
Matt

Can you please share the details of how exactly you’re trying to start the container? I’ve tried the following way and it started successfully:

docker run --name test_encryption -e MONGO_INITDB_ROOT_USERNAME=dba -e MONGO_INITDB_ROOT_PASSWORD=secret -v /certs/:/certs -v ./datadir:/data/db -d percona/percona-server-mongodb:6.0.4 --enableEncryption --kmipServerName 10.0.0.1 --kmipServerCAFile /certs/ca.crt --kmipClientCertificateFile /certs/mongod.pem

In addition, can you also please provide the info about key storage - is it some kind of KMIP server or HashiCorp Vault?

Hi Sandra,
Thanks for checking this out! I’m starting this from within Docker compose and it reads from a config file. I’ve also gotten to the point where I’m extending the Percona container directly so it’s easier to torubleshoot. I use that in the compose file below. My above report is based on using the container with no changes. That is, I’ve just copy-pasted Percona’s info into my own Docker container, without changes, and it works the same as using the container directly.
I’ve pasted the config file below. I’m also including the docker compose setup.

I just saw your follow-up note. We are using a local key file for the key, generated per the docs.

Any insight you can give would be great. Thanks!
Matt

# mongod.conf, Percona Server for MongoDB
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /data/db #-encrypted # Use a non-default folder name, like "db-encrypted", for containerized, encrypted version
  # The reason is that the container already defaults to a non-encrypted set of files in
  # /data/db, so when you start up in encrypted mode, the db will complain that the files
  # are corrupted, since they already exist and are unencrypted.
  journal:
    enabled: true
#  engine: wiredTiger
#  engine: inMemory

# Storage engine various options
#  More info for wiredTiger: https://docs.mongodb.com/v4.4/reference/configuration-options/#storage-wiredtiger-options
#  wiredTiger:
#    engineConfig:
#      cacheSizeGB: 1
#      checkpointSizeMB: 1000
#      statisticsLogDelaySecs: 0
#      journalCompressor: snappy
#      directoryForIndexes: false
#    collectionConfig:
#      blockCompressor: snappy
#    indexConfig:
#      prefixCompression: true

#  More info for inMemory: https://www.percona.com/doc/percona-server-for-mongodb/4.4/inmemory.html#configuring-percona-memory-engine
#  inMemory:
#    engineConfig:
#      inMemorySizeGB: 1
#      statisticsLogDelaySecs: 0

# Two options below can be used for wiredTiger and inMemory storage engines
setParameter:
  auditAuthorizationSuccess: true
#    wiredTigerConcurrentReadTransactions: 128
#    wiredTigerConcurrentWriteTransactions: 128

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

processManagement:
  pidFilePath: /var/run/mongod.pid
#   fork: true # This causes server to die, in a container

# network interfaces
net:
  port: 27017
#   bindIp: 127.0.0.1
  bindIpAll: true
#   tls:
#     mode: requireTLS
#     certificateKeyFile: /certs/settings_db/cert.pem
#     CAFile: /certs/settings_db/key.pem
#     allowInvalidCertificates: false
#     allowConnectionsWithoutCertificates: false
#     FIPSMode: true

security:
  javascriptEnabled: false
  authorization: enabled
  enableEncryption: true # Temporarily commented out
  encryptionCipherMode: AES256-CBC
  encryptionKeyFile: /etc/mongodb-keyfile
  relaxPermChecks: false

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

auditLog:
 destination: file
 format: BSON
 path: /var/log/mongodb/auditLog.bson
 # filter: '{ atype: { $in: [ "createCollection", "dropCollection" ] } }' # If filters are desired

#snmp:

Docker compose configuration:

settings_db:
    image: test/percona-mongo:${MONGODB_VERSION?err}
    restart: always
    domainname: ${SETTINGS_DB_HOST?err}
    networks:
      - settings
    expose:
      - ${SETTINGS_DB_PORT?err}
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${SETTINGS_DB_USER?err}
      - MONGO_INITDB_ROOT_PASSWORD=${SETTINGS_DB_PASSWORD?err}
#    user: 1001:995
    command: --config /etc/mongod.conf
    volumes_from:
      - certs:ro
    volumes:
      - type: bind
        source: ./mongodb/mongod.conf
        target: /etc/mongod.conf
        read_only: true
        bind:
          create_host_path: true
          selinux: Z
      - type: bind
        source: ./mongodb/mongodb-keyfile
        target: /etc/mongodb-keyfile
        read_only: true
        bind:
          create_host_path: true
          selinux: Z
      - type: bind
        source: ./mongodb_logs/mongod.log
        target: /var/log/mongodb/mongod.log
        bind:
          create_host_path: true
          selinux: Z

It seems that init script doesn’t like config file much :frowning: Please try passing encryption options in addition to config, f.e.

psmdbserver:
image: percona/percona-server-mongodb:6.0.4
environment:
- MONGO_INITDB_ROOT_USERNAME=dba
- MONGO_INITDB_ROOT_PASSWORD=secret
command:
- --config=/certs/mongod.conf
- --enableEncryption
- --encryptionKeyFile=/certs/mongodb-keyfile
volumes:
- /pykmip_workdir/:/certs
- ./datadir:/data/db

Hi Sandra,
Thanks, are you saying to replace config file options with command line arguments? Or only to pass the encryption arguments specifically? It appears so, but wanted to confirm.
Thank you again.
Matt

You can either replace, or leave in config as is - it actually works for both ways.

Thank you Sandra! I just tried this and it worked! Those were the magic incantations. I appreciate the help. The documentation or functionality should probably be updated to address the config file issue, as it’s basic functionality that appears broken though.
Thank you again!
Matt

1 Like

Glad to help! Sure, we’ll check whether it’s an issue in init script or whether it just should be more thoroughly documented.

Respective task in Jira
https://jira.percona.com/browse/PSMDB-1243

I am facing permission issue for secrets file


services:

  mongodb:
    image: 'percona/percona-server-mongodb:4.4'
    restart: always
    command: --wiredTigerCacheSizeGB=1 --enableEncryption --encryptionKeyFile=/mong_opener
    volumes:
      - mongodata:/data/db
      - /home/ubuntu/mong_opener:/mong_opener
    ports:
      - "27028:27017"
    ulimits:
      nofile:
        soft: "65536"
        hard: "65536"


volumes:
  mongodata:

Error

2023-06-30T09:28:39.464173819Z {"t":{"$date":"2023-06-30T09:28:39.464+00:00"},"s":"F",  "c":"STORAGE",  "id":29120,   "ctx":"initandlisten","msg":"Data-at-Rest Encryption Error","attr":{"error":{"what":"Can't create encryption key database","reason":{"what":"key reading failed","reason":"reading the master key from the encryption key file failed: cannot open specified encryption key file: /mong_opener"},"encryptionKeyDatabaseDirectory":"/data/db/key.db"}}}

Any help ?? I try to change the ownership however it changes to different when starting docker.