Setting Up and Monitoring MongoDB 8 Replica Sets with PMM 3 … | Percona Community

daniil.bazhenov · March 18, 2025, 3:02pm

This guide explains how to set up a MongoDB 8 Replica Set and monitor it using PMM 3, all within Docker. We’ll guide you through the steps to create a local environment, configure the necessary components, and connect them for effective monitoring and management.

This is a companion discussion topic for the original entry at https://percona.community/blog/2025/03/18/setting-up-and-monitoring-mongodb-8-replica-sets-with-pmm-3-using-docker-a-beginner-friendly-guide

Shahid_Aman · June 26, 2025, 8:21pm

mongodb-rs102 | {“t”:{“$date”:“2025-06-26T20:20:20.432+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20254, “ctx”:“main”,“msg”:“Read security file failed”,“attr”:{“error”:{“code”:30,“codeName”:“InvalidPath”,“errmsg”:“error opening file:/etc/secrets/mongodb-keyfile: bad file”}}}
i am getting this issue while using yo docker compose

daniil.bazhenov · June 27, 2025, 7:16am

Check that the keyfile has been created, recreate it again.

mkdir secrets
openssl rand -base64 128 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile

P.S. Probably copilot can help to create a keyfile for your operating system, this may be different from my Mac

If you used my example file, it needs to set permissions

chmod 600 secrets/mongodb-keyfile

rVLhIK2PhZKGxysjwMR4t1OmNppqdAzEs408hrbzg95D146mn9YENixId6pvIGCA
Cy9hc1k6OKKabbv7Rm347NwSFxbdPPx0/jnaO80U/a6/mv0XqSmEl8wdR91b4jIm
d98LobplwRs4b7g9cnLMUAIULr0WG+J36NtKIA6q4eE=

Shahid_Aman · June 27, 2025, 7:27am

I created the key file using the command. You provided

mkdir secrets
openssl rand -base64 128 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile

But still getting error opening file bad file error. I’m on Ubuntu 24.04 LTS

daniil.bazhenov · June 27, 2025, 7:31am

Hi, I don’t have Ubuntu handy, sorry.

Try in copilot to give the chat the full docker-compose.yaml file and the commands used to generate the key, probably copilot will indicate an error or suggest a better command for the key for Ubuntu

Shahid_Aman · June 27, 2025, 7:35am

Sure let me try in my personal computer which is running on mac.

daniil.bazhenov · June 27, 2025, 7:40am

Copilot answer

What the error means:

You’re seeing this MongoDB error:

"errmsg": "error opening file:/etc/secrets/mongodb-keyfile: bad file"

It indicates that MongoDB in the mongodb-rs102 container is unable to read the keyfile from the mounted location /etc/secrets/mongodb-keyfile.

Steps to fix it:

1. File permissions and ownership

MongoDB requires the keyfile to:

have permission 600
be owned by user ID 999, which is typically the user MongoDB runs as inside the container

Run these commands on your host machine:

bash

sudo chown 999:999 secrets/mongodb-keyfile
sudo chmod 600 secrets/mongodb-keyfile

2. Check the file is actually mounted inside the container

Try checking from inside the container:

bash

docker exec -it mongodb-rs102 ls -l /etc/secrets/mongodb-keyfile

You should see a real file with the correct owner and permissions. If you don’t, something may be wrong with the volume mount path.

3. Double-check you’re running Docker Compose from the right directory

Make sure you’re running docker compose up from the same directory that contains the ./secrets folder.

4. Verify the contents of the keyfile

Check for unwanted characters using:

bash

cat -A secrets/mongodb-keyfile

It should be a single line, no trailing special characters like ^M.

5. Try regenerating the keyfile

Just in case the original is broken or unreadable:

bash

mkdir -p secrets
openssl rand -base64 756 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile
chown 999:999 secrets/mongodb-keyfile

Let me know if you’d like help checking the container logs next or verifying replica set initialization!

Shahid_Aman · June 27, 2025, 9:51am

Yep it works on the macOS personal machine, so this has something to do with my ubuntu server,Any idea what is going wrong ?

daniil.bazhenov · June 27, 2025, 11:25am

you can try to generate a key as Copilot suggested, maybe it will help.

mkdir -p secrets
openssl rand -base64 756 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile
chown 999:999 secrets/mongodb-keyfile

Shahid_Aman · June 27, 2025, 11:27am

No luck tried on multiple cloud providers servers, oracle, digital ocean. Not able to spin up replica set, single instance work fine!

daniil.bazhenov · June 27, 2025, 11:28am

I’ll try it this weekend

daniil.bazhenov · June 27, 2025, 2:56pm

I got it to work on AWS EC2 on Ubuntu 24.04, but I’m not an expert and the solution doesn’t look the prettiest to me.

I had to replace Docker volumes and create regular folders for data files. In normal life I find this solution more convenient for development and backup.
Some magic with 999 permissions, I had to add it to docker-compose.yaml configuration and set these permissions for mongodb-keyfile. I don’t like it, but it works.

Eventually the replica set started and I connected from MongoDB Compass. But I’ll still be looking for another solution.

Created Project Structure
Created the following directories on the host machine:

mkdir -p secrets data/mongodb-101 data/mongodb-102 data/mongodb-103

Created the replica set keyfile:

openssl rand -base64 756 > secrets/mongodb-keyfile
chmod 400 secrets/mongodb-keyfile
sudo chown 999:999 secrets/mongodb-keyfile

Used Bind Mounts Instead of Named Volumes
Replaced named volumes with bind mounts in docker-compose.yaml

volumes:
  - ./data/mongodb-101:/data/db
  - ./secrets:/etc/secrets:ro

Ensured MongoDB had permission to write:

sudo chown -R 999:999 data/

Updated docker-compose.yaml

Added user: "999:999" to all MongoDB containers to match keyfile ownership

Docker compose file without PMM containers

version: '3.8'

services:
  mongodb-rs101:
    image: percona/percona-server-mongodb:8.0-multi
    container_name: mongodb-rs101
    ports:
      - "27017:27017"
    volumes:
      - ./data/mongodb-101:/data/db
      - ./secrets:/etc/secrets:ro
    command: >
      mongod --port 27017
             --replSet rs
             --keyFile /etc/secrets/mongodb-keyfile
             --bind_ip_all
             --profile 2
             --slowms 200
             --rateLimit 100
    environment:
      MONGO_INITDB_ROOT_USERNAME: databaseAdmin
      MONGO_INITDB_ROOT_PASSWORD: password

  mongodb-rs102:
    image: percona/percona-server-mongodb:8.0-multi
    container_name: mongodb-rs102
    ports:
      - "28017:28017"
    volumes:
      - ./data/mongodb-102:/data/db
      - ./secrets:/etc/secrets:ro
    command: >
      mongod --port 28017
             --replSet rs
             --keyFile /etc/secrets/mongodb-keyfile
             --bind_ip_all
             --profile 2
             --slowms 200
             --rateLimit 100
    environment:
      MONGO_INITDB_ROOT_USERNAME: databaseAdmin
      MONGO_INITDB_ROOT_PASSWORD: password

  mongodb-rs103:
    image: percona/percona-server-mongodb:8.0-multi
    container_name: mongodb-rs103
    ports:
      - "29017:29017"
    volumes:
      - ./data/mongodb-103:/data/db
      - ./secrets:/etc/secrets:ro
    command: >
      mongod --port 29017
             --replSet rs
             --keyFile /etc/secrets/mongodb-keyfile
             --bind_ip_all
             --profile 2
             --slowms 200
             --rateLimit 100
    environment:
      MONGO_INITDB_ROOT_USERNAME: databaseAdmin
      MONGO_INITDB_ROOT_PASSWORD: password

  rs-init:
    image: percona/percona-server-mongodb:8.0-multi
    container_name: rs-init
    depends_on:
      - mongodb-rs101
      - mongodb-rs102
      - mongodb-rs103
    volumes:
      - ./secrets:/etc/secrets:ro
    entrypoint: >
      sh -c "until mongosh --host mongodb-rs101 --port 27017
                      --username databaseAdmin --password password
                      --authenticationDatabase admin
                      --eval 'db.adminCommand({ ping: 1 })' > /dev/null 2>&1;
              do echo 'Waiting for MongoDB...'; sleep 5; done &&
              mongosh --host mongodb-rs101 --port 27017
                      --username databaseAdmin --password password
                      --authenticationDatabase admin
                      --eval 'rs.initiate({_id: \"rs\", members: [
                            {_id: 0, host: \"mongodb-rs101:27017\"},
                            {_id: 1, host: \"mongodb-rs102:28017\"},
                            {_id: 2, host: \"mongodb-rs103:29017\"}
                      ]})'"

Launched the Cluster

Started containers:

docker compose up -d

Verified containers were running:

docker ps -a

Connected to MongoDB for Testing

Connected from host into primary node:

docker exec -it mongodb-rs101 mongosh -u databaseAdmin -p password --authenticationDatabase admin

Ran replica set check:

rs.status()

Shahid_Aman · June 29, 2025, 4:33pm

Hi, you need not to do all these, I got it figured out.
The problem is in percona-server-mongodb image the mongod UID/GID is 1001:1 so you just need to

chown 1001:1 secrets/mongodb-keyfile

And boom your server should be healthy now!

Topic		Replies	Views
Running Percona Server for MongoDB with PMM monitoring and QAN query analytics in Docker Compose Percona Server for MongoDB	2	150	September 24, 2024
Percona Monitoring and Management 1.9.0 Is Now Available PMM 1.x	2	592	April 5, 2018
Percona Monitoring and Management 1.8.0 Is Now Available PMM 1.x	0	578	February 28, 2018
Percona Monitoring and Management (PMM) 1.10.0 Is Now Available PMM 1.x	0	572	April 23, 2018
Percona Monitoring and Management 1.1.3 is Now Available PMM 1.x	0	509	April 24, 2017

Setting Up and Monitoring MongoDB 8 Replica Sets with PMM 3 … | Percona Community

What the error means:

Steps to fix it:

1. File permissions and ownership

2. Check the file is actually mounted inside the container

3. Double-check you’re running Docker Compose from the right directory

4. Verify the contents of the keyfile

5. Try regenerating the keyfile

Related topics