Sharded instance for Mongodb failing to start

Hi all,

I reset my Mongo db clsuter. I deleted all the data from the data directory and reformatted the drive and remounted it. I ensure that mongod has the access to the log and data directory. However, when I start the mongod service it fails with the error below: I did delete the temp lock etc.

[root@gtm-server31 ~]# /usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongodShard3-1.conf &
[1] 16393
[root@server31 ~]# about to fork child process, waiting until server is ready for connections.
forked process: 16395
ERROR: child process failed, exited with 48

To see additional information in this output, start without the “–fork” option.

[1]+  Exit 48                 /usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongodShard3-1.conf

[root@server31 ~]# cat /etc/mongodShard3-1.conf 
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
  engine: wiredTiger
  wiredTiger:
     engineConfig:
        journalCompressor: none
        cacheSizeGB: 70
     collectionConfig:
        blockCompressor: none
     indexConfig:
        prefixCompression: false
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo
sharding:
  clusterRole: shardsvr
replication:
  replSetName: Shard3ReplSet
net:
  port: 27018
  bindIp: server31
**log fine output:**
{"t":{"$date":"2021-01-27T02:29:16.741-08:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2021-01-27T02:29:16.748-08:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2021-01-27T02:29:16.748-08:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2021-01-27T02:29:16.748-08:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":19287,"port":27018,"dbPath":"/var/lib/mongo","architecture":"64-bit","host":"gtm-server31"}}
{"t":{"$date":"2021-01-27T02:29:16.748-08:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.2-4","gitVersion":"5fca74b04eeebad5e7718e8ed65577b90de8bc5f","openSSLVersion":"OpenSSL 1.1.1g FIPS  21 Apr 2020","modules":[],"allocator":"tcmalloc","environment":{"distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-01-27T02:29:16.748-08:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"CentOS Linux release 8.3.2011","version":"Kernel 4.18.0-240.10.1.el8_3.x86_64"}}}
{"t":{"$date":"2021-01-27T02:29:16.748-08:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongodShard3-1.conf","net":{"bindIp":"gtm-server31","port":27018},"processManagement":{"fork":true,"pidFilePath":"/var/run/mongodb/mongod.pid","timeZoneInfo":"/usr/share/zoneinfo"},"replication":{"replSetName":"Shard3ReplSet"},"sharding":{"clusterRole":"shardsvr"},"storage":{"dbPath":"/var/lib/mongo","engine":"wiredTiger","journal":{"enabled":true},"wiredTiger":{"collectionConfig":{"blockCompressor":"none"},"engineConfig":{"cacheSizeGB":70.0,"journalCompressor":"none"},"indexConfig":{"prefixCompression":false}}},"systemLog":{"destination":"file","logAppend":true,"path":"/var/log/mongodb/mongod.log"}}}}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"E",  "c":"STORAGE",  "id":20568,   "ctx":"initandlisten","msg":"Error setting up listener","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Cannot assign requested address"}}}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"FTDC",     "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"STORAGE",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"STORAGE",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2021-01-27T02:29:16.749-08:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":48}}

Any suggestions to fix this?

1 Like

Hi, please make sure that no mongod instances are left running. Error 48 means the port is unavailable

1 Like

Thank you for responding. I ran netstat and I don’t see 27018 being used anywhere.
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:44321 0.0.0.0:* LISTEN 4397/pmcd
tcp 0 0 127.0.0.1:4330 0.0.0.0:* LISTEN 4745/pmlogger
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2457/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2542/cupsd
tcp6 0 0 ::1:44321 :::* LISTEN 4397/pmcd
tcp6 0 0 :::9090 :::* LISTEN 1/systemd
tcp6 0 0 ::1:4330 :::* LISTEN 4745/pmlogger
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 2457/sshd
tcp6 0 0 ::1:631 :::* LISTEN 2542/cupsd

netstat -tulpn
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:44321 0.0.0.0:* LISTEN 4397/pmcd
tcp 0 0 127.0.0.1:4330 0.0.0.0:* LISTEN 4745/pmlogger
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2457/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2542/cupsd
tcp6 0 0 ::1:44321 :::* LISTEN 4397/pmcd
tcp6 0 0 :::9090 :::* LISTEN 1/systemd
tcp6 0 0 ::1:4330 :::* LISTEN 4745/pmlogger
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 2457/sshd
tcp6 0 0 ::1:631 :::* LISTEN 2542/cupsd
udp 0 0 0.0.0.0:111 0.0.0.0:* 1/systemd
udp 0 0 127.0.0.1:323 0.0.0.0:* 2460/chronyd
udp6 0 0 :::111 :::* 1/systemd
udp6 0 0 ::1:323 :::* 2460/chronyd

1 Like

I agree with the diagnosis about a port being unavailable. This error message in the log file is the one that catches my eye:

I wonder if it might be:

  • Some sort of double action. I.e. there are two mongod instances at the time, which are both dead again by the time you ran the netstat(?) command.
  • The unix socket is the one that cannot be bound. Look for another file, particularly one created by root or another user, at /tmp/mongodb-PORTNUMBER.sock.
  • If “server31”, used in the bindIP value, is not resolving to a local ethernet device’s IP address.
1 Like