Mongod process getting crashed with Got signal: 11 (Segmentation fault)

Mongod process crashed many time with Got signal: 11 (Segmentation fault) error

h-4.2# mongod --version
db version v4.4.4-6
Build Info: {
“version”: “4.4.4-6”,
“gitVersion”: “f3dd4bc7c7500705a537de40bb4d6127ba498bd3”,
“openSSLVersion”: “OpenSSL 1.0.2k-fips 26 Jan 2017”,
“modules”: ,
“allocator”: “tcmalloc”,
“environment”: {
“distarch”: “x86_64”,
“target_arch”: “x86_64”
}
}

Error logs before crash

{“t”:{"$date":“2021-03-31T13:13:57.110+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5366”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38068”,“connectionId”:5366,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.110+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5367”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38072”,“connectionId”:5367,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.112+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“172.24.5.56:38080”,“connectionId”:5368,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.112+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5368”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38080”,“connectionId”:5368,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.113+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“172.24.5.56:38086”,“connectionId”:5369,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.113+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5369”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.5.56:38086”,“client”:“conn5369”,“doc”:{“driver”:{“name”:“mongo-java-driver”,“version”:“3.8.2”},“os”:{“type”:“Linux”,“name”:“Linux”,“architecture”:“amd64”,“version”:“3.10.0-514.26.2.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_45-b14”}}}
{“t”:{"$date":“2021-03-31T13:13:57.139+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5369”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38086”,“connectionId”:5369,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.140+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“172.24.5.56:38094”,“connectionId”:5370,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.141+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5370”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.5.56:38094”,“client”:“conn5370”,“doc”:{“driver”:{“name”:“mongo-java-driver”,“version”:“3.8.2”},“os”:{“type”:“Linux”,“name”:“Linux”,“architecture”:“amd64”,“version”:“3.10.0-514.26.2.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_45-b14”}}}
{“t”:{"$date":“2021-03-31T13:13:57.168+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5370”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38094”,“connectionId”:5370,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.177+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5364”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38052”,“connectionId”:5364,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:13:57.189+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5363”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38044”,“connectionId”:5363,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:14:01.978+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“127.0.0.1:58726”,“connectionId”:5371,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:14:01.978+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5371”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:58726”,“client”:“conn5371”,“doc”:{“application”:{“name”:“MongoDB Shell”},“driver”:{“name”:“MongoDB Internal Client”,“version”:“4.4.4-6”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.7.1908 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1062.12.1.el7.x86
_64”}}}}
{“t”:{"$date":“2021-03-31T13:14:01.979+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20249, “ctx”:“conn5371”,“msg”:“Authentication failed”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58726”,“result”:“AuthenticationFailed: Unable to use SCRAM-SHA-256 based authentication for user without any SCRAM-SHA-256 credentials registered”}}
{“t”:{"$date":“2021-03-31T13:14:01.989+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn5371”,“msg”:“Successful authentication”,“attr”:{“mechanism”:“SCRAM-SHA-1”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58726”}}
{“t”:{"$date":“2021-03-31T13:14:02.010+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5371”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:58726”,“connectionId”:5371,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:14:08.359+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”:{“cursorId”:7904269181106903313,“idleSince”:{"$date":“2021-03-31T13:04:04.681Z”}}}
{“t”:{"$date":“2021-03-31T13:14:48.361+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”:{“cursorId”:4853223127451353051,“idleSince”:{"$date":“2021-03-31T13:04:44.475Z”}}}
{“t”:{"$date":“2021-03-31T13:14:52.361+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”:{“cursorId”:5563441711628392626,“idleSince”:{"$date":“2021-03-31T13:04:50.155Z”}}}
{“t”:{"$date":“2021-03-31T13:14:53.110+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“172.18.1.20:47614”,“connectionId”:5372,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:14:53.111+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5372”,“msg”:“Connection ended”,“attr”:{“remote”:“172.18.1.20:47614”,“connectionId”:5372,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:15:01.371+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“127.0.0.1:58852”,“connectionId”:5373,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:15:01.371+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5373”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:58852”,“client”:“conn5373”,“doc”:{“application”:{“name”:“MongoDB Shell”},“driver”:{“name”:“MongoDB Internal Client”,“version”:“4.4.4-6”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.7.1908 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1062.12.1.el7.x86_64”}}}}
{“t”:{"$date":“2021-03-31T13:15:01.371+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20249, “ctx”:“conn5373”,“msg”:“Authentication failed”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58852”,“result”:“AuthenticationFailed: Unable to use SCRAM-SHA-256 based authentication for user without any SCRAM-SHA-256 credentials registered”}}
{“t”:{"$date":“2021-03-31T13:15:01.383+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn5373”,“msg”:“Successful authentication”,“attr”:{“mechanism”:“SCRAM-SHA-1”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58852”}}
{“t”:{"$date":“2021-03-31T13:15:01.405+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5373”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:58852”,“connectionId”:5373,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“172.24.6.16:44894”,“connectionId”:5374,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“172.24.6.16:44896”,“connectionId”:5375,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5374”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.6.16:44894”,“client”:“conn5374”,“doc”:{“driver”:{“name”:“mongo-java-driver|legacy”,“version”:“4.1.1”},“os”:{“type”:“Linux”,“name”:"Linux",“architecture”:“amd64”,“version”:“3.10.0-1062.9.1.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_252-b09”}}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5375”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.6.16:44896”,“client”:“conn5375”,“doc”:{“driver”:{“name”:“mongo-java-driver|legacy”,“version”:“4.1.1”},“os”:{“type”:“Linux”,“name”:"Linux",“architecture”:“amd64”,“version”:“3.10.0-1062.9.1.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_252-b09”}}}
{“t”:{"$date":“2021-03-31T13:15:07.401+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5375”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.6.16:44896”,“connectionId”:5375,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:15:07.401+00:00”},“s”:“I”, “c”:"-", “id”:20883, “ctx”:“conn5374”,“msg”:“Interrupted operation as its client disconnected”,“attr”:{“opId”:296689953}}
{“t”:{"$date":“2021-03-31T13:15:07.401+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5374”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.6.16:44894”,“connectionId”:5374,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:15:16.490+00:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“thread5363”,“msg”:“Writing fatal message”,“attr”:{“message”:“Invalid access at address: 0”}}
{“t”:{"$date":“2021-03-31T13:15:16.490+00:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“thread5363”,“msg”:“Writing fatal message”,“attr”:{“message”:“Got signal: 11 (Segmentation fault).\n”}}

Hi Murali.

I see the final messages are:

“Invalid access at address: 0”
“Got signal: 11 (Segmentation fault).\n

Which looks like a classic null-pointer dereference error. Classic for programming in general I mean, not for MongoDB particularly. I haven’t seen this much with MongoDB.

I don’t think the failed authentication from ~9 secs prior to the crash, the one with connectionId = 5375 is likely to be worth investigating.

Maybe the operation with “opId”:296689953 on connectionId 5374 is the culprit - but there’s no proof of that so far.

Is there a backtrace section from the log file?

1 Like

Hi Akira_Kurogane,

No backtrace for the log. will investigate the shared connection id and get back to you.

1 Like