Mongod process getting crashed with Got signal: 11 (Segmentation fault)

Mongod process crashed many time with Got signal: 11 (Segmentation fault) error

h-4.2# mongod --version
db version v4.4.4-6
Build Info: {
“version”: “4.4.4-6”,
“gitVersion”: “f3dd4bc7c7500705a537de40bb4d6127ba498bd3”,
“openSSLVersion”: “OpenSSL 1.0.2k-fips 26 Jan 2017”,
“modules”: ,
“allocator”: “tcmalloc”,
“environment”: {
“distarch”: “x86_64”,
“target_arch”: “x86_64”
}
}

Error logs before crash

{“t”:{"$date":“2021-03-31T13:13:57.110+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5366”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38068”,“connectionId”:5366,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.110+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5367”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38072”,“connectionId”:5367,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.112+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“172.24.5.56:38080”,“connectionId”:5368,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.112+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5368”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38080”,“connectionId”:5368,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.113+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“172.24.5.56:38086”,“connectionId”:5369,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.113+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5369”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.5.56:38086”,“client”:“conn5369”,“doc”:{“driver”:{“name”:“mongo-java-driver”,“version”:“3.8.2”},“os”:{“type”:“Linux”,“name”:“Linux”,“architecture”:“amd64”,“version”:“3.10.0-514.26.2.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_45-b14”}}}
{“t”:{"$date":“2021-03-31T13:13:57.139+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5369”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38086”,“connectionId”:5369,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.140+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“172.24.5.56:38094”,“connectionId”:5370,“connectionCount”:417}}
{“t”:{"$date":“2021-03-31T13:13:57.141+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5370”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.5.56:38094”,“client”:“conn5370”,“doc”:{“driver”:{“name”:“mongo-java-driver”,“version”:“3.8.2”},“os”:{“type”:“Linux”,“name”:“Linux”,“architecture”:“amd64”,“version”:“3.10.0-514.26.2.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_45-b14”}}}
{“t”:{"$date":“2021-03-31T13:13:57.168+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5370”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38094”,“connectionId”:5370,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:13:57.177+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5364”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38052”,“connectionId”:5364,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:13:57.189+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5363”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.5.56:38044”,“connectionId”:5363,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:14:01.978+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“127.0.0.1:58726”,“connectionId”:5371,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:14:01.978+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5371”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:58726”,“client”:“conn5371”,“doc”:{“application”:{“name”:“MongoDB Shell”},“driver”:{“name”:“MongoDB Internal Client”,“version”:“4.4.4-6”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.7.1908 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1062.12.1.el7.x86
_64”}}}}
{“t”:{"$date":“2021-03-31T13:14:01.979+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20249, “ctx”:“conn5371”,“msg”:“Authentication failed”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58726”,“result”:“AuthenticationFailed: Unable to use SCRAM-SHA-256 based authentication for user without any SCRAM-SHA-256 credentials registered”}}
{“t”:{"$date":“2021-03-31T13:14:01.989+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn5371”,“msg”:“Successful authentication”,“attr”:{“mechanism”:“SCRAM-SHA-1”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58726”}}
{“t”:{"$date":“2021-03-31T13:14:02.010+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5371”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:58726”,“connectionId”:5371,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:14:08.359+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”:{“cursorId”:7904269181106903313,“idleSince”:{"$date":“2021-03-31T13:04:04.681Z”}}}
{“t”:{"$date":“2021-03-31T13:14:48.361+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”:{“cursorId”:4853223127451353051,“idleSince”:{"$date":“2021-03-31T13:04:44.475Z”}}}
{“t”:{"$date":“2021-03-31T13:14:52.361+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”:{“cursorId”:5563441711628392626,“idleSince”:{"$date":“2021-03-31T13:04:50.155Z”}}}
{“t”:{"$date":“2021-03-31T13:14:53.110+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“172.18.1.20:47614”,“connectionId”:5372,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:14:53.111+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5372”,“msg”:“Connection ended”,“attr”:{“remote”:“172.18.1.20:47614”,“connectionId”:5372,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:15:01.371+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r emote”:“127.0.0.1:58852”,“connectionId”:5373,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:15:01.371+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5373”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:58852”,“client”:“conn5373”,“doc”:{“application”:{“name”:“MongoDB Shell”},“driver”:{“name”:“MongoDB Internal Client”,“version”:“4.4.4-6”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.7.1908 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1062.12.1.el7.x86_64”}}}}
{“t”:{"$date":“2021-03-31T13:15:01.371+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20249, “ctx”:“conn5373”,“msg”:“Authentication failed”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58852”,“result”:“AuthenticationFailed: Unable to use SCRAM-SHA-256 based authentication for user without any SCRAM-SHA-256 credentials registered”}}
{“t”:{"$date":“2021-03-31T13:15:01.383+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn5373”,“msg”:“Successful authentication”,“attr”:{“mechanism”:“SCRAM-SHA-1”,“principalName”:“monitor”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:58852”}}
{“t”:{"$date":“2021-03-31T13:15:01.405+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5373”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:58852”,“connectionId”:5373,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“172.24.6.16:44894”,“connectionId”:5374,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“172.24.6.16:44896”,“connectionId”:5375,“connectionCount”:416}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5374”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.6.16:44894”,“client”:“conn5374”,“doc”:{“driver”:{“name”:“mongo-java-driver|legacy”,“version”:“4.1.1”},“os”:{“type”:“Linux”,“name”:"Linux",“architecture”:“amd64”,“version”:“3.10.0-1062.9.1.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_252-b09”}}}
{“t”:{"$date":“2021-03-31T13:15:07.373+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn5375”,“msg”:“client metadata”,“attr”:{“remote”:“172.24.6.16:44896”,“client”:“conn5375”,“doc”:{“driver”:{“name”:“mongo-java-driver|legacy”,“version”:“4.1.1”},“os”:{“type”:“Linux”,“name”:"Linux",“architecture”:“amd64”,“version”:“3.10.0-1062.9.1.el7.x86_64”},“platform”:“Java/Oracle Corporation/1.8.0_252-b09”}}}
{“t”:{"$date":“2021-03-31T13:15:07.401+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5375”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.6.16:44896”,“connectionId”:5375,“connectionCount”:415}}
{“t”:{"$date":“2021-03-31T13:15:07.401+00:00”},“s”:“I”, “c”:"-", “id”:20883, “ctx”:“conn5374”,“msg”:“Interrupted operation as its client disconnected”,“attr”:{“opId”:296689953}}
{“t”:{"$date":“2021-03-31T13:15:07.401+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn5374”,“msg”:“Connection ended”,“attr”:{“remote”:“172.24.6.16:44894”,“connectionId”:5374,“connectionCount”:414}}
{“t”:{"$date":“2021-03-31T13:15:16.490+00:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“thread5363”,“msg”:“Writing fatal message”,“attr”:{“message”:“Invalid access at address: 0”}}
{“t”:{"$date":“2021-03-31T13:15:16.490+00:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“thread5363”,“msg”:“Writing fatal message”,“attr”:{“message”:“Got signal: 11 (Segmentation fault).\n”}}

Hi Murali.

I see the final messages are:

“Invalid access at address: 0”
“Got signal: 11 (Segmentation fault).\n

Which looks like a classic null-pointer dereference error. Classic for programming in general I mean, not for MongoDB particularly. I haven’t seen this much with MongoDB.

I don’t think the failed authentication from ~9 secs prior to the crash, the one with connectionId = 5375 is likely to be worth investigating.

Maybe the operation with “opId”:296689953 on connectionId 5374 is the culprit - but there’s no proof of that so far.

Is there a backtrace section from the log file?

1 Like

Hi Akira_Kurogane,

No backtrace for the log. will investigate the shared connection id and get back to you.

1 Like

Hi Team,
Again today the mongodb got crashed with following error .

{“t”:{"$date":“2021-07-08T11:58:57.127+00:00”},“s”:“F”, “c”:"-", “id”:23080, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:"\n\n***aborting
after invariant() failure\n\n"}
{“t”:{"$date":“2021-07-08T11:58:57.127+00:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:“Writing fatal m
essage”,“attr”:{“message”:“Got signal: 6 (Aborted).\n”}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31431, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:"BACKTRACE: {bt}
.

Here is the complete log before crashed
emote":“xx.xx.xx.xx:36696”,“connectionId”:67997,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:53:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn67997”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“xx.xx.xx.xx:36696”,“connectionId”:67997,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:53:18.515+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:377223788387066660,“idleSince”:{"$date":“2021-07-08T11:43:15.140Z”}}}
{“t”:{"$date":“2021-07-08T11:53:26.516+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:8859598790519006230,“idleSince”:{"$date":“2021-07-08T11:43:23.292Z”}}}
{“t”:{"$date":“2021-07-08T11:53:34.516+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:4256969131112290776,“idleSince”:{"$date":“2021-07-08T11:43:34.125Z”}}}
{“t”:{"$date":“2021-07-08T11:53:34.516+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:5802009825555165581,“idleSince”:{"$date":“2021-07-08T11:43:32.539Z”}}}
{“t”:{"$date":“2021-07-08T11:53:42.517+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:2313105868267765464,“idleSince”:{"$date":“2021-07-08T11:43:42.482Z”}}}
{“t”:{"$date":“2021-07-08T11:53:54.518+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:3301981260009307571,“idleSince”:{"$date":“2021-07-08T11:43:53.403Z”}}}
{“t”:{"$date":“2021-07-08T11:54:06.519+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:7637104258329324527,“idleSince”:{"$date":“2021-07-08T11:44:03.965Z”}}}
{“t”:{"$date":“2021-07-08T11:54:10.519+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:1204502043730160407,“idleSince”:{"$date":“2021-07-08T11:44:07.442Z”}}}
{“t”:{"$date":“2021-07-08T11:54:10.520+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:7112901260208356327,“idleSince”:{"$date":“2021-07-08T11:44:09.617Z”}}}
{“t”:{"$date":“2021-07-08T11:54:14.520+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:7129352708288117004,“idleSince”:{"$date":“2021-07-08T11:44:13.696Z”}}}
{“t”:{"$date":“2021-07-08T11:54:15.860+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r
emote”:“xx.xx.xx.xx:37292”,“connectionId”:67998,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:54:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn67998”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“xx.xx.xx.xx:37292”,“connectionId”:67998,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:54:22.521+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:8185101709135123398,“idleSince”:{"$date":“2021-07-08T11:44:19.955Z”}}}
{“t”:{"$date":“2021-07-08T11:54:34.521+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:7761858418928150960,“idleSince”:{"$date":“2021-07-08T11:44:34.454Z”}}}
{“t”:{"$date":“2021-07-08T11:54:58.523+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:6056470634275818920,“idleSince”:{"$date":“2021-07-08T11:44:55.452Z”}}}
{“t”:{"$date":“2021-07-08T11:55:01.643+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r
emote”:“127.0.0.1:33480”,“connectionId”:67999,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:55:01.644+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn67999”,“msg”:“client metadata”,“attr”:{“remo
te”:“127.0.0.1:33480”,“client”:“conn67999”,“doc”:{“application”:{“name”:“MongoDB Shell”},“driver”:{“name”:“MongoDB Internal Client”,“version”:
“4.4.4-6”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.7.1908 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1062.9.1.el7.x8
6_64”}}}}
{“t”:{"$date":“2021-07-08T11:55:01.644+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20249, “ctx”:“conn67999”,“msg”:“Authentication failed”,“attr”:
{“mechanism”:“SCRAM-SHA-256”,“principalName”:“xxxxxx”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:33480”,“result”:“AuthenticationFai
led: Unable to use SCRAM-SHA-256 based authentication for user without any SCRAM-SHA-256 credentials registered”}}
{“t”:{"$date":“2021-07-08T11:55:01.654+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn67999”,“msg”:“Successful authentication”,“at
tr”:{“mechanism”:“SCRAM-SHA-1”,“principalName”:“xxxxxx”,“authenticationDatabase”:“admin”,“client”:“127.0.0.1:33480”}}
{“t”:{"$date":“2021-07-08T11:55:01.660+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn67999”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“127.0.0.1:33480”,“connectionId”:67999,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:55:15.860+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r
emote”:“xx.xx.xx.xx:37870”,“connectionId”:68000,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:55:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn68000”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“xx.xx.xx.xx:37870”,“connectionId”:68000,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:55:42.527+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:7372937045780781111,“idleSince”:{"$date":“2021-07-08T11:45:42.287Z”}}}
{“t”:{"$date":“2021-07-08T11:55:46.527+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:8124829944653316967,“idleSince”:{"$date":“2021-07-08T11:45:44.598Z”}}}
{“t”:{"$date":“2021-07-08T11:55:46.527+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:8989965420745887868,“idleSince”:{"$date":“2021-07-08T11:45:44.397Z”}}}
{“t”:{"$date":“2021-07-08T11:55:50.528+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:3928038101692991331,“idleSince”:{"$date":“2021-07-08T11:45:49.224Z”}}}
{“t”:{"$date":“2021-07-08T11:56:06.529+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:8475410858587847881,“idleSince”:{"$date":“2021-07-08T11:46:04.009Z”}}}
{“t”:{"$date":“2021-07-08T11:56:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r
emote”:“xx.xx.xx.xx:38450”,“connectionId”:68001,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:56:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn68001”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“xx.xx.xx.xx:38450”,“connectionId”:68001,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:57:15.860+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r
emote”:“xx.xx.xx.xx:39030”,“connectionId”:68002,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:57:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn68002”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“xx.xx.xx.xx:39030”,“connectionId”:68002,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:57:18.535+00:00”},“s”:“I”, “c”:“QUERY”, “id”:20529, “ctx”:“clientcursormon”,“msg”:“Cursor timed out”,“attr”
:{“cursorId”:713370189066619809,“idleSince”:{"$date":“2021-07-08T11:47:17.052Z”}}}
{“t”:{"$date":“2021-07-08T11:58:15.860+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“r
emote”:“xx.xx.xx.xx:39610”,“connectionId”:68003,“connectionCount”:444}}
{“t”:{"$date":“2021-07-08T11:58:15.861+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn68003”,“msg”:“Connection ended”,“attr”:{“rem
ote”:“xx.xx.xx.xx:39610”,“connectionId”:68003,“connectionCount”:443}}
{“t”:{"$date":“2021-07-08T11:58:57.127+00:00”},“s”:“F”, “c”:"-", “id”:23079, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:“Invariant failu
re”,“attr”:{“expr”:“opCtx != nullptr && _opCtx == nullptr”,“file”:“src/mongo/db/client.cpp”,“line”:126}}
{“t”:{"$date":“2021-07-08T11:58:57.127+00:00”},“s”:“F”, “c”:"-", “id”:23080, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:"\n\n***aborting
after invariant() failure\n\n"}
{“t”:{"$date":“2021-07-08T11:58:57.127+00:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:“Writing fatal m
essage”,“attr”:{“message”:“Got signal: 6 (Aborted).\n”}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31431, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:“BACKTRACE: {bt}
“,“attr”:{“bt”:{“backtrace”:[{“a”:“55FDAE5C8D56”,“b”:“55FDAB722000”,“o”:“2EA6D56”,“s”:”_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStack
TraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.608”,“s+”:“1A6”},{“a”:“55FDAE5CA889”,“b”:“55FDAB722000”,“o”:“2EA8889”,“s”:"_ZN5mongo1
5printStackTraceEv",“s+”:“29”},{“a”:“55FDAE5C7AE6”,“b”:“55FDAB722000”,“o”:“2EA5AE6”,“s”:“ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo
tPv”,“s+”:“66”},{“a”:“7F6D53BA85F0”,“b”:“7F6D53B99000”,“o”:“F5F0”,“s”:"_L_unlock_13",“s+”:“34”},{“a”:“7F6D53801337”,“b”:“7F6D537CB000”,“o”:“36
337”,“s”:“gsignal”,“s+”:“37”},{“a”:“7F6D53802A28”,“b”:“7F6D537CB000”,“o”:“37A28”,“s”:“abort”,“s+”:“148”},{“a”:“55FDAC6EB328”,“b”:“55FDAB722000
“,“o”:“FC9328”,“s”:”_ZN5mongo15invariantFailedEPKcS1_j”,“s+”:“12C”},{“a”:“55FDAC6BD7A2”,“b”:“55FDAB722000”,“o”:“F9B7A2”,“s”:"_ZN5mongo6Client1
9setOperationContextEPNS_16OperationContextE.cold.133",“s+”:“18”},{“a”:“55FDAE484B65”,“b”:“55FDAB722000”,“o”:“2D62B65”,“s”:"_ZN5mongo14Service
Context20makeOperationContextEPNS_6ClientE",“s+”:“125”},{“a”:“55FDAE479AF7”,“b”:“55FDAB722000”,“o”:“2D57AF7”,“s”:"_ZN5mongo6Client20makeOperat
ionContextEv",“s+”:“27”},{“a”:“55FDAE273AB2”,“b”:“55FDAB722000”,“o”:“2B51AB2”,“s”:“ZN5mongo21KeysCollectionManager14PeriodicRunner18_doPeriod
icRefreshEPNS_14ServiceContextESsNS_8DurationISt5ratioILl1ELl1000EEEE",“s+”:“162”},{“a”:“55FDAE275B87”,“b”:“55FDAB722000”,“o”:“2B53B87”,“s”:"

ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_21KeysCollectionManager14PeriodicRunner5startEPNS3_14ServiceContex
tERKSsNS3_8DurationISt5ratioILl1ELl1000EEEEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv”,“s+”:“A7”},{“a”:“55FDAE8E330F”,“b”:“55FDAB722000”,“o”:"
31C130F",“s”:“execute_native_thread_routine”,“s+”:“F”},{“a”:“7F6D53BA0E65”,“b”:“7F6D53B99000”,“o”:“7E65”,“s”:“start_thread”,“s+”:“C5”},{“a”:“7
F6D538C988D”,“b”:“7F6D537CB000”,“o”:“FE88D”,“s”:“clone”,“s+”:“6D”}],“processInfo”:{“mongodbVersion”:“4.4.4-6”,“gitVersion”:“f3dd4bc7c7500705a5
37de40bb4d6127ba498bd3”,“compiledModules”:,“uname”:{“sysname”:“Linux”,“release”:“3.10.0-1062.9.1.el7.x86_64”,“version”:"#1 SMP Fri Dec 6 15:
49:49 UTC 2019",“machine”:“x86_64”},“somap”:[{“b”:“55FDAB722000”,“elfType”:3,“buildId”:“9C273C1D1F388E3DF5FF7BAE1D9FD4899A87149D”},{“b”:“7F6D5
3B99000”,“path”:"/lib64/libpthread.so.0",“elfType”:3,“buildId”:“8B33F7F8C86F8D544C63C5541A8E42B3DDFEF8B1”},{“b”:“7F6D537CB000”,“path”:"/lib64/
libc.so.6",“elfType”:3,“buildId”:“398944D32CF16A67AF51067A326E6C0CC14F90ED”}]}}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE5C8D56”,“b”:“55FDAB722000”,“o”:“2EA6D56”,“s”:"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplE
RKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.608",“s+”:“1A6”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE5CA889”,“b”:“55FDAB722000”,“o”:“2EA8889”,“s”:"_ZN5mongo15printStackTraceEv",“s+”:“29”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE5C7AE6”,“b”:“55FDAB722000”,“o”:“2EA5AE6”,“s”:"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv",“s+”:“6
6”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“7F6D53BA85F0”,“b”:“7F6D53B99000”,“o”:“F5F0”,“s”:"_L_unlock_13",“s+”:“34”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“7F6D53801337”,“b”:“7F6D537CB000”,“o”:“36337”,“s”:“gsignal”,“s+”:“37”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“7F6D53802A28”,“b”:“7F6D537CB000”,“o”:“37A28”,“s”:“abort”,“s+”:“148”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAC6EB328”,“b”:“55FDAB722000”,“o”:“FC9328”,“s”:"_ZN5mongo15invariantFailedEPKcS1_j",“s+”:“12C”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAC6BD7A2”,“b”:“55FDAB722000”,“o”:“F9B7A2”,“s”:"_ZN5mongo6Client19setOperationContextEPNS_16OperationContextE.cold
.133",“s+”:“18”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE484B65”,“b”:“55FDAB722000”,“o”:“2D62B65”,“s”:"_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE",“s+
“:“125”}}}
{“t”:{”$date”:“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE479AF7”,“b”:“55FDAB722000”,“o”:“2D57AF7”,“s”:"_ZN5mongo6Client20makeOperationContextEv",“s+”:“27”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE273AB2”,“b”:“55FDAB722000”,“o”:“2B51AB2”,“s”:"_ZN5mongo21KeysCollectionManager14PeriodicRunner18_doPeriodicRefr
eshEPNS_14ServiceContextESsNS_8DurationISt5ratioILl1ELl1000EEEE",“s+”:“162”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE275B87”,“b”:“55FDAB722000”,“o”:“2B53B87”,“s”:"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6th
readC4IZNS3_21KeysCollectionManager14PeriodicRunner5startEPNS3_14ServiceContextERKSsNS3_8DurationISt5ratioILl1ELl1000EEEEEUlvE_JELi0EEET_DpOT0
_EUlvE_EEEEE6_M_runEv",“s+”:“A7”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“55FDAE8E330F”,“b”:“55FDAB722000”,“o”:“31C130F”,“s”:“execute_native_thread_routine”,“s+”:“F”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“7F6D53BA0E65”,“b”:“7F6D53B99000”,“o”:“7E65”,“s”:“start_thread”,“s+”:“C5”}}}
{“t”:{"$date":“2021-07-08T11:58:57.313+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:31427, “ctx”:“xxxxxxing-keys-for-HMAC”,“msg”:" Frame: {frame
}",“attr”:{“frame”:{“a”:“7F6D538C988D”,“b”:“7F6D537CB000”,“o”:“FE88D”,“s”:“clone”,“s+”:“6D”}}}

1 Like

The backtrace certainly changes things.

I can see that the last normal code function being used before an assertion was thrown was one for mongo::client::setOperationContext().

void Client::setOperationContext(OperationContext* opCtx) {
    // We can only set the OperationContext once before resetting it.
    invariant(opCtx != nullptr && _opCtx == nullptr);
    _opCtx = opCtx;
}

OperationContext objects are widely used though, so that’s not so specific.

Where that OperationContext was being created or referenced from was KeysCollectionManager::PeriodicRunner::_doPeriodicRefresh(). There are two lines inside_doPeriodicRefresh() which use the makeOperationContext() function the backtrace includes, I’m not sure which it would be yet.

It appears that when ServiceContext::makeOperationContext() was called:

  • the following lines managed to pass an opCtx pointer that was null, OR
  • the _opCtx member in the the service context class had somehow already been made null.
    {
        stdx::lock_guard<Client> lk(*client);
        client->setOperationContext(opCtx.get());
    }

And that’s the code in the latest supported version v4.4.6. That’s still the version you’re using, right?

1 Like