I am getting frequent crashes in my postgres 9.5 and 12 databases . Cannot figure out what might be the reason for this. Can someone guide me to the right direction on what config or linux setting to look for.
The crashes usually occur once in 2 days.
Sep 6 18:20:36 VMPROD postgres[146238]: [37-1] [unknown] rw_user x.x.x.x(46180) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[146238]: [37-2] [unknown] rw_user x.x.x.x(46180) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[146238]: [37-3] [unknown] rw_user x.x.x.x(46180) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[145289]: [37-1] [unknown] rw_user x.x.x.x(48552) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[145289]: [37-2] [unknown] rw_user x.x.x.x(48552) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[145289]: [37-3] [unknown] rw_user x.x.x.x(48552) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[145289]: [37-4] [unknown] rw_user x.x.x.x(48552) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[162093]: [37-1] [unknown] rw_user x.x.x.x(51472) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.691:59290589): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=162093 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[141743]: [37-1] [unknown] rw_user 192.168.163.113(60790) 192.168.163.113WARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[141743]: [37-2] [unknown] rw_user 192.168.163.113(60790) 192.168.163.113DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[141743]: [37-3] [unknown] rw_user 192.168.163.113(60790) 192.168.163.113HINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[41237]: [37-1] [unknown] rw_user x.x.x.x(43992) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[41237]: [37-2] [unknown] rw_user x.x.x.x(43992) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD kernel: [22369632.208751] Pid 139659(postgres) over core_pipe_limit
Sep 6 18:20:36 VMPROD kernel: [22369632.208755] Skipping core dump
Sep 6 18:20:36 VMPROD kernel: [22369632.211748] Pid 138455(postgres) over core_pipe_limit
Sep 6 18:20:36 VMPROD kernel: [22369632.211750] Skipping core dump
Sep 6 18:20:36 VMPROD kernel: [22369632.213554] Pid 134566(postgres) over core_pipe_limit
Sep 6 18:20:36 VMPROD kernel: [22369632.213558] Skipping core dump
Sep 6 18:20:36 VMPROD kernel: [22369632.216834] Pid 159130(postgres) over core_pipe_limit
Sep 6 18:20:36 VMPROD kernel: [22369632.216837] Skipping core dump
Sep 6 18:20:36 VMPROD postgres[41237]: [37-3] [unknown] rw_user x.x.x.x(43992) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[41237]: [37-4] [unknown] rw_user x.x.x.x(43992) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.693:59290590): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=26483 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[142795]: [37-1] [unknown] rw_user 192.168.163.113(60986) 192.168.163.113WARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.694:59290591): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=142795 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[41328]: [37-1] [unknown] rw_user x.x.x.x(44058) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[41328]: [37-2] [unknown] rw_user x.x.x.x(44058) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[41328]: [37-3] [unknown] rw_user x.x.x.x(44058) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[41328]: [37-4] [unknown] rw_user x.x.x.x(44058) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[13025]: [37-4] [unknown] rw_user x.x.x.x(55774) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[139614]: [37-1] [unknown] rw_user x.x.x.x(46634) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[139614]: [37-2] [unknown] rw_user x.x.x.x(46634) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[139614]: [37-3] [unknown] rw_user x.x.x.x(46634) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[139614]: [37-4] [unknown] rw_user x.x.x.x(46634) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[136895]: [37-1] [unknown] rw_user 192.168.163.113(59716) 192.168.163.113WARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[136895]: [37-2] [unknown] rw_user 192.168.163.113(59716) 192.168.163.113DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[136895]: [37-3] [unknown] rw_user 192.168.163.113(59716) 192.168.163.113HINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[136895]: [37-4] [unknown] rw_user 192.168.163.113(59716) 192.168.163.113LOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[79120]: [37-1] [unknown] rw_user x.x.x.x(36144) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[79120]: [37-2] [unknown] rw_user x.x.x.x(36144) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[79120]: [37-3] [unknown] rw_user x.x.x.x(36144) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[79120]: [37-4] [unknown] rw_user x.x.x.x(36144) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[130320]: [37-1] [unknown] rw_user x.x.x.x(34580) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[130320]: [37-2] [unknown] rw_user x.x.x.x(34580) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[130320]: [37-3] [unknown] rw_user x.x.x.x(34580) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[130320]: [37-4] [unknown] rw_user x.x.x.x(34580) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[7934]: [37-1] [unknown] rw_user x.x.x.x(51014) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[56072]: [38-2] [unknown] rw_user x.x.x.x(38754) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[7934]: [37-2] [unknown] rw_user x.x.x.x(51014) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[7934]: [37-3] [unknown] rw_user x.x.x.x(51014) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[7934]: [37-4] [unknown] rw_user x.x.x.x(51014) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.702:59290592): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=56072 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[136807]: [37-1] [unknown] rw_user x.x.x.x(44148) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[136807]: [37-2] [unknown] rw_user x.x.x.x(44148) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[136807]: [37-3] [unknown] rw_user x.x.x.x(44148) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[136807]: [37-4] [unknown] rw_user x.x.x.x(44148) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[136288]: [37-1] [unknown] rw_user x.x.x.x(44038) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[136288]: [37-2] [unknown] rw_user x.x.x.x(44038) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[136288]: [37-3] [unknown] rw_user x.x.x.x(44038) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[136288]: [37-4] [unknown] rw_user x.x.x.x(44038) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.706:59290593): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=134566 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[134566]: [37-1] [unknown] rw_user x.x.x.x(47740) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[146072]: [37-2] [unknown] rw_user x.x.x.x(47924) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[146072]: [37-3] [unknown] rw_user x.x.x.x(47924) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[146072]: [37-4] [unknown] rw_user x.x.x.x(47924) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[132857]: [37-1] [unknown] rw_user x.x.x.x(44888) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[132857]: [37-2] [unknown] rw_user x.x.x.x(44888) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[132857]: [37-3] [unknown] rw_user x.x.x.x(44888) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[132857]: [37-4] [unknown] rw_user x.x.x.x(44888) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[14621]: [37-1] [unknown] rw_user x.x.x.x(59122) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[115469]: [37-2] [unknown] rw_user x.x.x.x(60132) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[115469]: [37-3] [unknown] rw_user x.x.x.x(60132) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[115469]: [37-4] [unknown] rw_user x.x.x.x(60132) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.709:59290594): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=14621 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[115458]: [37-1] [unknown] rw_user x.x.x.x(44634) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[115458]: [37-2] [unknown] rw_user x.x.x.x(44634) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[115458]: [37-3] [unknown] rw_user x.x.x.x(44634) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[115458]: [37-4] [unknown] rw_user x.x.x.x(44634) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[100960]: [37-1] [unknown] rw_user x.x.x.x(33668) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[100960]: [37-2] [unknown] rw_user x.x.x.x(33668) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[100960]: [37-3] [unknown] rw_user x.x.x.x(33668) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[100960]: [37-4] [unknown] rw_user x.x.x.x(33668) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[96043]: [37-1] [unknown] rw_user x.x.x.x(60990) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[96043]: [37-2] [unknown] rw_user x.x.x.x(60990) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[96043]: [37-3] [unknown] rw_user x.x.x.x(60990) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[96043]: [37-4] [unknown] rw_user x.x.x.x(60990) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[108276]: [37-1] [unknown] rw_user x.x.x.x(35288) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD audispd: node=VMPROD type=ANOM_ABEND msg=audit(1662468636.711:59290595): auid=4294967295 uid=1005 gid=1006 ses=4294967295 pid=108276 comm="postgres" reason="memory violation" sig=11
Sep 6 18:20:36 VMPROD postgres[139176]: [37-1] [unknown] rw_user x.x.x.x(51352) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[139176]: [37-2] [unknown] rw_user x.x.x.x(51352) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[139176]: [37-3] [unknown] rw_user x.x.x.x(51352) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 6 18:20:36 VMPROD postgres[139176]: [37-4] [unknown] rw_user x.x.x.x(51352) x.x.x.xLOCATION: quickdie, postgres.c:2637
Sep 6 18:20:36 VMPROD postgres[139693]: [37-1] [unknown] rw_user x.x.x.x(42094) x.x.x.xWARNING: 57P02: terminating connection because of crash of another server process
Sep 6 18:20:36 VMPROD postgres[139693]: [37-2] [unknown] rw_user x.x.x.x(42094) x.x.x.xDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 6 18:20:36 VMPROD postgres[139693]: [37-3] [unknown] rw_user x.x.x.x(42094) x.x.x.xHINT: In a moment you should be able to reconnect to the database and repeat your command.