After updating to PMM 2.3.0, upon reboot fails to start properly. Doing systemctl restart supervisord gives:
‘Job for supervisord.service failed because a timeout was exceeded. See “systemctl status supervisord.service” and “journalctl -xe” for details’
So it seems supervisord isnt setup properly. I can start it fine with “/usr/bin/supervisord -c /etc/supervisord.conf” and that is running fine.
Any idea what would be wrong?
The supervisord log says (Also not sure what is the issue with qan-api2 exit status 1.):
2020-02-28 08:40:26,745 INFO Included extra file “/etc/supervisord.d/pmm.ini” during parsing
2020-02-28 08:40:26,745 INFO Included extra file “/etc/supervisord.d/prometheus.ini” during parsing
2020-02-28 08:40:26,745 INFO Included extra file “/etc/supervisord.d/qan-api2.ini” during parsing
2020-02-28 08:40:26,745 INFO Set uid to user 0 succeeded
2020-02-28 08:40:26,758 INFO RPC interface ‘supervisor’ initialized
2020-02-28 08:40:26,758 INFO supervisord started with pid 12958
2020-02-28 08:40:27,762 INFO spawned: ‘postgresql’ with pid 12963
2020-02-28 08:40:27,764 INFO spawned: ‘clickhouse’ with pid 12964
2020-02-28 08:40:27,767 INFO spawned: ‘grafana’ with pid 12965
2020-02-28 08:40:27,770 INFO spawned: ‘nginx’ with pid 12966
2020-02-28 08:40:27,773 INFO spawned: ‘cron’ with pid 12967
2020-02-28 08:40:27,777 INFO spawned: ‘prometheus’ with pid 12968
2020-02-28 08:40:27,781 INFO spawned: ‘dashboard-upgrade’ with pid 12969
2020-02-28 08:40:27,783 INFO spawned: ‘qan-api2’ with pid 12970
2020-02-28 08:40:27,786 INFO spawned: ‘pmm-managed’ with pid 12971
2020-02-28 08:40:27,789 INFO spawned: ‘pmm-agent’ with pid 12977
2020-02-28 08:40:27,819 INFO success: dashboard-upgrade entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2020-02-28 08:40:27,827 INFO exited: qan-api2 (exit status 1; not expected)
2020-02-28 08:40:28,541 INFO exited: dashboard-upgrade (exit status 0; expected)
2020-02-28 08:40:28,904 INFO success: postgresql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: prometheus entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,908 INFO spawned: ‘qan-api2’ with pid 13067
2020-02-28 08:40:28,908 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,908 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,932 INFO exited: qan-api2 (exit status 1; not expected)
2020-02-28 08:40:31,010 INFO spawned: ‘qan-api2’ with pid 13114
2020-02-28 08:40:32,027 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:41:56,393 WARN received SIGTERM indicating exit request
2020-02-28 08:41:56,394 INFO waiting for postgresql, pmm-agent, grafana, cron, nginx, prometheus, pmm-managed, qan-api2, clickhouse to die
2020-02-28 08:41:56,410 INFO exited: grafana (exit status 1; not expected)
2020-02-28 08:41:56,410 INFO exited: cron (exit status 0; expected)
2020-02-28 08:41:56,411 INFO exited: nginx (exit status 0; expected)
2020-02-28 08:41:56,411 INFO stopped: pmm-agent (exit status 0)
2020-02-28 08:41:56,411 INFO exited: qan-api2 (exit status 0; expected)
2020-02-28 08:41:56,421 INFO stopped: pmm-managed (exit status 0)
2020-02-28 08:41:56,465 INFO exited: postgresql (exit status 0; expected)
2020-02-28 08:41:56,949 INFO exited: clickhouse (exit status 0; expected)
2020-02-28 08:42:00,178 INFO waiting for prometheus to die
2020-02-28 08:42:03,455 INFO waiting for prometheus to die
2020-02-28 08:42:06,701 INFO waiting for prometheus to die
2020-02-28 08:42:07,683 INFO stopped: prometheus (exit status 0)