Not the answer you need?
Register and ask your own question!

VM server: Supervisord fails to start properly

fredricjfredricj EntrantCurrent User Role Contributor
After updating to PMM 2.3.0, upon reboot fails to start properly. Doing systemctl restart supervisord gives:
'Job for supervisord.service failed because a timeout was exceeded. See "systemctl status supervisord.service" and "journalctl -xe" for details'
So it seems supervisord isnt setup properly. I can start it fine with "/usr/bin/supervisord -c /etc/supervisord.conf" and that is running fine.
Any idea what would be wrong?

The supervisord log says (Also not sure what is the issue with qan-api2 exit status 1.):
2020-02-28 08:40:26,745 INFO Included extra file "/etc/supervisord.d/pmm.ini" during parsing
2020-02-28 08:40:26,745 INFO Included extra file "/etc/supervisord.d/prometheus.ini" during parsing
2020-02-28 08:40:26,745 INFO Included extra file "/etc/supervisord.d/qan-api2.ini" during parsing
2020-02-28 08:40:26,745 INFO Set uid to user 0 succeeded
2020-02-28 08:40:26,758 INFO RPC interface 'supervisor' initialized
2020-02-28 08:40:26,758 INFO supervisord started with pid 12958
2020-02-28 08:40:27,762 INFO spawned: 'postgresql' with pid 12963
2020-02-28 08:40:27,764 INFO spawned: 'clickhouse' with pid 12964
2020-02-28 08:40:27,767 INFO spawned: 'grafana' with pid 12965
2020-02-28 08:40:27,770 INFO spawned: 'nginx' with pid 12966
2020-02-28 08:40:27,773 INFO spawned: 'cron' with pid 12967
2020-02-28 08:40:27,777 INFO spawned: 'prometheus' with pid 12968
2020-02-28 08:40:27,781 INFO spawned: 'dashboard-upgrade' with pid 12969
2020-02-28 08:40:27,783 INFO spawned: 'qan-api2' with pid 12970
2020-02-28 08:40:27,786 INFO spawned: 'pmm-managed' with pid 12971
2020-02-28 08:40:27,789 INFO spawned: 'pmm-agent' with pid 12977
2020-02-28 08:40:27,819 INFO success: dashboard-upgrade entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2020-02-28 08:40:27,827 INFO exited: qan-api2 (exit status 1; not expected)
2020-02-28 08:40:28,541 INFO exited: dashboard-upgrade (exit status 0; expected)
2020-02-28 08:40:28,904 INFO success: postgresql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,905 INFO success: prometheus entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,908 INFO spawned: 'qan-api2' with pid 13067
2020-02-28 08:40:28,908 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,908 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:40:28,932 INFO exited: qan-api2 (exit status 1; not expected)
2020-02-28 08:40:31,010 INFO spawned: 'qan-api2' with pid 13114
2020-02-28 08:40:32,027 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:41:56,393 WARN received SIGTERM indicating exit request
2020-02-28 08:41:56,394 INFO waiting for postgresql, pmm-agent, grafana, cron, nginx, prometheus, pmm-managed, qan-api2, clickhouse to die
2020-02-28 08:41:56,410 INFO exited: grafana (exit status 1; not expected)
2020-02-28 08:41:56,410 INFO exited: cron (exit status 0; expected)
2020-02-28 08:41:56,411 INFO exited: nginx (exit status 0; expected)
2020-02-28 08:41:56,411 INFO stopped: pmm-agent (exit status 0)
2020-02-28 08:41:56,411 INFO exited: qan-api2 (exit status 0; expected)
2020-02-28 08:41:56,421 INFO stopped: pmm-managed (exit status 0)
2020-02-28 08:41:56,465 INFO exited: postgresql (exit status 0; expected)
2020-02-28 08:41:56,949 INFO exited: clickhouse (exit status 0; expected)
2020-02-28 08:42:00,178 INFO waiting for prometheus to die
2020-02-28 08:42:03,455 INFO waiting for prometheus to die
2020-02-28 08:42:06,701 INFO waiting for prometheus to die
2020-02-28 08:42:07,683 INFO stopped: prometheus (exit status 0)

Comments

  • PeterPeter Percona CEO Percona Moderator Role

    Hm,

    Per log "2020-02-28 08:41:56,393 WARN received SIGTERM indicating exit request"

    Something sent it SIGTERM so it existed... did it only happens once after reboot or happens all the time ?

  • fredricjfredricj Entrant Current User Role Contributor

    It happens all the time after the update from the initial install. The SIGTERM is from systemctl after 90s when it thinks supervisord didnt start properly

Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.