Not the answer you need?
Register and ask your own question!

PMM is down and not loading hosts inventory. Prometheus API DOWN

I am trying to run Prometheus API service Up and Running.

[email protected]# pmm-admin check-network
PMM Network Status

Server Address |
Client Address |

* System Time
NTP Server ( | 2019-12-10 22:33:50 +0000 UTC
PMM Server | 2019-12-10 22:33:50 +0000 GMT
PMM Client | 2019-12-10 17:33:50 -0500 EST
PMM Server Time Drift | OK
PMM Client Time Drift | OK
PMM Client to PMM Server Time Drift | OK

* Connection: Client --> Server


Consul API OK
Prometheus API DOWN
Query Analytics API OK

Connection duration | 42.549592ms
Request duration | 5.284758ms
Full round trip | 47.83435ms

Prometheus is down. Please check if PMM server container runs properly.

On the PMM Server, i am seeing these error logs.

Dec 10 22:40:22 dedbpmm01 prometheus: level=error ts=2019-12-10T22:40:22.02986138Z caller=main.go:596 err="Opening storage failed open block /srv/prometheus2/01DVBG22NNK2ZM7V3JPERR5JZT: open /srv/prometheus2/01DVBG22NNK2ZM7V3JPERR5JZT/tombstones: too many open files"
Dec 10 22:40:22 dedbpmm01 systemd: prometheus.service: main process exited, code=exited, status=1/FAILURE
Dec 10 22:40:22 dedbpmm01 systemd: Unit prometheus.service entered failed state.
Dec 10 22:40:22 dedbpmm01 systemd: prometheus.service failed.

Please advise on how to fix the issue and bring the home page back.

Current state of the PMM Home page is
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.