I am trying to run Prometheus API service Up and Running.
roo@pmm# pmm-admin check-network
PMM Network Status
Server Address | xxx.xxx.xxx.xxx
Client Address | xxx.xxx.xxx.xxx
-
System Time
NTP Server (0.pool.ntp.org) | 2019-12-10 22:33:50 +0000 UTC
PMM Server | 2019-12-10 22:33:50 +0000 GMT
PMM Client | 2019-12-10 17:33:50 -0500 EST
PMM Server Time Drift | OK
PMM Client Time Drift | OK
PMM Client to PMM Server Time Drift | OK -
Connection: Client → Server
SERVER SERVICE STATUS
Consul API OK
Prometheus API DOWN
Query Analytics API OK
Connection duration | 42.549592ms
Request duration | 5.284758ms
Full round trip | 47.83435ms
Prometheus is down. Please check if PMM server container runs properly.
On the PMM Server, i am seeing these error logs.
Dec 10 22:40:22 dedbpmm01 prometheus: level=error ts=2019-12-10T22:40:22.02986138Z caller=main.go:596 err=“Opening storage failed open block /srv/prometheus2/01DVBG22NNK2ZM7V3JPERR5JZT: open /srv/prometheus2/01DVBG22NNK2ZM7V3JPERR5JZT/tombstones: too many open files”
Dec 10 22:40:22 dedbpmm01 systemd: prometheus.service: main process exited, code=exited, status=1/FAILURE
Dec 10 22:40:22 dedbpmm01 systemd: Unit prometheus.service entered failed state.
Dec 10 22:40:22 dedbpmm01 systemd: prometheus.service failed.
Please advise on how to fix the issue and bring the home page back.
Current state of the PMM Home page is
