PMM is down and not loading hosts inventory. Prometheus API DOWN

I am trying to run Prometheus API service Up and Running.

roo@pmm# pmm-admin check-network
PMM Network Status

Server Address | xxx.xxx.xxx.xxx
Client Address | xxx.xxx.xxx.xxx

  • System Time
    NTP Server (0.pool.ntp.org) | 2019-12-10 22:33:50 +0000 UTC
    PMM Server | 2019-12-10 22:33:50 +0000 GMT
    PMM Client | 2019-12-10 17:33:50 -0500 EST
    PMM Server Time Drift | OK
    PMM Client Time Drift | OK
    PMM Client to PMM Server Time Drift | OK

  • Connection: Client --> Server


SERVER SERVICE STATUS


Consul API OK
Prometheus API DOWN
Query Analytics API OK

Connection duration | 42.549592ms
Request duration | 5.284758ms
Full round trip | 47.83435ms

Prometheus is down. Please check if PMM server container runs properly.

On the PMM Server, i am seeing these error logs.

Dec 10 22:40:22 dedbpmm01 prometheus: level=error ts=2019-12-10T22:40:22.02986138Z caller=main.go:596 err=“Opening storage failed open block /srv/prometheus2/01DVBG22NNK2ZM7V3JPERR5JZT: open /srv/prometheus2/01DVBG22NNK2ZM7V3JPERR5JZT/tombstones: too many open files”
Dec 10 22:40:22 dedbpmm01 systemd: prometheus.service: main process exited, code=exited, status=1/FAILURE
Dec 10 22:40:22 dedbpmm01 systemd: Unit prometheus.service entered failed state.
Dec 10 22:40:22 dedbpmm01 systemd: prometheus.service failed.

Please advise on how to fix the issue and bring the home page back.

Current state of the PMM Home page is