Grafana has failed to load

[Jyoti_Rajai]

Hi Team,

Greetings !

I am facing issue depicted in below snapshot frequently on PMM.

Please guide me to fix this issue.

Thank you in advance.

Warm Regards,
Jyoti Rajai

[Jyoti_Rajai]

FJYI : I have already tired solution provided on this link: HTTP Error Bad Gateway when using prometheus · Issue #14629 · grafana/grafana · GitHub

But nothing helps.

[Naresh9999]

Hi @Jyoti_Rajai

What are the system requirements.?

1. Is CPU and RAM are doing good?
2. Check your internet, due to the internet slowness we may get this issue.
3. Did you restarted the pmm server?
4. Check the pmm2 server status weather it is healthy or not? docker ps -a

[Vadim_Yalovets]

Hi Jyoti,

Could you check if it’s enough disk space in the container?

[Jyoti_Rajai]

@Naresh9999

What are the system requirements.?

1. Is CPU and RAM are doing good? CPU - 8 core , 32GB RAM
2. Check your internet, due to the internet slowness we may get this issue. – No issue in internet
3. Did you restarted the pmm server? – Nope
4. Check the pmm2 server status weather it is healthy or not? docker ps -a – Healthy

docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0d3c3316ff30 percona/pmm-server:latest “/opt/entrypoint.sh” 3 weeks ago 3 weeks ago 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp pmm-server
9fc459a9fdd6 percona/pmm-server:latest “/bin/true” 3 weeks ago pmm-data

[Jyoti_Rajai]

Could you check if it’s enough disk space in the container? – Yeah, 400+gb free.

[Naresh9999]

Hi @Jyoti_Rajai

Seems everything looks good.
Can you restart the PMM server and try once?

docker stop 0d3c3316ff30

docker stop 9fc459a9fdd6

systemctl restart docker

[Jyoti_Rajai]

docker stop 0d3c3316ff30
0d3c3316ff30

docker stop 9fc459a9fdd6
9fc459a9fdd6

systemctl restart docker
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0d3c3316ff30 percona/pmm-server:latest “/opt/entrypoint.sh” 3 weeks ago Up 9 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp pmm-server
9fc459a9fdd6 percona/pmm-server:latest “/bin/true” 3 weeks ago pmm-data
[root@testhodb4node3 ~]#

[Jyoti_Rajai]

It didn’t helped.

Now, URL is giving below error :

500 Internal Server Error

nginx

[Naresh9999]

Hi @Jyoti_Rajai

Can you please re-check the PMM serer status, weather it is healthy or not?

[Jyoti_Rajai]

500 Internal Server Error

nginx

[Naresh9999]

Hi @Jyoti_Rajai

After restarting the PMM server, is it healthy or not?

docker ps -a

and, can you try to clear the cache or try it in different browser?

[Jyoti_Rajai]

@Naresh9999

and, can you try to clear the cache or try it in different browser?

Didn’t helped.

docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0d3c3316ff30 percona/pmm-server:latest “/opt/entrypoint.sh” 3 weeks ago Up 9 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp pmm-server
9fc459a9fdd6 percona/pmm-server:latest “/bin/true” 3 weeks ago pmm-data

[Naresh9999]

Hi @Jyoti_Rajai

For me it is showing healthy, but for you its not showing anything either its health/unhealthy in the docker status.

and, can you please check the PMM server container log file.

docker logs --tail=100 be330f386f3f
docker logs --tail=100

[Jyoti_Rajai]

@Naresh9999

docker logs --tail=100

[root@testhodb4node3 ~]# docker logs --tail=20 f6b1ae985878
2022-02-09 12:53:57,802 INFO exited: postgresql (exit status 1; not expected)
2022-02-09 12:54:04,359 INFO spawned: ‘postgresql’ with pid 11516
2022-02-09 12:54:04,372 INFO exited: postgresql (exit status 1; not expected)
2022-02-09 12:54:11,803 INFO spawned: ‘postgresql’ with pid 11517
2022-02-09 12:54:11,816 INFO exited: postgresql (exit status 1; not expected)
2022-02-09 12:54:20,669 INFO spawned: ‘postgresql’ with pid 11518
2022-02-09 12:54:20,680 INFO exited: postgresql (exit status 1; not expected)
2022-02-09 12:54:29,819 INFO spawned: ‘postgresql’ with pid 11519
2022-02-09 12:54:29,829 INFO exited: postgresql (exit status 1; not expected)
2022-02-09 12:54:40,830 INFO spawned: ‘postgresql’ with pid 11520
2022-02-09 12:54:40,840 INFO exited: postgresql (exit status 1; not expected)
2022-02-09 12:54:41,830 INFO gave up: postgresql entered FATAL state, too many start retries too quickly
2022-02-10 00:01:01,705 INFO reaped unknown pid 11722
2022-02-10 01:01:01,735 INFO reaped unknown pid 11741
2022-02-10 02:01:01,764 INFO reaped unknown pid 11760
2022-02-10 03:32:01,806 INFO reaped unknown pid 11779
2022-02-11 00:01:01,447 INFO reaped unknown pid 12178
2022-02-11 01:01:01,476 INFO reaped unknown pid 12197
2022-02-11 02:01:01,509 INFO reaped unknown pid 12216
2022-02-11 03:33:01,549 INFO reaped unknown pid 12235

[Naresh9999]

Hi @Jyoti_Rajai

Seems postgresql DB has some issue, it did not start properly. Can you download the server Diagnostics and check the details. It may help this issue.

Can you stop the docker service and restart the Linux server node then try once.

[Jyoti_Rajai]

@Naresh9999

Seems postgresql DB has some issue, it did not start properly. Can you download the server Diagnostics and check the details. It may help this issue.

I am unable to load the page you shown in screenshot.

[Naresh9999]

Hi @Jyoti_Rajai

Can you do the below steps ?

1. docker stop

2. stop the docker service

3. Reboot the Linux server node then try once.?

[Jyoti_Rajai]

Why to reboot linux server ? That doesn’t sounds proper.

Additionally, what i have found is the docker stores all data in memory instead of given path for volume.

[alwaysdba]

@Jyoti_Rajai I am also facing similar kind of issue. PMM server is working fine and but grafana is not showing up.