# Grafana Graph works only sporadic

**URL:** <https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171>\
**Category:** PMM 1.x\
**Created:** [October 26, 2016, 5:40am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171 "2016-10-26T05:40:07Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![shoman](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@shoman](https://forums.percona.com/u/shoman)\
**Post date:** [October 26, 2016, 5:40am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/1 "2016-10-26T05:40:07Z")

</div>

A have installed PMM 1.05 and adding some hosts with linux:metrics and mysql:metrics, but Grafana is showing the graph very sporadic.

In times i have no graphs the client state is down.

```auto

SERVICE TYPE NAME REMOTE ENDPOINT STATUS
-------------- ------- ---------------------- -------
linux:metrics db-1-1 144.76.15.145:42000 DOWN 
mysql:metrics db-1-1 144.76.15.145:42002 DOWN 

```

I run pmm-server as following to prevent sync issues.

```auto
docker run -d \
-p 44444:443 \
-e METRICS_RESOLUTION=5s \
--volumes-from pmm-data \
--name pmm-server \
-v /usr/share/pmm:/etc/nginx/ssl \
--restart always \
percona/pmm-server:1.0.5

```

But i don’t think this a problem

```auto
Connection duration | 323.078µs
Request duration | 4.35313ms
Full round trip | 4.676208ms

```

```auto
PING db-1-1 (144.76.15.145) 56(84) bytes of data.
64 bytes from db-1-1 (x.x.x.x): icmp_seq=1 ttl=60 time=0.311 ms
64 bytes from db-1-1 (x.x.x.x): icmp_seq=2 ttl=60 time=0.333 ms
64 bytes from db-1-1 (x.x.x.x): icmp_seq=3 ttl=60 time=0.684 ms

```

```auto
Server side: Wed Oct 26 10:56:51 UTC 2016
Client Side: Wed Oct 26 12:56:51 CEST 2016

```

If i take a look in /var/log/prometheus.log there are no error entries. Sometimes, there are absolute no entries over a few hours.

I also can do in time i get no data:

```auto
root&#64;756c3afeceba:/opt# curl http://db-1-1:42002
<html>
<head><title>MySQL 3-in-1 exporter</title></head>
<body>
<h1>MySQL 3-in-1 exporter</h1>
<li><a href="/metrics-hr">high-res metrics</a></li>
<li><a href="/metrics-mr">medium-res metrics</a></li>
<li><a href="/metrics-lr">low-res metrics</a></li>
</body>
</html>

```

I have no idea, why sometimes data is available an often not.

 ![photoid=46261](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/7/7831e70a00c3a1735ff3233290aed5486122fe3a.jpeg)

---

<div class="post-metadata">

**Author:** ![weber](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@weber](https://forums.percona.com/u/weber)\
**Post date:** [October 26, 2016, 6:06am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/2 "2016-10-26T06:06:20Z")

</div>

You can watch how the container performs `docker stats pmm-server`. May be the monitor server is not having enough resources.

---

<div class="post-metadata">

**Author:** ![shoman](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@shoman](https://forums.percona.com/u/shoman)\
**Post date:** [October 26, 2016, 6:14am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/3 "2016-10-26T06:14:26Z")

</div>

Hi,

thanks for the fast answer.

On the physical machine there are more as enough resources and here is the output from docker:

```auto
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
pmm-server 3.13% 0 B / 0 B 0.00% 172.2 MB / 224.5 MB 116.3 MB / 294.9 GB 0

```

I think it’s another problem.

---

<div class="post-metadata">

**Author:** ![geoiii](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/geoiii/32/1147_2.png) [@geoiii](https://forums.percona.com/u/geoiii)\
**Post date:** [October 26, 2016, 2:44pm UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/4 "2016-10-26T14:44:35Z")

</div>

I have found that I needed to bump up the memory in the docker with:

METRICS\_MEMORY=2097152

---

<div class="post-metadata">

**Author:** ![geoiii](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/geoiii/32/1147_2.png) [@geoiii](https://forums.percona.com/u/geoiii)\
**Post date:** [October 26, 2016, 3:57pm UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/5 "2016-10-26T15:57:26Z")

</div>

I have been having this (or similar issue) and it seems to be that it runs out of memory to ingest the metrics coming in to prometheus.  
And usually when it dies it really dies and I have to restart the docker.

I have been bumping up the memory with this command:

METRICS\_MEMORY=2097152

In the docker run command line. It has some fairly small default - that was working fine until I added one too many servers or has a peak in events.

You might take a look at the Prometheus graph in grafana.

My running server looks like this when I run `docker stats pmm-server`

```auto
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O

pmm-server 71.24% 6.77 GB / 7.934 GB 85.33% 6.932 GB / 379.2 MB 3.766 GB / 18.15 GB

```

---

<div class="post-metadata">

**Author:** ![shoman](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@shoman](https://forums.percona.com/u/shoman)\
**Post date:** [October 27, 2016, 3:21am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/6 "2016-10-27T03:21:07Z")

</div>

I test it again …

… adding slowly monitored hosts to the pmm server.

After doing this for the first 9 server only 3 minutes after adding number 10 the monitoring stopped completely.

Again there are no log entries in /var/log/prometheus.log and ALL metrics on ALL servers are in running state, but without connectivity to the server:

```auto
SERVICE TYPE NAME REMOTE ENDPOINT STATUS
-------------- ------- ---------------------- -------
linux:metrics db-1-2 x.x.x.x:42000 DOWN 
mysql:metrics db-1-2 x.x.x.x:42002 DOWN

```

PMM is a nice and helpfull tool, but it seems not very robust and we are not able to work with it at the moment.

What can I do?

![photoid=46278](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/d/dbe0998d6a1bca3d6edfb7c79e07e813ad3bf6fd.jpeg)

---

<div class="post-metadata">

**Author:** ![weber](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@weber](https://forums.percona.com/u/weber)\
**Post date:** [October 27, 2016, 3:57am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/7 "2016-10-27T03:57:11Z")

</div>

What is OS and docker version?

When the connectivity is lost, does this work `curl http://db-1-2:42000/metrics` from inside the container?

---

<div class="post-metadata">

**Author:** ![shoman](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@shoman](https://forums.percona.com/u/shoman)\
**Post date:** [October 27, 2016, 5:56am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/8 "2016-10-27T05:56:35Z")

</div>

> [@weber;46280](#):
>
> What is OS and docker version?
> 
> When the connectivity is lost, does this work `curl http://db-1-2:42000/metrics` from inside the container?

```auto
Distributor ID: Debian
Description: Debian GNU/Linux 8.6 (jessie)
Release: 8.6
Codename: jessie

```

```auto
Docker version 1.12.2, build bb80604

```

If I run curl [http://db-1-2:42000/metrics](http://db-1-2:42000/metrics) from inside the container:

```auto
process_cpu_seconds_total 23.42
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.370112e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.47749268343e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.7041792e+07
...

```

Since my last post i got 3 times data without changing anything on the infrastructure.

It’s strange.

---

<div class="post-metadata">

**Author:** ![weber](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@weber](https://forums.percona.com/u/weber)\
**Post date:** [October 27, 2016, 6:06am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/9 "2016-10-27T06:06:15Z")

</div>

When `pmm-admin check-network` shows DOWN for endpoint, does curl from inside container for the same one work? I don’t think it’s possible.

---

<div class="post-metadata">

**Author:** ![shoman](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@shoman](https://forums.percona.com/u/shoman)\
**Post date:** [October 27, 2016, 6:57am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/10 "2016-10-27T06:57:50Z")

</div>

Thx weber and geoiii,

i run the container now as following:

```auto

docker run -d \
-p 44444:443 \
-e METRICS_RESOLUTION=5s \
-e METRICS_MEMORY=4124672 \
--volumes-from pmm-data \
--name pmm-server \
-v /usr/share/pmm:/etc/nginx/ssl \
-v /etc/localtime:/etc/localtime:ro \
--restart always \
percona/pmm-server:1.0.5

```

Sometimes the time in the docker container changes, so i sync the timezone with the local ones and start the container with more memory.

At the moment all is working fine.

The stats are now more plausible, but mem usage looks a little bit strange. There are no values, but it works 🙂

```auto
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
pmm-server 10.72% 0 B / 0 B 0.00% 77.36 MB / 3.013 MB 105.8 MB / 1.107 GB 0

```

---

<div class="post-metadata">

**Author:** ![weber](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@weber](https://forums.percona.com/u/weber)\
**Post date:** [October 27, 2016, 8:36am UTC](https://forums.percona.com/t/grafana-graph-works-only-sporadic/5171/11 "2016-10-27T08:36:54Z")

</div>

Oh, I misread, I thought it was your post about METRICS\_MEMORY as well 🙂