Docker timesync

This is my first experience with docker, and I am trying to diagnose why Grafana monitor essentially stopped working. The first 15 minutes it did and was showing stats, then after everything was empty. I figured out how to get into the docker console and checked the prometheus.log and that has filled up with:

time=“2016-09-14T15:29:27Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“db2”, job=“mysql”} => 0 @[1473866966.329] source=“scrape.go:500”

So I checked the time between both the host and the docker image:
Host: Thu Sep 15 07:09:17 CDT 2016
Docker: Thu Sep 15 12:09:00 UTC 2016

I am guessing the next step is to get the timesynced, but that I have not figured out how to do. Host is Oracle Linux 7.2

The different timezones are not the problem if time is in sync. Also this error should not be critical. How many of those do you see? What does [url]http://server/prometheus/targets[/url] page show? (you can watch it if endpoints are OK)

I was thinking it was timezone/timesync issue, but after looking at the page you suggested I see something else for all servers:
State: Down
Error: context deadline exceeded

I just did pmm-admin check-network and my remote state is “down”. I have disabled firewall and SELinux is set to permissive. I can ping the PMM server just fine. Any suggestions?

Is there any network latency between client and server? Is it the same network? (pmm-admin check-network shows some numbers).

Also you can enter container “docker exec -ti pmm-server bash” and try “curl http://endpoint:numeric-port-number” where endpoint is target in problem from that web page.

One server gives this for pmm-admin check-network:
Connection duration | 454.593µs
Request duration | 1.451193ms
Full round trip | 1.905786ms
Another gives this;
Connection duration | 12.256637ms
Request duration | 13.10624ms
Full round trip | 25.362877ms

I did try “docker exec -ti pmm-server bash” and what I found may be the source of my problems. I get just timeouts using curl http://SERVER:PORT, I also can not ping any other address except for the host server. So the host at 10.10.7.150 responds, neighbor at 10.10.7.6 does not. From the host server I can however ping both MySQL servers (one locally. other on remote subnet via VPN).

Maybe explain why Query Analytics works, does show results, but Metrics Monitor is empty. Is one pushing via the agent and the other attempting to poll to the remote agent?

So network latency is a problem to scrape metrics with 1s resolution.
Try recreating server container (preserving data one) with the option “-e METRICS_RESOLUTION=5s”.
It will set minimal resolution to 5s. Bigger value will not work with the current implementation.

Figured it all out, and got it working :slight_smile:

Like I said, this was first time I ever worked with docker before, and from what I first understood was that I would use the IP address of the host OS. But when the docker VM connects to a remote server, it is in fact using its own internal 172.17.0.2 address, and not proxying through the hosts IP. That presented a problem for remote VMs that had no knowledge of where to find that subnet. 2nd factor was that 2 of the remote servers were located though a VPN and I also had to add in the 172.17.0.2 to an ACL on both firewalls. Once I added in static route at the gateway and the ACLs, all 3 MySQL servers showed up in Grafana! So overall was slight hiccup and good learning process to docker, but well worth it as this PMM is really amazing. And thanks for your assistance on this weber, very much appreciated.