Grafana + Very High CPU Docker Image

Percona Monitoring and Management (PMM) PMM 2.x
1

I am attempting to figure out where the bottleneck or issue might be… At times I get the following on the web page

{“traceID”:“”}

In the /var/log/messages I get alot of these… Yet I do not see the server being stressed

the only item I see if that when it happens the Context Switches Spike and they belong to the grafana process (not a process I would have expected to see high switches on)

Suggestions? Grafana 9.2.20 in a docker container
Disk IO and CPU look fine as well as memory when it happens…

Jul 31 09:52:04 servername dockerd: time="2023-07-31T09:52:04.983312807-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:52:13 servername dockerd: time="2023-07-31T09:52:13.063232203-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:52:30 servername dockerd: time="2023-07-31T09:52:30.475737909-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:52:32 servername containerd: time="2023-07-31T09:52:32.478868673-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:52:32 servername containerd: time="2023-07-31T09:52:32.478975370-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:52:41 servername dockerd: time="2023-07-31T09:52:41.633827806-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:53:17 servername dockerd: time="2023-07-31T09:53:17.912492731-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:53:19 servername containerd: time="2023-07-31T09:53:19.930180112-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:53:19 servername containerd: time="2023-07-31T09:53:19.930269531-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:53:47 servername dockerd: time="2023-07-31T09:53:47.824246803-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:54:08 servername dockerd: time="2023-07-31T09:54:08.350363748-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:54:23 servername dockerd: time="2023-07-31T09:54:23.687149236-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:54:25 servername containerd: time="2023-07-31T09:54:25.690803296-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:54:25 servername containerd: time="2023-07-31T09:54:25.690904905-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:54:34 servername dockerd: time="2023-07-31T09:54:34.863299809-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:54:36 servername containerd: time="2023-07-31T09:54:36.867166390-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:54:36 servername containerd: time="2023-07-31T09:54:36.867123055-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:54:39 servername dockerd: time="2023-07-31T09:54:39.867738910-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:54:41 servername containerd: time="2023-07-31T09:54:41.874119953-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:54:41 servername containerd: time="2023-07-31T09:54:41.874207169-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:54:57 servername dockerd: time="2023-07-31T09:54:57.584823060-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:54:59 servername containerd: time="2023-07-31T09:54:59.588919189-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:54:59 servername containerd: time="2023-07-31T09:54:59.589005700-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:55:39 servername dockerd: time="2023-07-31T09:55:39.873576503-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:55:41 servername containerd: time="2023-07-31T09:55:41.879982578-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:55:41 servername containerd: time="2023-07-31T09:55:41.880108042-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:55:58 servername dockerd: time="2023-07-31T09:55:58.088163254-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:56:00 servername containerd: time="2023-07-31T09:56:00.092247357-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:56:00 servername containerd: time="2023-07-31T09:56:00.092167125-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:56:03 servername dockerd: time="2023-07-31T09:56:03.093622845-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:56:05 servername containerd: time="2023-07-31T09:56:05.099829837-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:56:05 servername containerd: time="2023-07-31T09:56:05.099953869-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:56:21 servername dockerd: time="2023-07-31T09:56:21.006549836-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:56:23 servername containerd: time="2023-07-31T09:56:23.010842116-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:56:23 servername containerd: time="2023-07-31T09:56:23.010924996-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:56:32 servername dockerd: time="2023-07-31T09:56:32.200693614-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:57:28 servername dockerd: time="2023-07-31T09:57:28.590350664-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:57:36 servername dockerd: time="2023-07-31T09:57:36.680934357-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
2

@Naresh9999

I had seen that same thing… in the OS level /var/log/messages do you see this as well?

I also then moved my loki service to another server and the amount of times I saw the errors largely decreased… YET The same issue continued. The only items I noticed was I WAS installing the agent on the host OS and I removed that and let the installed docker agent record the records…

(I had posted this earlier but did not know how to work with docker at all when I did)

I started off trying to figure out what was stdout inside a docker image and made sure was fully up to date on all software versions…

Jul 31 09:56:05 servername containerd: time="2023-07-31T09:56:05.099953869-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:56:21 servername dockerd: time="2023-07-31T09:56:21.006549836-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:56:23 servername containerd: time="2023-07-31T09:56:23.010842116-04:00" level=warning msg="error copying stdout" runtime=io.containerd.runc.v2
Jul 31 09:56:23 servername containerd: time="2023-07-31T09:56:23.010924996-04:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2
Jul 31 09:56:32 servername dockerd: time="2023-07-31T09:56:32.200693614-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:57:28 servername dockerd: time="2023-07-31T09:57:28.590350664-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
Jul 31 09:57:36 servername dockerd: time="2023-07-31T09:57:36.680934357-04:00" level=warning msg="Health check for container 60b37029dc405e1bb86463602c9baf00e1c27e7fbb198d1bc37e5622bb7adefe error: context deadline exceeded"
3

@meyerder Yes, I can see the above errors in the /var/log/messages log file in the PMM server.

Can you please check if any errors in the Grafana log file?