Hi,
My bad for not being clear. The server we are working on is our lab MySql server. Any load was generated by me trying to produce something for the analyzer to see.
What I have done this morning as suggested is:
-bash-4.1$ date
Wed Jan 17 08:14:00 EST 2018
-bash-4.1$ sudo pmm-admin stop --all
OK, stopped 3 services.
-bash-4.1$ date
Wed Jan 17 08:14:34 EST 2018
-bash-4.1$ sudo pmm-admin start --all
OK, started 3 services.
-bash-4.1$ tail -n 20 pmm-mysql-queries-0.log
2018/01/16 08:05:57.792828 WARNING data-sender Cannot connect to API: bad status
2018/01/16 08:27:01.132122 WARNING qan-analyzer-mysql-b476a926-worker Interval out of sequence: got 1, expected 1251
2018/01/16 14:45:01.132163 WARNING qan-analyzer-mysql-b476a926-worker Interval out of sequence: got 1, expected 379
2018/01/16 15:53:35.011098 WARNING agent ws error: EOF
2018/01/16 15:53:35.012706 WARNING agent Lost connection to API, reconnecting
2018/01/16 15:53:35.605911 WARNING agent-ws bad status
2018/01/16 15:53:35.606035 WARNING log Lost connection to API
2018/01/16 15:53:35.608337 WARNING log-ws bad status
2018/01/17 08:14:29.780422 main.go:359: Caught terminated signal, shutting down
2018/01/17 08:14:29.781400 main.go:385: Stopping QAN…
2018/01/17 08:14:29.793910 main.go:392: Waiting 2 seconds to flush agent log to API…
2018/01/17 08:14:31.794027 main.go:167: Agent has stopped
Version: percona-qan-agent 1.5.3
Basedir: /usr/local/percona/qan-agent
PID: 21428
API: 192.168.145.100/qan-api
UUID: 9ae8c065d53742f26ef7d654dd241c59
2018/01/17 08:14:45.414812 main.go:163: Starting agent…
2018/01/17 08:14:45.428166 main.go:331: Agent is ready
2018/01/17 08:14:45.506450 main.go:204: API is ready
That is all that is in the file at this point. I will check again in an hour or so to see if there is any other entries.
In the afternoon I can try to add a different server to validate if that is a factor.
My Telecomm admin assures me that currently I am setup TCP ports any to any between the PMM server and the agent server in both directions.