Issue is resolved. Removed all client services (pmm-admin rm --all), removed pmm-server and pmm-data, then added pmm-server with -e METRICS_RESOLUTION=5s -e METRICS_MEMORY=786432 options, then added back all clients same as before. Adding those two options seems to have done the trick. Load and disk io on Prometheus server is fine and steady, and no more storage rushed mode. Still seeing “sample timestamp out of order” and “sample discarded” messages though.
