After deciding to try something different, migrated a server (Ubuntu 20.04) from MySQL 8 (Ubuntu repo) to Percona 8 by dumping all databases to .sql files, removing MySQL, installing Percona, created databases and imported .sql files.
I am seeing significant (2 digit MB) Read and Write IO from the mysqld process (running under the mysql user) according to htop, and using iotop to confirm the process and user.
To rule out client access (databases are used by two external clients), the mysql user passwords were changed for both accounts and mysql was restarted, however it has not had a noticeable effect on the IO.
Where might I look to see what is causing the IO?
iotop to see what files are being accessed. I would also recommend setting up a PMM server to collect more specific information about your MySQL server and all IO/memory metrics.
After the DBs has just been restarted, the buffer pool might be cold and a lot of disk read activity might be required to satisfy the read requests.
Do you also see high disk read activity if there is no DB activity ?
To monitor disk read activity from MySQL you can check MySQL :: MySQL 5.7 Reference Manual :: 5.1.9 Server Status Variables with "SHOW GLOBAL STATUS LIKE ‘%buffer_pool_reads%’; every a few seconds
You can also check “free buffers” from “SHOW ENGINE INNODB STATUS \G” and check if the buffer pool is mostly empty and slowly being warmed.
From outside MySQL you can use other OS tools like iotop, BCC tools or even inotifywatch to check which files are being accessed.
innotop showed no open tables, but eventually (~2 hours) it calmed down.
iotop shows the process, user, and read and write usage, but not the open files.
I should have just done a watch n 1 lsof /var/lib/mysql, but it had been a long day.
Much as it would be nice, I don’t the capacity (or budget) for a PMM server (-:
iotop -oa should show you all ongoing disk activity. You probably need to
sudo it. It works perfectly on my Ubuntu 18 LTS server for finding out what program is causing disk IO.
iotop shows disk activity, but does not show files (or filesystems) that are being read from or written to.