Hello, first, allow me to outline my current service setup. I have created the following containers on the same server using Docker: pmm-server, pmm-client, MySQL (8.0.35), and minio-server. After setting up these services, I resolved several issues. I added my S3 storage location for MySQL backups via the Storage Locations section in the web interface. Subsequent backups of MySQL succeeded, but I encountered problems during restoration. Since my pmm and MySQL containers reside in separate containers, I mounted the MySQL Docker data volume as read-only to pmm-client:deep_db_db-data:/var/lib/pmm_mysql_backup:ro. I avoided mounting directly to /var/lib/mysql because clicking restore triggered client logs indicating the path was in use. So I changed the path and created a symbolic link between /var/lib/pmm_mysql and /var/lib/pmm_mysql_backup. When I clicked restore, the /var/lib/mysql directory within the pmm client was renamed, then a new /var/lib/mysql directory was generated. I wonder if, when deploying the PMM client via containers for recovery, the operation only restores /var/lib/mysql within the client container itself. It doesn’t synchronize the restored files to the MySQL data volume. This likely explains why my data wasn’t successfully recovered after restarting the database. I’d like to know if the current PMM supports full backup and restore workflows for MySQL. If so, I haven’t found documentation explaining this capability. If container isolation is mandatory for backup/restore, how should I address this issue?
Welcome to Percona Forum @hey,
Your diagnosis is correct. When PMM restores a MySQL backup, it writes the data to /var/lib/mysql inside the pmm-client container, not to your MySQL container’s data volume. This is a known limitation of the PMM backup/restore architecture with multi-container Docker setups.
For context, here’s why. The restore agent (mysql_restore_job.go) hardcodes three assumptions: (1) the MySQL data directory is /var/lib/mysql on the same filesystem as pmm-agent, (2) MySQL can be stopped/started via systemctl, and (3) the mysql system user exists locally. All three fail when pmm-agent and MySQL run in separate containers. Backup works because it connects to MySQL over the network; restore assumes local filesystem access.
This is documented in the MySQL backup prerequisites, which require MySQL to run as a systemd service with pmm-agent on the same host. MySQL backup remains in “Technical Preview” status in PMM 3.x. Several related JIRA tickets track improvements: PMM-7797 (customizable pre/post restore commands, open since 2021), PMM-8128 (prerequisites check that would warn Docker users upfront), and PMM-10984 (Docker restore hangs).
Workaround: bypass PMM’s restore and use XtraBackup directly. I tested this with a setup similar to yours (MySQL 8.0.35 + MinIO + XtraBackup 8.0.35-31, Docker volumes). Run a percona/percona-xtrabackup:8.0.35-31 container that shares your MySQL data volume. You need --user 0:0 because the XtraBackup image runs as UID 1001 by default, which cannot read MySQL’s data files (owned by UID 999).
# Stop MySQL first, then run the restore container as root
# docker stop your-mysql-container
docker run --rm -it --user 0:0 \
--network your_network \
-v your_mysql_data_volume:/var/lib/mysql \
percona/percona-xtrabackup:8.0.35-31 bash
# Inside the container:
# 1. Download and extract the backup from MinIO
mkdir -p /tmp/restore
xbcloud get --storage=s3 \
--s3-endpoint=http://minio:9000 \
--s3-access-key=YOUR_KEY \
--s3-secret-key=YOUR_SECRET \
--s3-bucket=YOUR_BUCKET \
"backup-folder/backup-name" | xbstream -x -C /tmp/restore
# 2. Decompress and prepare
xtrabackup --decompress --target-dir=/tmp/restore
xtrabackup --prepare --target-dir=/tmp/restore
# 3. Clear the datadir and restore
# IMPORTANT: Back up your volume first (docker volume create + cp, or snapshot).
# "mv /var/lib/mysql" does NOT work with Docker volumes; you must clear contents.
MYSQL_UID=$(stat -c %u /var/lib/mysql/ibdata1)
MYSQL_GID=$(stat -c %g /var/lib/mysql/ibdata1)
rm -rf /var/lib/mysql/* /var/lib/mysql/.* 2>/dev/null
xtrabackup --copy-back --datadir=/var/lib/mysql --target-dir=/tmp/restore
chown -R $MYSQL_UID:$MYSQL_GID /var/lib/mysql
# 4. Start MySQL: docker start your-mysql-container
Two important notes from testing. First, you must clear the datadir completely (including hidden files) before --copy-back, because Docker volume mount points cannot be renamed and xtrabackup --copy-back refuses to write into a non-empty directory (“Original data directory is not empty!”). Back up your volume before this step. Second, the UID/GID of the mysql user differs between containers, so capture the original UID from existing files before clearing, then chown after the copy-back.
Also worth noting: MySQL 8.0.35 reaches end-of-life in April 2026. Consider upgrading to 8.0.41 or 8.4 LTS if you plan to maintain this setup.
References:
- Docs: MySQL backup prerequisites
- Source: mysql_restore_job.go
- PMM-10984 (Docker restore hangs)
- PMM-7797 (customizable pre/post restore commands)
- PMM-8128 (prerequisites check for Docker users)