GR cluster member, in-place upgrade from 8.0.29 to 8.0.30 failed

MySQL & MariaDB Percona Server for MySQL 8.0
1

Dear All,

I have:

  • a 3-nodes GR cluster
  • running on Debian 11 VMs
  • with ZFS for the data and binary logs directories (/var/lib/mysql/data and /var/lib/mysql/log).
$ uname -r
5.10.0-19-cloud-amd64
$ sudo zfs version
zfs-2.1.5-1~bpo11+1
zfs-kmod-2.1.5-1~bpo11+1

The problem

Today I detached one of the secondary nodes from the GR and I tried to upgrade in-place Percona Server from 8.0.29 to 8.0.30. The upgrade process failed and MySQL does not start anymore:

$ sudo apt upgrade
[...]
Setting up percona-server-server (8.0.30-22-1.bullseye) ...

[...]
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mysql, action "restart" failed.
â—Ź mysql.service - Percona Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mysql.service.d
             └─override.conf
     Active: failed (Result: exit-code) since Mon 2022-12-05 05:01:58 UTC; 4ms ago
    Process: 8648 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Process: 8694 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
   Main PID: 8694 (code=exited, status=1/FAILURE)
     Status: "Server shutdown complete"
        CPU: 5.403s

Dec 05 05:01:56 mysql-cluster-red-1 mysqld[8694]: InnoDB initialization has started.
Dec 05 05:01:57 mysql-cluster-red-1 mysqld[8694]: Cannot create redo log files because data files are corrupt or the database was not shut down cleanly after creating the data files.
Dec 05 05:01:58 mysql-cluster-red-1 mysqld[8694]: Failed to initialize DD Storage Engine
Dec 05 05:01:58 mysql-cluster-red-1 mysqld[8694]: Data Dictionary initialization failed.
Dec 05 05:01:58 mysql-cluster-red-1 mysqld[8694]: Aborting
Dec 05 05:01:58 mysql-cluster-red-1 mysqld[8694]: /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-22)  Percona Server (GPL), Release '22', Revision '7e301439b65'.
[...]
dpkg: error processing package percona-server-server (--configure):
 installed percona-server-server package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.31-13+deb11u5) ...
Errors were encountered while processing:
 percona-server-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

Config file excerpt

Options that differ from default and/or can be relevant:

datadir                        = /var/lib/mysql/data
innodb_log_group_home_dir      = /var/lib/mysql/log
log_bin                        = /var/lib/mysql/log/mysql-binlog
log_bin_index                  = mysql-bin.index
binlog_checksum                = CRC32
binlog_transaction_dependency_tracking = WRITESET_SESSION
relay_log                      = /var/lib/mysql/log/relay
innodb_log_file_size           = 1G
innodb_log_buffer_size         = 32M
innodb_log_write_ahead_size    = 16384 

innodb_flush_log_at_trx_commit = 1
innodb_file_per_table          = 1
innodb_checksum_algorithm      = NONE
innodb_flush_neighbors         = 0
innodb_fast_shutdown           = 2
innodb_flush_method            = fsync  # ZFS does not support O_DIRECT
innodb_use_fdatasync           = ON
innodb_doublewrite             = 0      # ZFS is transactional

plugin_load_add                         = 'group_replication.so'
plugin_load_add                         = 'mysql_clone.so'
group_replication_group_name            = '4f1ab3dc-d037-4dc5-a5f5-d66eee3fc61a'
group_replication_start_on_boot         = OFF
group_replication_bootstrap_group       = OFF
group_replication_consistency           = AFTER
group_replication_compression_threshold = 0
group_replication_single_primary_mode   = ON
group_replication_enforce_update_everywhere_checks = OFF

Did anyone observe the same issue?

I would be grateful for any advice and I will gladly provide more details if needed.

Thank you,
Corrado

1 Like
2

Partially answering myself, this problem could be a combination of a different REDO log format in 8.0.30 and my bad idea of setting innodb_fast_shutdown to 2 instead of leaving the default (safer) value of 1.

It appears (https://ubuntuforums.org/showthread.php?t=2477581) that setting it to 2 disables the data upgrade process. Quoting from the linked post (slightly reformatted for readability):

[...] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2.
This redo log was created with MySQL 8.0.29, and it appears logically non empty.
Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2022-07-30T11:17:42.902278Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-07-30T11:17:43.037545Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-07-30T11:17:43.037961Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
1 Like