MySQL 8.0.35 to MySQL 8.0.36 - Can't restore backup on slave (replication)

After upgrading from MySQL 8.0.35 to MySQL 8.0.36 it’s no longer possible to restore the slave (replication) from a previous backup from XtraBackup

Slave error

Replica failed to initialize applier metadata structure from the repository

Before the upgrade the following script worked

echo "Unpacking backup file: $bak"
xbstream -x -C "$mysql_restore_dir" < "$bak"
xtrabackup --decompress --remove-original --target-dir="$mysql_restore_dir" > "$db_import_log" 2>&1
xtrabackup_error "$db_import_log"

line=$(head -n 1 "$mysql_restore_dir/xtrabackup_binlog_info")
logfile=$(echo $line | awk -F '[ ,]' '{print $1}')
logpos=$(echo $line | awk -F '[ ,]' '{print $2}')
gtid="$(echo $line | awk -F '[ ]' '{print $3}')"

echo "\nLog file: $logfile\nLog pos: $logpos\nGTID: $gtid"

echo "Preparing: $mysql_restore_dir"
xtrabackup --prepare --target-dir="$mysql_restore_dir" > "$db_import_log" 2>&1
xtrabackup_error "$db_import_log"

xtrabackup --move-back --target-dir="$mysql_restore_dir" > "$db_import_log" 2>&1
xtrabackup_error "$db_import_log"

rm -r $mysql_restore_dir/*
chown -R mysql:mysql $mysql_dir

service mysql start

mysql -u root -p$mysql_pass -e "RESET MASTER;\
	SET GLOBAL gtid_purged='$gtid';\
	CHANGE MASTER TO MASTER_HOST='$master_host', MASTER_USER='repl', MASTER_PASSWORD='$mysql_pass', MASTER_AUTO_POSITION=1, GET_MASTER_PUBLIC_KEY=1;\
	START SLAVE"

The above script would print something like

Log file: mysql-bin.000002
Log pos: 197
GTID: 2d74e213-978b-11ee-895a-4c52620e063f:1-4322940,3ccc2392-45ed-11e7-bc98-4061862b8d34:1-35942477,4ea12ff1-ed48-11ee-b747-4c52620e063f:1-141,fe51e8df-b7c4-11e9-be21-4061862b8d34:1-57690654

And the last line in the import log xtrabackup.log

2024-03-28T23:06:28.231560+01:00 0 [Note] [MY-011825] [Xtrabackup] completed OK!

When checking the slave status

               Slave_IO_State:
                  Master_Host: xx.xx.xx.xx
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File:
          Read_Master_Log_Pos: 4
               Relay_Log_File: rep-dynaccount-relay-bin.000001
                Relay_Log_Pos: 4
        Relay_Master_Log_File:
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB: dynaccount
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 13124
                   Last_Error: Replica failed to initialize applier metadata structure from the repository
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 0
              Relay_Log_Space: 157
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 13124
               Last_SQL_Error: Replica failed to initialize applier metadata structure from the repository
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
                  Master_UUID:
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp: 240328 23:06:32
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set: 2d74e213-978b-11ee-895a-4c52620e063f:1-4322940,
3ccc2392-45ed-11e7-bc98-4061862b8d34:1-35942477,
4ea12ff1-ed48-11ee-b747-4c52620e063f:1-141,
fe51e8df-b7c4-11e9-be21-4061862b8d34:1-57690654
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
       Master_public_key_path:
        Get_master_public_key: 1
            Network_Namespace:

What’s in MySQL’s error log? I would also have done a RESET REPLICA ALL before doing the CHANGE REPLICATION SOURCE

Hi Matthewb

Thanks for a quick reply during easter :slight_smile:

Have now updated from CHANGE MASTER TO ... to CHANGE REPLICATION SOURCE TO ... and added RESET REPLICA ALL

mysql -u root -p$mysql_pass -e "RESET MASTER;\
	RESET REPLICA ALL;\
	SET GLOBAL gtid_purged='$gtid';\
	CHANGE REPLICATION SOURCE TO SOURCE_HOST='$master_host', SOURCE_USER='repl', SOURCE_PASSWORD='$mysql_pass', SOURCE_AUTO_POSITION=1, GET_MASTER_PUBLIC_KEY=1;\
	START REPLICA;"

error log

2024-03-29T07:28:19.108161Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.36) starting as process 3776714
2024-03-29T07:28:19.130687Z 0 [Warning] [MY-010075] [Server] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: ea47e68c-ed9d-11ee-8aa7-96000080ccbb.
2024-03-29T07:28:19.143744Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-03-29T07:28:19.590654Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-03-29T07:28:19.708478Z 1 [Warning] [MY-012351] [InnoDB] Tablespace 1, name 'sys/sys_config', file './sys/sys_config.ibd' is missing!
2024-03-29T07:28:19.767872Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2024-03-29T07:28:19.793339Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2024-03-29T07:28:19.858334Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-03-29T07:28:19.858380Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-03-29T07:28:19.882228Z 0 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a replica and has his hostname changed!! Please use '--relay-log=rep-dynaccount-relay-bin' to avoid this problem.
2024-03-29T07:28:19.888121Z 0 [ERROR] [MY-010544] [Repl] Failed to open the relay log './Application-Intel-Xeon-64gb-relay-bin.000001' (relay_log_pos 4).
2024-03-29T07:28:19.888152Z 0 [ERROR] [MY-011059] [Repl] Could not find target log file mentioned in applier metadata in the index file './rep-dynaccount-relay-bin.index' during relay log initialization.
2024-03-29T07:28:19.889302Z 0 [ERROR] [MY-010426] [Repl] Replica: Failed to initialize the connection metadata structure for channel ''; its record may still be present in the applier metadata repository, consider deleting it.
2024-03-29T07:28:19.889369Z 0 [ERROR] [MY-010529] [Repl] Failed to create or recover replication info repositories.
2024-03-29T07:28:19.895133Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.36'  socket: '/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2024-03-29T07:28:19.973894Z 8 [System] [MY-010916] [Server] @@GLOBAL.GTID_PURGED was changed from '' to '2d74e213-978b-11ee-895a-4c52620e063f:1-4322940, 3ccc2392-45ed-11e7-bc98-4061862b8d34:1-35942477, 4ea12ff1-ed48-11ee-b747-4c52620e063f:1-4489, fe51e8df-b7c4-11e9-be21-4061862b8d34:1-57690654'.
2024-03-29T07:28:19.973953Z 8 [System] [MY-010917] [Server] @@GLOBAL.GTID_EXECUTED was changed from '' to '2d74e213-978b-11ee-895a-4c52620e063f:1-4322940, 3ccc2392-45ed-11e7-bc98-4061862b8d34:1-35942477, 4ea12ff1-ed48-11ee-b747-4c52620e063f:1-4489, fe51e8df-b7c4-11e9-be21-4061862b8d34:1-57690654'.
2024-03-29T07:28:19.980497Z 8 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL '' executed'. Previous state source_host='', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='195.201.241.223', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2024-03-29T07:28:19.990997Z 9 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-03-29T07:28:20.005605Z 9 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'repl@195.201.241.223:3306' with server_uuid=4ea12ff1-ed48-11ee-b747-4c52620e063f, server_id=1. Starting GTID-based replication.

master my.cnf

[mysqld]

# GENERAL #
user                           = mysql
port                           = 3306
pid-file                       = /run/mysqld/mysqld.pid
socket                         = /run/mysqld/mysqld.sock

# SAFETY #
max_allowed_packet             = 16M
skip_name_resolve
sql_mode                       = STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION

# DATA STORAGE #
datadir                        = /var/lib/mysql/

# BINARY LOGGING #
server_id                      = 1
log_bin                        = /var/lib/mysql/mysql-bin
binlog_expire_logs_seconds     = 864000
sync_binlog                    = 1
log_bin_trust_function_creators
source_verify_checksum

# REPLICATION #
gtid_mode                      = ON
enforce_gtid_consistency

# CACHES AND LIMITS #
tmp_table_size                 = 64M
max_heap_table_size            = 64M
max_connections                = 500
thread_cache_size              = 50
table_definition_cache         = 4096
table_open_cache               = 4096
#query_cache_type               = 0
#query_cache_size               = 0

# INNODB #
innodb_redo_log_capacity       = 2G
innodb_log_buffer_size         = 64M
innodb_file_per_table          = 1
innodb_buffer_pool_size        = 48G
innodb_buffer_pool_instances   = 12
innodb_read_io_threads         = 12
innodb_write_io_threads        = 12
innodb_stats_on_metadata       = 0

# LOGGING #
log_error                      = /var/lib/mysql/mysql-error.log
log_queries_not_using_indexes
slow_query_log
slow_query_log_file            = /var/lib/mysql/mysql-slow.log
long_query_time                = 1


performance-schema = 0
local-infile = 0
mysqlx = 0

connect_timeout = 5
interactive_timeout = 30

read_buffer_size = 256K
join_buffer_size = 512K
sort_buffer_size = 512K
read_rnd_buffer_size = 512K

replica my.cnf

[mysqld]

# GENERAL #
user                           = mysql
port                           = 3306
pid-file                       = /run/mysqld/mysqld.pid
socket                         = /run/mysqld/mysqld.sock

# SAFETY #
max_allowed_packet             = 16M
skip_name_resolve
sql_mode                       = STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION

# DATA STORAGE #
datadir                        = /var/lib/mysql/

# BINARY LOGGING #
server_id                      = 2
log_bin                        = /var/lib/mysql/mysql-bin
log_replica_updates
binlog_expire_logs_seconds     = 864000
sync_binlog                    = 1
log_bin_trust_function_creators
replica_sql_verify_checksum

# REPLICATION #
gtid_mode                      = ON
enforce_gtid_consistency
replicate_do_db                = my_replicated_db
#super_read_only                = 1

# CACHES AND LIMITS #
tmp_table_size                 = 64M
max_heap_table_size            = 64M
max_connections                = 500
thread_cache_size              = 50
table_definition_cache         = 4096
table_open_cache               = 4096

# INNODB #
innodb_redo_log_capacity       = 512M
innodb_log_buffer_size         = 64M
innodb_file_per_table          = 1
innodb_buffer_pool_size        = 512M
innodb_buffer_pool_instances   = 1
innodb_read_io_threads         = 12
innodb_write_io_threads        = 12
innodb_stats_on_metadata       = 0

# LOGGING #
log_error                      = /var/lib/mysql/mysql-error.log
log_queries_not_using_indexes
slow_query_log
slow_query_log_file            = /var/lib/mysql/mysql-slow.log
long_query_time                = 1


performance-schema = 0
local-infile = 0
mysqlx = 0

connect_timeout = 5
interactive_timeout = 30

read_buffer_size = 256K
join_buffer_size = 512K
sort_buffer_size = 512K
read_rnd_buffer_size = 512K

It may seem like this line in the error log breaks everything

[ERROR] [MY-010544] [Repl] Failed to open the relay log './Application-Intel-Xeon-64gb-relay-bin.000001' (relay_log_pos 4)

It says Please use '--relay-log=rep-dynaccount-relay-bin' to avoid this problem.
But how to I set that?

Or even better. Where do I override the hostname in mysql to rep-dynaccount which is what is defiend in /etc/hostname ?

Application-Intel-Xeon-64gb is the hostname from the source/master

In the slave status it already says

Relay_Log_File: rep-dynaccount-relay-bin.000001

So it’s a bit confusing the slave knows to look for rep-dynaccount-relay-bin but mysql fails to actually look for the file in the mysql error log

Ok, I got it to work after adding relay-log=rep-dynaccount-relay-bin to the recplica my.cnf

But this feels a bit like a bug in mysql?

The host defined in /etc/hostname is rep-dynaccount and the relay files are also called that on the replica in /var/lib/mysql

But when mysql is starting it is looking the the hostname from the source/master?

But the output from show slave status shows the correct relay file with rep-dynaccount ?!