Replication IO Thread does not start

Hello,

I have a problem with mit replication setup:

my slave IO Thread always stops with an checksum verification failed error:

2025-08-04T13:26:27.959431+02:00 540 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2025-08-04T13:26:27.963192+02:00 540 [System] [MY-014001] [Repl] Replica receiver thread for channel '': connected to source 'replication@192.168.131.52:3306' with server_uuid=86cc02aa-326d-11f0-baae-8c84742a1c11, server_id=202505021. Starting replication from file 'dui2-sr2-mysql-db01.006101', position '168597998'.
2025-08-04T13:26:42.621438+02:00 540 [ERROR] [MY-013115] [Repl] Replica I/O for channel '': Replication event checksum verification failed while reading from network. Error_code: MY-013115
2025-08-04T13:26:42.621480+02:00 540 [ERROR] [MY-013122] [Repl] Replica I/O for channel '': Relay log write failure: could not queue event from source, Error_code: MY-013122
2025-08-04T13:26:42.621486+02:00 540 [Note] [MY-010570] [Repl] Replica I/O thread exiting for channel '', read up to log 'dui2-sr2-mysql-db01.006101', position 168597998
mysql> show replica status\G
*************************** 1. row ***************************
             Replica_IO_State: 
                  Source_Host: 192.168.131.52
                  Source_User: replication
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: dui2-sr2-mysql-db01.006101
          Read_Source_Log_Pos: 168597998
               Relay_Log_File: dui3-sr3-mysql-db02-relay.000006
                Relay_Log_Pos: 336
        Relay_Source_Log_File: dui2-sr2-mysql-db01.006101
           Replica_IO_Running: No
          Replica_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: xxx.tmp_xxx
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Source_Log_Pos: 168597998
              Relay_Log_Space: 735
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Source_SSL_Allowed: No
           Source_SSL_CA_File: 
           Source_SSL_CA_Path: 
              Source_SSL_Cert: 
            Source_SSL_Cipher: 
               Source_SSL_Key: 
        Seconds_Behind_Source: NULL
Source_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 13122
                Last_IO_Error: Relay log write failure: could not queue event from source
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Source_Server_Id: 202505021
                  Source_UUID: 86cc02aa-326d-11f0-baae-8c84742a1c11
             Source_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                  Source_Bind: 
      Last_IO_Error_Timestamp: 250804 13:26:42
     Last_SQL_Error_Timestamp: 
               Source_SSL_Crl: 
           Source_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 86cc02aa-326d-11f0-baae-8c84742a1c11:1-206831697
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Source_TLS_Version: 
       Source_public_key_path: 
        Get_Source_public_key: 0
            Network_Namespace: 
1 row in set (0.00 sec)

root@dui3-sr3-mysql-db02:/home/pg# ls -la /var/mysql_binlog/relaylog/
total 76
drwxr-xr-x 2 mysql mysql 28672 Aug  4 13:26 .
drwxr-xr-x 4 mysql mysql 36864 Jul 28 14:00 ..
-rw-r----- 1 mysql mysql   399 Aug  4 13:26 dui3-sr3-mysql-db02-relay.000005
-rw-r----- 1 mysql mysql   336 Aug  4 13:26 dui3-sr3-mysql-db02-relay.000006
-rw-r----- 1 mysql mysql   120 Aug  4 13:26 dui3-sr3-mysql-db02-relay.index

I tried to turn off checksum verification on the slave, but it did not made any difference

mysql> set global replica_sql_verify_checksum=OFF;

i checked the binlog on the master with mysqlbinlog, and it was readable

Does anybody has an idea whats wrong with the replication?

Kind regards
Phillip

Hi @Phillip85 ,

Welcome to Percona Community.

Without going into details I’d just look at this one error:

Last_IO_Error: Relay log write failure: could not queue event from source

It tells me that you cannot queue event and relay logs are not being written to! I’d go and check disk.

Is you disk full? If not,

Is this something that was working earlier -or- is this a new setup?

Thanks,

K

disk is not full, permissions are ok too:

root@dui3-sr3-mysql-db02:/home/pg# ls -ltr /var/mysql_binlog/relaylog/
total 12
-rw-r----- 1 mysql mysql 120 Aug  4 13:58 dui3-sr3-mysql-db02-relay.index
-rw-r----- 1 mysql mysql 336 Aug  4 13:58 dui3-sr3-mysql-db02-relay.000009
-rw-r----- 1 mysql mysql 399 Aug  4 13:58 dui3-sr3-mysql-db02-relay.000008

the setup worked earlier, but I had to reset the replica and start it again with

reset replica;
change master to master_host='192.168.131.52', master_user='replication', master_password='xxx', master_log_file='dui2-sr2-mysql-db01.006101', master_log_pos=168597998;
start replica;

ok… now it becomes very strange…

I read some old mysql forum threads where they talked about problems with binlog files > 4G
the 6101 file was bigger then 4G, so for testing if the io thread is working with smaller files, I changed the master source file to 6100 which was just 1G.

reset replica;
change master to master_host='192.168.131.52', master_user='replication', master_password='xxx', master_log_file='dui2-sr2-mysql-db01.006100', master_log_pos=126;
start replica;

now the replica is working, there are not even duplicate key errors on the slave…

************************** 1. row ***************************
             Replica_IO_State: Queueing source event to the relay log
                  Source_Host: 192.168.131.52
                  Source_User: replication
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: dui2-sr2-mysql-db01.006121
          Read_Source_Log_Pos: 645378040
               Relay_Log_File: dui3-sr3-mysql-db02-relay.000008
                Relay_Log_Pos: 883735245
        Relay_Source_Log_File: dui2-sr2-mysql-db01.006102
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: xxx.tmp_xxx
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Source_Log_Pos: 883735009
              Relay_Log_Space: 29432564873
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Source_SSL_Allowed: No
           Source_SSL_CA_File: 
           Source_SSL_CA_Path: 
              Source_SSL_Cert: 
            Source_SSL_Cipher: 
               Source_SSL_Key: 
        Seconds_Behind_Source: 178752
Source_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Source_Server_Id: 202505021
                  Source_UUID: 86cc02aa-326d-11f0-baae-8c84742a1c11
             Source_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
    Replica_SQL_Running_State: Waiting for dependent transaction to commit
           Source_Retry_Count: 86400
                  Source_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Source_SSL_Crl: 
           Source_SSL_Crlpath: 
           Retrieved_Gtid_Set: 86cc02aa-326d-11f0-baae-8c84742a1c11:206117745-218265419
            Executed_Gtid_Set: 86cc02aa-326d-11f0-baae-8c84742a1c11:1-207606142
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Source_TLS_Version: 
       Source_public_key_path: 
        Get_Source_public_key: 0
            Network_Namespace: 

I dont really get it

Yes, it is correct. More details can be found inside this bug https://bugs.mysql.com/bug.php?id=55231

yeah that was the forum thread i found too.
But what i don’t understand is, why does the 4GB file works when I have a smaller file first,
i just changed the master_log_file to one file before, and then both files got processed.

now the slave is working normal again for the last 3 days