Seconds_Behind_Master keeps increasing

Hi,

I’m having an issue where “seconds_behind_master” keeps increasing on my replica server. I can’t figure out why so any ideas would be helpful.

Both servers are running:

MASTER
======
2024-04-02T21:07:38.379338-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --log_bin=mysql-bin --server-id=1
xtrabackup version 8.0.35-30 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 6beb4b49)
mysql Ver 8.0.33-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))

REPLICA
=======
2024-04-02T21:06:45.939781-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --log_bin=mysql-bin --server-id=2
xtrabackup version 8.0.35-30 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 6beb4b49)
mysql Ver 8.0.36-0ubuntu0.22.04.1 for Linux on x86_64 ((Ubuntu))

Both servers are in the same ISP so I don’t think it’s a networking or resource issue.

The replica server has:
replica_parallel_type = LOGICAL_CLOCK
replica_preserve_commit_order = ON
replica_parallel_workers = 4
binlog_format = ROW

22:42pm ! 02/04/2024

REPLICA STATUS
==============
Slave_IO_State Waiting for source to send event
Master_Host //MASTER IP ADDRESS
Master_User repl
Master_Port 3306
Connect_Retry 60
Master_Log_File mysql-bin.000291
Read_Master_Log_Pos 95339364
Relay_Log_File 724805F-relay-bin.000372
Relay_Log_Pos 62343421
Relay_Master_Log_File mysql-bin.000194
Slave_IO_Running Yes
Slave_SQL_Running Yes
Replicate_Do_DB
Replicate_Ignore_DB
Replicate_Do_Table
Replicate_Ignore_Table
Replicate_Wild_Do_Table
Replicate_Wild_Ignore_Table
Last_Errno 0
Last_Error
Skip_Counter 0
Exec_Master_Log_Pos 62343205
Relay_Log_Space 10209233453
Until_Condition None
Until_Log_File
Until_Log_Pos 0
Master_SSL_Allowed No
Master_SSL_CA_File
Master_SSL_CA_Path
Master_SSL_Cert
Master_SSL_Cipher
Master_SSL_Key
Seconds_Behind_Master 176327

22:52pm ! 02/04/2024

REPLICA STATUS
==============
Slave_IO_State Waiting for source to send event
Master_Host //MASTER IP ADDRESS
Master_User repl
Master_Port 3306
Connect_Retry 60
Master_Log_File mysql-bin.000292
Read_Master_Log_Pos 16192157
Relay_Log_File 724805F-relay-bin.000372
Relay_Log_Pos 80616437
Relay_Master_Log_File mysql-bin.000194
Slave_IO_Running Yes
Slave_SQL_Running Yes
Replicate_Do_DB
Replicate_Ignore_DB
Replicate_Do_Table
Replicate_Ignore_Table
Replicate_Wild_Do_Table
Replicate_Wild_Ignore_Table
Last_Errno 0
Last_Error
Skip_Counter 0
Exec_Master_Log_Pos 80616221
Relay_Log_Space 10234954547
Until_Condition None
Until_Log_File
Until_Log_Pos 0
Master_SSL_Allowed No
Master_SSL_CA_File
Master_SSL_CA_Path
Master_SSL_Cert
Master_SSL_Cipher
Master_SSL_Key
Seconds_Behind_Master 176613

16:00pm ! 03/04/2024

REPLICA STATUS
==============
Slave_IO_State Waiting for source to send event
Master_Host //MASTER IP ADDRESS
Master_User repl
Master_Port 3306
Connect_Retry 60
Master_Log_File mysql-bin.000321
Read_Master_Log_Pos 70852457
Relay_Log_File 724805F-relay-bin.000408
Relay_Log_Pos 25340929
Relay_Master_Log_File mysql-bin.000212
Slave_IO_Running Yes
Slave_SQL_Running Yes
Replicate_Do_DB
Replicate_Ignore_DB
Replicate_Do_Table
Replicate_Ignore_Table
Replicate_Wild_Do_Table
Replicate_Wild_Ignore_Table
Last_Errno 0
Last_Error
Skip_Counter 0
Exec_Master_Log_Pos 25340713
Relay_Log_Space 11390823716
Until_Condition None
Until_Log_File
Until_Log_Pos 0
Master_SSL_Allowed No
Master_SSL_CA_File
Master_SSL_CA_Path
Master_SSL_Cert
Master_SSL_Cipher
Master_SSL_Key
Seconds_Behind_Master 195472

22:52pm ! 02/04/2024

MASTER STATUS

Variable Value
File mysql-bin.000292
Position 14494109
Binlog_Do_DB
Binlog_Ignore_DB

Hello @CL69,
On the replica, try setting the following:

innodb_flush_log_at_trx_commit=0
sync_binlog=0

Those are the heaviest IO users. This should help replication catch up. I would also do replica_parallel_workers=8

What this also means is that your replica might be disk-bound and I would check the await% and disk utilization. Use PMM to see where most of your waits are happening.

1 Like

Hi @matthewb ,

You Sir are a genius and an angel :raised_hands:. Thanks so much for your suggestions, the servers have now caught up and I have:

Seconds_Behind_Master 0

Thanks again!

1 Like