# Xtrabackup 2.4 error last checkpoint LSN is larger than last copied LSN **URL:** \ **Category:** Percona XtraBackup\ **Created:** [February 18, 2022, 9:53am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474 "2022-02-18T09:53:11Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![edhunter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/edhunter/32/18922_2.png) [@edhunter](https://forums.percona.com/u/edhunter)\ **Post date:** [February 18, 2022, 9:53am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/1 "2022-02-18T09:53:11Z")
Hi, I am having strange problem that is happening sometimes during my nightly backups. Backup does not complete OK and have the error in the end: **xtrabackup: error: last checkpoint LSN (109863974838) is larger than last copied LSN (109863967574).** Here is my “setup” db-4 is read only slave server running running MySQL 5.7. db-restore is the server which runs nightly cronjob script using xtrabackup (percona-xtrabackup-24-2.4.24-1.el8.x86\_64) to copy the databases from db-4 to db-restore. datadir from db-4 is shared via nfs (ro,no\_root\_squash) and mounted on db-restore. Both servers are in same network, OS is Oracle Linux 8.5. On db-4 I have three mysql instances. Each of them is copied (one after another) during the night to db-restore. Only one of the instances is failing. And the strange thing is that the smallest and mostly inactive instance is failing. During the day, if I run the same backup on the same instance manually it completes successfully. Below is the log with removed most of the bulk repeating rows and tables. (Sorry, I am new user and have no permissions to attach yet). xtrabackup: recognized server arguments: --datadir=/srv/config/nfsmount --parallel=6 xtrabackup: recognized client arguments: --password=\* --backup=1 --host=192.168.111.41 --port=3316 --user=backup --target-dir=/srv/config/backup --safe-slave-backup=1 --slave-info=1 220218 00:45:02 version\_check Connecting to MySQL server with DSN ‘dbi:mysql:;mysql\_read\_default\_group=xtrabackup;host=192.168.111.41;port=3316’ as ‘backup’ (using password: YES). 220218 00:45:02 version\_check Connected to MySQL server 220218 00:45:02 version\_check Executing a version check against the server… 220218 00:45:02 version\_check Done. 220218 00:45:02 Connecting to MySQL server host: 192.168.111.41, user: backup, password: set, port: 3316, socket: not set Using server version 5.7.37-log Warning: MySQL variable ‘datadir’ points to nonexistent directory ‘/srv/cpanel/data/’ Warning: option ‘datadir’ has different values: ‘/srv/config/nfsmount’ in defaults file ‘/srv/cpanel/data/’ in SHOW VARIABLES /usr/bin/xtrabackup version 2.4.24 based on MySQL server 5.7.35 Linux (x86\_64) (revision id: b4ee263) xtrabackup: uses posix\_fadvise(). xtrabackup: cd to /srv/config/nfsmount xtrabackup: open files limit requested 0, set to 1024 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb\_data\_home\_dir = . xtrabackup: innodb\_data\_file\_path = ibdata1:12M:autoextend xtrabackup: innodb\_log\_group\_home\_dir = ./ xtrabackup: innodb\_log\_files\_in\_group = 2 xtrabackup: innodb\_log\_file\_size = 50331648 InnoDB: Number of pools: 1 220218 00:45:02 \>\> log scanned up to (109863683631) InnoDB: Opened 2 undo tablespaces InnoDB: 0 undo tablespaces made active xtrabackup: Generating a list of tablespaces InnoDB: Allocated tablespace ID 34 for mysql/servers, old maximum was 2 xtrabackup: Starting 6 threads for parallel data files transfer 220218 00:45:02 [04] Copying ./mysql/servers.ibd to /srv/config/backup/mysql/servers.ibd 220218 00:45:02 [05] Copying ./mysql/plugin.ibd to /srv/config/backup/mysql/plugin.ibd 220218 00:45:02 [01] Copying ./ibdata1 to /srv/config/backup/ibdata1 … 220218 00:47:43 \>\> log scanned up to (109863967574) 220218 00:47:44 \>\> log scanned up to (109863967574) 220218 00:47:44 Slave open temp tables: 0 220218 00:47:44 Slave is safe to backup. 220218 00:47:44 Executing FLUSH NO\_WRITE\_TO\_BINLOG TABLES… 220218 00:47:44 Executing FLUSH TABLES WITH READ LOCK… 220218 00:47:44 Starting to backup non-InnoDB tables and files 220218 00:47:44 [01] Copying ./mysql/db.opt to /srv/config/backup/mysql/db.opt 220218 00:47:44 [01] …done … 220218 00:47:45 [01] …done 220218 00:47:45 Finished backing up non-InnoDB tables and files 220218 00:47:45 [00] Writing /srv/config/backup/xtrabackup\_slave\_info 220218 00:47:45 [00] …done 220218 00:47:45 [00] Writing /srv/config/backup/xtrabackup\_binlog\_info 220218 00:47:45 [00] …done 220218 00:47:45 Executing FLUSH NO\_WRITE\_TO\_BINLOG ENGINE LOGS… xtrabackup: The latest check point (for incremental): ‘109863974838’ xtrabackup: Stopping log copying thread. .220218 00:47:45 \>\> log scanned up to (109863967574) 220218 00:47:45 Executing UNLOCK TABLES 220218 00:47:45 All tables unlocked Starting slave SQL thread 220218 00:47:45 Backup created in directory ‘/srv/config/backup/’ MySQL binlog position: filename ‘ulo-db4-cpanel-bin.000111’, position ‘375017735’, GTID of the last change ‘1b39f160-b43d-3316-1dc1-0050569ccaa1:1-57363962, 1b4a83f7-0000-3316-2dc1-0050569ccaa1:1-5000840, 1b4a83f7-b43d-3316-2dc1-0050569ccaa1:1-123972, 1b75d65b-0000-3316-5dc1-0050569ccaa1:1-2, 1b8220fc-0000-3316-1dc2-0050569ccaa1:1-2745845, 1b8220fc-b43d-3316-1dc2-0050569ccaa1:1-380600, 1b8ec709-0000-3316-2dc2-0050569ccaa1:1-131440, 1b8ec709-b43d-3316-2dc2-0050569ccaa1:1-102, ff9c1e99-0000-3316-2dc1-22c50dd5f44b:1-518708’ MySQL slave binlog position: master host ‘192.168.111.21’, purge list ‘1b39f160-b43d-3316-1dc1-0050569ccaa1:1-57363962, 1b4a83f7-0000-3316-2dc1-0050569ccaa1:1-5000840, 1b4a83f7-b43d-3316-2dc1-0050569ccaa1:1-123972, 1b75d65b-0000-3316-5dc1-0050569ccaa1:1-2, 1b8220fc-0000-3316-1dc2-0050569ccaa1:1-2745845, 1b8220fc-b43d-3316-1dc2-0050569ccaa1:1-380600, 1b8ec709-0000-3316-2dc2-0050569ccaa1:1-131440, 1b8ec709-b43d-3316-2dc2-0050569ccaa1:1-102, ff9c1e99-0000-3316-2dc1-22c50dd5f44b:1-518708’, channel name: ‘u2’ 220218 00:47:45 [00] Writing /srv/config/backup/backup-my.cnf 220218 00:47:45 [00] …done 220218 00:47:45 [00] Writing /srv/config/backup/xtrabackup\_info 220218 00:47:45 [00] …done xtrabackup: Transaction log of lsn (109863677198) to (109863967574) was copied. xtrabackup: error: last checkpoint LSN (109863974838) is larger than last copied LSN (109863967574). ---
**Author:** ![edhunter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/edhunter/32/18922_2.png) [@edhunter](https://forums.percona.com/u/edhunter)\ **Post date:** [February 22, 2022, 7:58am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/2 "2022-02-22T07:58:19Z")
No one can help here? I changed the number of parallel threads from 6 to 2, and for three days there is no problem. Then I made parallel again 6, and still no problem. Very strange, I am not sure that this is actually xtrabackup issue. Or if it is, I cannot replicate or understand the conditions causing it. ---
**Author:** ![svetasmirnova](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/svetasmirnova/32/21_2.png) [@svetasmirnova](https://forums.percona.com/u/svetasmirnova)\ **Post date:** [February 22, 2022, 8:10am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/3 "2022-02-22T08:10:37Z")
Your InnoDB redo log file size is too small, so it is rewritten while backup was running. You need to increase innodb\_log\_file\_size or take backup in less busy time. ---
**Author:** ![edhunter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/edhunter/32/18922_2.png) [@edhunter](https://forums.percona.com/u/edhunter)\ **Post date:** [February 22, 2022, 9:02am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/4 "2022-02-22T09:02:43Z")
Hi and thanks for the reply. However, I dont think that the log\_file\_size is the problem here. This is small and very lightly loaded instance. During the the backup there are less than 200 rows modified. delete - 0 rows update - 1 rows insert - 185 rows The default 2x48MB log space I believe should be enough. p.s. I dont have monitoring on the backed up server, but it is read only slave and I have monitor on the master server (and it has the same log\_file\_size configuration). I checked it and for the night the checkpoint writes are very very low - 3KB/s, and the max checkpoing lag is 30KB. ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/7/7953d6389d6107cf6d5aed21e0c8804a7a5081b6.png) ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/2/2b50966a3ee43bb2c4b8f0a1ee64969b8476a00e.png) Anyway I will increase log\_file\_size and will monitor the backups. ---
**Author:** ![edhunter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/edhunter/32/18922_2.png) [@edhunter](https://forums.percona.com/u/edhunter)\ **Post date:** [February 24, 2022, 8:01am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/5 "2022-02-24T08:01:16Z")
changing log\_file\_size from 48 to 128MB, did not help, it failed again ☹ ---
**Author:** ![svetasmirnova](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/svetasmirnova/32/21_2.png) [@svetasmirnova](https://forums.percona.com/u/svetasmirnova)\ **Post date:** [February 25, 2022, 1:11am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/6 "2022-02-25T01:11:25Z")
The “last checkpoint LSN” is an LSN before backup started. If it is larger than the last copied LSN this means that either log files were overwritten or something went terribly wrong. I noticed two issues with your XtraBackup run. ```auto Warning: MySQL variable ‘datadir’ points to nonexistent directory ‘/srv/cpanel/data/’ Warning: option ‘datadir’ has different values: ‘/srv/config/nfsmount’ in defaults file ‘/srv/cpanel/data/’ in SHOW VARIABLES ``` Please check why datadir variable in SHOW VARIABLES points to not existent directory. If you use NFS note that it cannot provide a reliable answer for XtraBackup and all kinds of inconsistency issues could happen. Also, you are using MySQL 5.7.37 while the latest version of Percona Server is 5.7.36. While XtraBackup should work for MySQL Community 5.7.37 and I do not see any backward incompatibility changes in the release notes it would be good if you check how it works with the same datadir and load with Percona Server 5.7.36. ---
**Author:** ![edhunter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/edhunter/32/18922_2.png) [@edhunter](https://forums.percona.com/u/edhunter)\ **Post date:** [February 25, 2022, 10:55am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/7 "2022-02-25T10:55:37Z")
In my opening post I explained the configuration. About datadir warning: This is because I run xtrabackup on a remote computer. Mysql server datadir /srv/cpanel/data/ is mounted via nfs in /srv/config/nfsmount (on the computer running xtrabackup). And I set this as a argument to xtrabackup --datadir=/srv/config/nfsmount About the versions: I have two other instances on the same source server, using the same mysqld 5.7.37, and they do not have problems with xtrabackup. ---
**Author:** ![svetasmirnova](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/svetasmirnova/32/21_2.png) [@svetasmirnova](https://forums.percona.com/u/svetasmirnova)\ **Post date:** [February 25, 2022, 11:28am UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/8 "2022-02-25T11:28:21Z")
While it is an unlikely reason for the redo log file issue, XtraBackup is designed to run locally. I suggest you try a larger redo log file size or try running XtraBackup on the idle instance. ---
**Author:** ![edhunter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/edhunter/32/18922_2.png) [@edhunter](https://forums.percona.com/u/edhunter)\ **Post date:** [February 25, 2022, 12:20pm UTC](https://forums.percona.com/t/xtrabackup-2-4-error-last-checkpoint-lsn-is-larger-than-last-copied-lsn/14474/9 "2022-02-25T12:20:53Z")
Forgot to ask earlier. > [@svetasmirnova](#): > > The “last checkpoint LSN” is an LSN before backup started. If it is larger than the last copied LSN this means that either log files were overwritten or something went terribly wrong. So if this is the case why the backup is not aborted in the begining when it starts to scan the logs? > [@svetasmirnova](#): > > I suggest you try a larger redo log file size or try running XtraBackup on the idle instance. Redo logs are not a problem, I already tried increasing them and showed that this does not make things any better. This instance is only 60G. If it does not fail with error, backup completes for 3 minutes. Seems we are heading a dead end here. Is there anything useful I could try to debug or report more details or whatever without changing generally my setup that has been so far flawlessly. Hm, in that relation, I think that the issue started to happen when I switched from centos 7 to oracle linux 8 (on both source and destination server). But it is strange, that the other two instances on the same servers are not affected. We have mysql enterprise subscription and mysql enterprise backup, but few years ago when I was implementing our backups, I chose percona xtrabackup and I would like to stay with it.