XtraDB 8.0.29 Bug Related? PS-8292

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

I am having a bit of a issue. Recently I moved from 8.0.28 to the .29 version

Recently I had a crash that exceeded the IST time and I am attempting to do a SST transfer. Mind you I have done this quite a few times on these systems with no issues at all. I made sure that the xtrabackup also was upgraded on the systems in question.

After spending most of the day on this I am thinking that on a pretty busy database cluster the following is going on

SST logs on the Donor system shows Completed OK with no issues

The System I am attempting to recover during the prepare phase shows the followin

2023-08-22T17:29:23.649319-04:00 0 [Note] [MY-012533] [InnoDB] 50%
2023-08-22T17:29:26.637211-04:00 0 [Note] [MY-012533] [InnoDB] 60%
2023-08-22T17:29:29.262166-04:00 0 [Note] [MY-012533] [InnoDB] 70%
2023-08-22T17:29:31.522048-04:00 0 [Note] [MY-012533] [InnoDB] 80%
2023-08-22T17:29:33.849720-04:00 0 [Note] [MY-012533] [InnoDB] 90%
2023-08-22T17:29:35.415138-04:00 0 [Note] [MY-012533] [InnoDB] 100%
2023-08-22T17:29:36.137850-04:00 0 [ERROR] [MY-012711] [InnoDB] Wrong owned count 1, 4, rec 13915
2023-08-22T17:29:36.137918-04:00 0 [Note] [MY-011876] [InnoDB] Page dump in ascii and hex (16384 bytes):

And Crashes… This lead me down the path of thinking that the innodb files were somehow corrupt or that the donor system was having some sort of issue or even a LAN/WAN issue. All items as far as I could tell checked out (I tried loading it from 4 different Donor systems all items appear to be the same)

I then found this and wonder if this is the cause of the issue

https://jira.percona.com/browse/PS-8292

It does not appear that the rsync and mysqldump methods are no longer supported at all.

Before I upgrade to the .30 version can someone potentially confirm that this is the issue?

Thank you

2

This appears to be the root cause… I upgraded two of the 5 nodes. Set the donor to the .30 version and items completed properly In the event anyone else has this issue…

3

Hey @meyerder,
8.0.29 had an extremely critical bug that could lead to data loss. It was recommended that everyone skip that release. It should be noted in the release notes.

4

@matthewb I reviewed the change log for .30 and at least at my level of understanding I do not see that notice…

Could you point out where as I am not seeing it (And I figured I would see it)

Thank you

5

Question MAY be associated with this… One of the Two that I upgraded last night crashed again… I know that running different versions in the cluster is not suggested but I was not ready to pull the trigger yet on all the systems. I noticed this on the system that was at .30 which COULD be related maybe?

2023-08-22T23:42:38.771342Z 12 [Note] [MY-000000] [WSREP] Updating WSREPXid: 40deb9d6-ab4f-11ea-bfba-532a142ff8ef:6903465699
2023-08-22T23:42:38.771385Z 12 [Note] [MY-000000] [WSREP] Thread with id (12) granted turn to proceed
2023-08-22T23:42:38.771396Z 12 [Note] [MY-000000] [WSREP] Un-Registering thread with id (12) from wsrep group commit queue
2023-08-22T23:42:38.771730Z 12 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 6903465699, 0
2023-08-22T23:42:38.771789Z 12 [Note] [MY-000000] [WSREP] wsrep: committed write set (6903465699)
2023-08-22T23:42:38.771799Z 12 [Note] [MY-000000] [WSREP] wsrep_after_apply 12
2023-08-22T23:42:38.771871Z 0 [Warning] [MY-000000] [Galera] Failed to report last committed 40deb9d6-ab4f-11ea-bfba-532a142ff8ef:6903465698, -77 (File descriptor in bad state)
6

Without going too deep, I would recommend you go to 8.0.32. .29, .30, and .31 all had various issues that came from upstream. Percona has some fixes in place, but things were just wobbly during those versions. .32 is the best stable version right now, IMO.

7

@matthewb Took your advice on a Smaller system of mine… One item that just does not seem right… and for some users it seems like if the lib they are using is older they get a ssl bad handshake at times… (Python) a upgrade of the librarys for python seems to fix it…

8.0.33-25.1 has this
Percona XtraDB Cluster (GPL), Release rel25, Revision 0c56202, WSREP version 26.1.4.3

8.0.29-21.1 has this
Percona XtraDB Cluster (GPL), Release rel21, Revision 250bc93, WSREP version 26.4.3

Seems really weird that the version would DECREASE as this is just a comment maybe a typo?

version_comment              Percona XtraDB Cluster (GPL), Release rel25, Revision 0c56202, WSREP version 26.1.4.3                                                                                                    Percona XtraDB Cluster (GPL), Release rel21, Revision 250bc93, WSREP version 26.4.3
8

Possibly a typo; Since everything is 100% OSS, you can go through the source code on github and see where the change to the version_comment happened.