xtrabackup has messed up my MySQL installation

gdevelekgdevelek EntrantInactive User Role Participant
I'm posting here at the recommendation of [email protected], after a 1st post at StackExchange.

Last Thursday I experimented a bit with Percona xtrabackup on a MySQL 5.5.23 server (we're stuck at this very old version, that's another discussion). I installed the Percona repo as part of the task. I uninstalled the tool before the end of the day, but didn't uninstall the repo.

On Friday I noticed that when I ran "mysql -u root -p" I got a segmentation fault!

I investigated and found (using the rpm -qa --last command) that at 5:19am Friday morning, when we were all asleep, somehow, Percona-Server-Shared was installed (by itself??), and /usr/lib64/libmysqlclient.so.16.0.0 is now part of that package, as I confirm with the rpm -qf command. I suspect that's why the plain vanilla mysql client is dumping core, it can't find the libraries it expects.

When I attempt to uninstall Percona server, I see that it cascades to remove the mysql-server package itself (!), so I can't do it.

At StackExchange I was told that this is a known bug that plagues the 5.5.23 MySQL version and I received the recommendation to uninstall Percona-Server-Shared using the --nodeps option, and then install mysql-libs.

I must fix this problem ASAP, the server is a staging server (no I didn't do it on a production server :-) ) It's running Redhat 6.9.

Thanks for your help,

George

Comments

  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Mentor
    Hello George, thanks for joining the Forum. It's not to obfuscate - I'll happily update the StackExchange post if we can bottom this out - but many of our engineers don't feedback on that site so you are more likely to get the answer over here.

    OK ... the advice on the last comment is what we'll need to help you here, so that's a good start. I've reproduced that here, though if you could provide the requested output that might move us forward. :

    Please provide - output of
    rpm -qa --last
    
    - output of the
    yum install xtrabackup
    
    > libmysqlclient.so.16.0.0 is now part of that package, Percona no longer provides libmysqlclient.

    This should fix:
    rpm -ev --nodeps Percona-Server-shared
    yum install mysql-libs
    

    As everyone else advised: - spin up free-tier aws - setup 5.5.23 - setup xtrabackup - remove xtrabackup - check if same problem - apply the steps suggested above.
  • gdevelekgdevelek Entrant Inactive User Role Participant
    Hi Lorraine,

    Attached is the output of rpm -qa --last on the problematic host. I can't go back far enough to show you the output of yum install xtrabackup.
    I have launched a CentOS 6.9 AWS instance with MySQL 5.5.60 (I don't think I can find 5.5.23). I installed the Percona repo and then xtrabackup-2.4 Then I removed xtrabackup. Now I'll have to wait till tomorrow to see if the host (I guess through yum auto tasks) will install Percona Shared Server during the night. That's what worries me, if this host does not run whatever it was that the other host ran with yum and therefore does not install PSS overnight, this test we're running will fail. Do you know more about this, i.e. how PSS was installed "on its own"?

    Thanks,
    George

    # rpm -qa --last |grep Perco
    Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64 Fri 01 Jun 2018 03:19:49 AM BST
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Mentor
    Hi George, thanks for the update.
    I can't personally advance this (not enough knowledge), but I will bring it to the attention of the tech team, and hopefully they will have some insight for you on this issue.
    What is your current situation?
    If you prefer you are welcome to email me direct with that info [email protected]
  • gdevelekgdevelek Entrant Inactive User Role Participant
    My current situation is as I have described it above. If something is unclear I'll be happy to clarify.

    I'm looking forward to any feedback from your eng team.
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Mentor
    Hi again George

    OK, I have some feedback. The suggestions provided by Marcos and that I reproduced above are sound, and should help you out of this.

    Unfortunately (or fortunately, whichever way you look at it) the version that you are trialling on, 5.5.60, won't reproduce the issue since it was fixed in 5.5.26 (and your version I understand is 5.5.23). So to reproduce the problem, you'd need 5.5.23

    To explain what's happened: during the set up of XtraBackup that you did, there would have been some questions that you had to answer Y or N to, and one of those would have been whether to "replace mysql-libs with Percona-Server-shared-51-5.1.73-rel14.12.625.rhel6.x86_64" - we can see that from the logs. So that is likely to be where the problem arose.

    In the original post on DBA Stack Exchange you mentioned that you were currently stuck with this old version of MySQL. If you you are in a position to and want to get help with that, our support or consultancy services might well be of use to you. Likewise if you need help on setting up a rigorous back up strategy for your current set up. I realize not everyone can or wants to do that, but I'm throwing it out there in case it's of benefit.

    Meanwhile, I hope the suggested steps work for you!
  • gdevelekgdevelek Entrant Inactive User Role Participant
    OK, thanks, I'll try to find 5.5.23 for my trial.
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Mentor
    You should be able to get it from here ... https://downloads.mysql.com/archives/community/ ... ?
  • gdevelekgdevelek Entrant Inactive User Role Participant
    Are you perhaps aware of a repository where old versions are archived? I can't find it anywhere....

    What if I go ahead and explicitly install using yum the PSS, wouldn't I effectively reproduce the problem?
  • gdevelekgdevelek Entrant Inactive User Role Participant
    Oh I see I missed your post, yes I'll get it from there tomorrow, thanks.
  • gdevelekgdevelek Entrant Inactive User Role Participant
    Good morning,
    Here is my update:

    I removed 5.5.60 and installed 5.5.23 from the link you kindly provided.
    I ensured I can login as root with mysql client.
    I then installed xtrabackup-24 (log below).
    I was not asked to replace libs as you wrote above.
    I checked and found I can still login as root with mysql client, there is no problem now.
    Conclusion: no problem this time, the problematic situation was not replicated



    # yum install percona-xtrabackup-24
    Loaded plugins: presto, security
    Setting up Install Process
    epel/metalink | 19 kB 00:00
    percona-release-noarch | 2.9 kB 00:00
    percona-release-x86_64 | 2.9 kB 00:00
    remi-safe | 2.9 kB 00:00
    ultra-centos-6.9-base | 3.4 kB 00:00
    ultra-centos-6.9-extras | 3.3 kB 00:00
    ultra-centos-6.9-updates | 3.4 kB 00:00
    ultra-centos-6.x-glusterfs | 2.9 kB 00:00
    ultra-centos-6.x-ultrarepo | 2.9 kB 00:00
    Resolving Dependencies
    --> Running transaction check
    ---> Package percona-xtrabackup-24.x86_64 0:2.4.11-1.el6 will be installed
    --> Finished Dependency Resolution

    Dependencies Resolved

    ================================================== ==============================
    Package Arch Version Repository Size
    ================================================== ==============================
    Installing:
    percona-xtrabackup-24 x86_64 2.4.11-1.el6 percona-release-x86_64 8.1 M

    Transaction Summary
    ================================================== ==============================
    Install 1 Package(s)

    Total download size: 8.1 M
    Installed size: 8.1 M
    Is this ok [y/N]: y
    Downloading Packages:
    Setting up and reading Presto delta metadata
    Processing delta metadata
    Package(s) data still to download: 8.1 M
    percona-xtrabackup-24-2.4.11-1.el6.x86_64.rpm | 8.1 MB 00:00
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : percona-xtrabackup-24-2.4.11-1.el6.x86_64 1/1
    Verifying : percona-xtrabackup-24-2.4.11-1.el6.x86_64 1/1

    Installed:
    percona-xtrabackup-24.x86_64 0:2.4.11-1.el6

    Complete!
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Mentor
    Could you let us have the output of this, please?
    rpm -qa |egrep -i "mysql|maria|percona";
    

    Thanks!
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.