Unable to install XtraDBCluster on a server where Percona Server for MySQL is running

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

I have Percona Sever for MySQL 5.7 installed in my linux server. When i install XtraDBCluster 5.7 in my server. I get the following error (highlighted in red). Based on the error message I see that there are conflicting files between the percona-xtradb-cluster-server and percona-server-common. I tried uninstalling the percona-server-common files and then installling the xtradb-cluster. The installation is successful but the percona mysql server doesn’t come up when I try to bootstrap the server. How do I resolve this issue?
OS: Ubuntu TLS 16.04
Server: AWS EC2 instances

[I]$ sudo apt-get install percona-xtradb-cluster-57
[sudo] password for indiadev:
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
libboost-program-options1.58.0 percona-server-common-5.7 percona-xtradb-cluster-garbd-5.7 percona-xtradb-cluster-garbd-debug-5.7
Use ‘sudo apt autoremove’ to remove them.
The following additional packages will be installed:
percona-xtradb-cluster-client-5.7 percona-xtradb-cluster-common-5.7 percona-xtradb-cluster-server-5.7
Suggested packages:
tinyca
The following packages will be REMOVED:
percona-server-client-5.7 percona-server-server-5.7
The following NEW packages will be installed:
percona-xtradb-cluster-57 percona-xtradb-cluster-client-5.7 percona-xtradb-cluster-common-5.7 percona-xtradb-cluster-server-5.7
0 upgraded, 4 newly installed, 2 to remove and 7 not upgraded.
Need to get 0 B/12.5 MB of archives.
After this operation, 71.3 MB disk space will be freed.
Do you want to continue? [Y/n] y
Preconfiguring packages …
(Reading database … 84071 files and directories currently installed.)
Removing percona-server-server-5.7 (5.7.22-22-1.xenial) …
Removing percona-server-client-5.7 (5.7.22-22-1.xenial) …
Processing triggers for man-db (2.7.5-1) …
Selecting previously unselected package percona-xtradb-cluster-common-5.7.
(Reading database … 83904 files and directories currently installed.)
Preparing to unpack …/percona-xtradb-cluster-common-5.7_5.7.21-29.26-1.xenial_amd64.deb …
Unpacking percona-xtradb-cluster-common-5.7 (5.7.21-29.26-1.xenial) …
Selecting previously unselected package percona-xtradb-cluster-client-5.7.
Preparing to unpack …/percona-xtradb-cluster-client-5.7_5.7.21-29.26-1.xenial_amd64.deb …
Unpacking percona-xtradb-cluster-client-5.7 (5.7.21-29.26-1.xenial) …
Setting up percona-xtradb-cluster-common-5.7 (5.7.21-29.26-1.xenial) …
dpkg-query: no path found matching pattern /etc/mysql/my.cnf

  • The suggested mysql options and settings are in /etc/mysql/percona-server.conf.d/mysqld.cnf
  • If you want to use mysqld.cnf as default configuration file please make backup of /etc/my.cnf
  • Once it is done please execute the following commands:
    rm -rf /etc/mysql/my.cnf
    update-alternatives --install /etc/mysql/my.cnf my.cnf “/etc/mysql/percona-xtradb-cluster.cnf” 200

Selecting previously unselected package percona-xtradb-cluster-server-5.7.
(Reading database … 83933 files and directories currently installed.)
Preparing to unpack …/percona-xtradb-cluster-server-5.7_5.7.21-29.26-1.xenial_amd64.deb …
…Unpacking percona-xtradb-cluster-server-5.7 (5.7.21-29.26-1.xenial) …
[COLOR=#FF0000]dpkg: error processing archive /var/cache/apt/archives/percona-xtradb-cluster-server-5.7_5.7.21-29.26-1.xenial_amd64.deb (–unpack):
trying to overwrite ‘/usr/share/mysql/danish/errmsg.sys’, which is also in package percona-server-common-5.7 5.7.22-22-1.xenial
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Selecting previously unselected package percona-xtradb-cluster-57.
Preparing to unpack …/percona-xtradb-cluster-57_5.7.21-29.26-1.xenial_amd64.deb …
Unpacking percona-xtradb-cluster-57 (5.7.21-29.26-1.xenial) …
Processing triggers for man-db (2.7.5-1) …
Errors were encountered while processing:
/var/cache/apt/archives/percona-xtradb-cluster-server-5.7_5.7.21-29.26-1.xenial_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)[/I]

2

You can’t install Percona Server for MySQL and Percona-XtraDB-Cluster on the same server, you either have one or the other. What were you trying to achieve?

3

We are migrating away from Maria DB (AWS). As a first step we set up the Percona MySQL Server and imported the data. Next we are trying to set up the XtraDBClustering. The documentation says that we should have a MySQL instance in place before setting up the XtraDB cluster.

4

Hey there percona_user_1981 let me see if I can get someone from the team to get you on the right track. Bear with me? Thanks!

5

Hi

You mention that:

Do you mean the installation of percona-xtradb-cluster or percona-server packages?

As mentioned by phoenix you’ll need to remove Percona Server packages before installing Percona XtraDB Cluster.

In case you successfully installed PXC and the issue is bootstrapping, error log should provide details of why it’s failing. Can you please share it here.

–
Hrvoje

6

@hrvojem What i meant was I am able to install only if i remove the conflicting percona my sql server package. I don’t see a error message in the shell nor do i see it in the error log. I have given the commands I used and please see the output. Note, when I installed XtraDBCluster, it didn’t ask for a root password to be set for the database.

Test-Mysql-cluster-01:~/percona$ sudo service mysql start
Failed to start mysql.service: Unit mysql.service is masked.
Test-Mysql-cluster-01:~/percona$ sudo /etc/init.d/mysql
Test-Mysql-cluster-01:~/percona$ date
Wed Jun 13 14:28:21 UTC 2018
Test-Mysql-cluster-01:~/percona$ sudo /etc/init.d/mysql
Test-Mysql-cluster-01:~/percona$ sudo /etc/init.d/mysql bootstrap-pxc
Test-Mysql-cluster-01:~/percona$ sudo service mysql status
● mysql.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead) since Tue 2018-06-12 16:10:17 UTC; 22h ago
Main PID: 63108 (code=exited, status=0/SUCCESS)

Jun 12 14:11:44 Test-Mysql-cluster-01 systemd[1]: Starting Percona Server…
Jun 12 14:12:02 Test-Mysql-cluster-01 systemd[1]: Started Percona Server.
Jun 12 16:10:09 Test-Mysql-cluster-01 systemd[1]: Stopping Percona Server…
Jun 12 16:10:17 Test-Mysql-cluster-01 systemd[1]: Stopped Percona Server.
Jun 12 16:10:34 Test-Mysql-cluster-01 systemd[1]: Stopped mysql.service.
Jun 13 14:22:40 Test-Mysql-cluster-01 systemd[1]: Stopped mysql.service.

7

Any inputs on how to proceed further? Hope the steps that I am trying to take is correct
Step 1: installed percona server for mysql
Step 2: Imported mysql dump extracted from AWS Maria DB
Step 3: Installation of XtraDB cluster on server 1
Step 4: Bootstrap the node 1

8

Can you point me to the docs that mention that Percona Server needs to be installed first when migrating? How did you import data (mysqldump)?
Installing the PXC and then importing data should work (without installing Percona Server first).

Please check the configuration file (defaults to: /etc/mysql/my.cnf) to see where the error log is (default should be: /var/log/mysql.log). Error log should provide more information on why it won’t bootstrap.

My suggestion would be to first set up a cluster as described in the how-to: https://www.percona.com/doc/percona-xtradb-cluster/5.7/howtos/ubuntu_howto.html
and then to import the data.

This is because you’ve set it up when you installed Percona Server, and PXC uses the same (default) datadir. This means the password should be the same as one you entered when installing PS.

–
Hrvoje

9

@hrvojem Thank you for your response. I did like you said, removed all the installations and then installed only XtraDBCluster. The installation was successful.
I did the same steps in node 2 and XtraDB cluster installation was successful.

I bootstrapped node 1 - this step was successful
I started node 2 and the node 2 joined the cluster and node 1 was preparing for data transfer to node 2. At this step, I see a failure that I am unable to resolve. Shared snippents from different log files below. Kindly let me know what could be going wrong

Node 1 Donor’s innobackup.backup.log:

[I]180615 09:22:07 Executing UNLOCK BINLOG
180615 09:22:07 Executing UNLOCK TABLES
180615 09:22:07 All tables unlocked
180615 09:22:07 [00] Streaming ib_buffer_pool to
180615 09:22:07 [00] …done
180615 09:22:07 Backup created in directory ‘/tmp/pxc_sst_YhU0/donor_xb_YwGx/’
MySQL binlog position: filename ‘Test-Mysql-cluster-01-bin.000005’, position ‘154’
180615 09:22:07 [00] Streaming
180615 09:22:07 [00] …done
180615 09:22:07 [00] Streaming
180615 09:22:07 [00] …done
xtrabackup: Transaction log of lsn (2558687) to (2558712) was copied.
09:22:07 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x3b)[0xd31ccb]
xtrabackup(handle_fatal_signal+0x281)[0xaa0e31]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd524a78390]
xtrabackup(_Z16os_event_destroyRP8os_event+0x7)[0x8731a7]
xtrabackup(_Z11os_aio_freev+0x34)[0x8535f4]
xtrabackup(_Z19xb_data_files_closev+0xd1)[0x6fba81]
xtrabackup(_Z22xtrabackup_backup_funcv+0x2446)[0x6ff436]
xtrabackup(main+0x137a)[0x6e110a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd522a2e830]
xtrabackup(_start+0x29)[0x6f5309]

Please report a bug at [url]Log in - Percona JIRA

Node 1 MySQL Error Log:
2018-06-15T09:21:52.726412Z 0 [Note] WSREP: Member 0.0 (rs-platform-db-node2) requested state transfer from ‘any’. Selected 1.0 (rs-platform-db-node1)(SYNCED) as donor.
2018-06-15T09:21:52.726441Z 0 [Note] WSREP: Shifting SYNCED → DONOR/DESYNCED (TO: 3)
2018-06-15T09:21:52.726532Z 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2018-06-15T09:21:52.726695Z 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘172.17.117.22:4444/xtrabackup_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘13dbbb02-6fff-11e8-baa3-ca759da50a8b:3’)
2018-06-15T09:21:52.727185Z 6 [Note] WSREP: DONOR thread signaled with 0
encryption: using gcrypt 1.6.5
2018-06-15T09:21:54.683744Z 0 [Note] WSREP: (be775797, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2018-06-15T09:22:03.053771Z WSREP_SST: [INFO] Streaming the backup to joiner at 172.17.117.22 4444
2018-06-15T09:22:07.414811Z 528 [Note] Aborted connection 528 to db: ‘unconnected’ user: ‘sstuser’ host: ‘localhost’ (Got an error reading communication packets)
2018-06-15T09:22:07.417226Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************
2018-06-15T09:22:07.419154Z WSREP_SST: [ERROR] xtrabackup finished with error: 2. Check /var/lib/mysql//innobackup.backup.log
2018-06-15T09:22:07.422709Z WSREP_SST: [ERROR] ******************************************************
2018-06-15T09:22:07.425057Z WSREP_SST: [ERROR] Cleanup after exit with status:22
2018-06-15T09:22:07.458112Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘172.17.117.22:4444/xtrabackup_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘13dbbb02-6fff-11e8-baa3-ca759da50a8b:3’: 22 (Invalid argument)
2018-06-15T09:22:07.458186Z 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role ‘donor’ --address ‘172.17.117.22:4444/xtrabackup_sst//1’ --socket ‘/var/run/mysqld/mysqld.sock’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ ‘’ --gtid ‘13dbbb02-6fff-11e8-baa3-ca759da50a8b:3’
2018-06-15T09:22:07.458634Z 0 [Warning] WSREP: 1.0 (rs-platform-db-node1): State transfer to 0.0 (rs-platform-db-node2) failed: -22 (Invalid argument)
2018-06-15T09:22:07.458659Z 0 [Note] WSREP: Shifting DONOR/DESYNCED → JOINED (TO: 3)
2018-06-15T09:22:07.458957Z 0 [Note] WSREP: Member 1.0 (rs-platform-db-node1) synced with group.
2018-06-15T09:22:07.458969Z 0 [Note] WSREP: Shifting JOINED → SYNCED (TO: 3)
2018-06-15T09:22:07.459042Z 2 [Note] WSREP: Synchronized with group, ready for connections
2018-06-15T09:22:07.459082Z 2 [Note] WSREP: Setting wsrep_ready to true
2018-06-15T09:22:07.459093Z 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2018-06-15T09:22:08.459403Z 0 [Note] WSREP: forgetting 895852b1 (tcp://172.17.117.22:4567)

Node 2 MySQL Log:

2018-06-15T09:21:52.730176Z 0 [Note] WSREP: Member 0.0 (rs-platform-db-node2) requested state transfer from ‘any’. Selected 1.0 (rs-platform-db-node1)(SYNCED) as donor.
2018-06-15T09:21:52.730212Z 0 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 3)
2018-06-15T09:21:52.730244Z 2 [Note] WSREP: Requesting state transfer: success, donor: 1
2018-06-15T09:21:52.730267Z 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 → 13dbbb02-6fff-11e8-baa3-ca759da50a8b:3
2018-06-15T09:21:53.102382Z WSREP_SST: [INFO] Proceeding with SST…
encryption: using gcrypt 1.6.5
2018-06-15T09:21:54.732463Z 0 [Note] WSREP: (895852b1, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2018-06-15T09:21:56.171996Z WSREP_SST: [INFO] …Waiting for SST streaming to complete!
2018-06-15T09:22:07.431719Z WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst
2018-06-15T09:22:07.462673Z 0 [Warning] WSREP: 1.0 (rs-platform-db-node1): State transfer to 0.0 (rs-platform-db-node2) failed: -22 (Invalid argument)
2018-06-15T09:22:07.462710Z 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():766: Will never receive state. Need to abort.
2018-06-15T09:22:07.462733Z 0 [Note] WSREP: gcomm: terminating thread
2018-06-15T09:22:07.462752Z 0 [Note] WSREP: gcomm: joining thread
2018-06-15T09:22:07.462814Z 0 [Note] WSREP: gcomm: closing backend
2018-06-15T09:22:10.734622Z 0 [Note] WSREP: (895852b1, ‘tcp://0.0.0.0:4567’) connection to peer be775797 with addr tcp://172.17.101.184:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)
2018-06-15T09:22:10.734749Z 0 [Note] WSREP: (895852b1, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://172.17.101.184:4567
2018-06-15T09:22:12.234623Z 0 [Note] WSREP: (895852b1, ‘tcp://0.0.0.0:4567’) reconnecting to be775797 (tcp://172.17.101.184:4567), attempt 0
2018-06-15T09:22:12.962976Z 0 [Note] WSREP: declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)
2018-06-15T09:22:12.963018Z 0 [Note] WSREP: evs::proto(895852b1, LEAVING, view_id(REG,895852b1,6)) suspecting node: be775797
2018-06-15T09:22:12.963027Z 0 [Note] WSREP: evs::proto(895852b1, LEAVING, view_id(REG,895852b1,6)) suspected node without join message, declaring inactive
2018-06-15T09:22:12.963050Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,895852b1,6)
memb {
895852b1,0
}
joined {
}
left {
}
partitioned {
be775797,0
}
)
2018-06-15T09:22:12.963083Z 0 [Note] WSREP: Current view of cluster as seen by this node

10

It looks like you’ve encountered a Percona XtraBackup bug. Can you please report it [URL]Log in - Percona JIRA?
Please also add dpkg -l | grep percona output to the bug report as well.

–
Hrvoje

11

Hrvoje,

I have a similar situation. The fact is that I have a Percone 5.7 server, and now I’d like to upgrade it into a cluster.

So before I begin, what I understand is that I have to temporary stop the server, remove it, and install the cluster package, bootstrap it (will it retain the already existing database, or should I make a backup and probably import…???), and then when I start it normally it will start working as the mysql server. So in the next step I could add nodes to it, they would sync and blue skies to everyone?

Best,
Istvan

12

…Or anyone?

13

Are you saying you have encountered the same problem or just that you want to upgrade to a cluster?

14

I just want to upgrade a prod server and asking this before I encounter a problem xD

15

On Debian 9 ,if I sudo apt-get remove percona-server-client-5.7 percona-server-common-5.7 percona-server-server-5.7 and then sudo apt-get install percona-xtradb-cluster-57, the installation appears to work fine , but once completed /etc/init.d/mysql does not do anything. So that is one problem you may run into when upgrading. I have not found a fix, m work around at the moment is to start from a machine that has never had Percona or MySQL installed before.

Interestingly, on my working version of /etc/init.d/mysql and my “broken” version are identical.