Some Help with Percona Cluster please

MySQL & MariaDB Other MySQL® Questions
1

Hi Guys
I seem to have an issue with one of my nodes in a cluster, it wouldn’t start then it started to replicate and I am not sure if it is actually running now.

So firstly this node is
mysql --version
mysql Ver 14.14 Distrib 5.6.26-74.0, for debian-linux-gnu (x86_64) using readline 6.3

The other nodes are all
mysql Ver 14.14 Distrib 5.6.15, for Linux (x86_64) using EditLine wrapper

When I look at the node I see

service mysql status

  • MySQL (Percona XtraDB Cluster) is stopped. Check log

But when I check with

ps -ef | grep mysql
ldadmin 15010 14537 0 10:25 pts/0 00:00:00 grep --color=auto mysql
root 27595 1 0 Nov03 pts/1 00:00:00 /bin/sh /usr/bin/mysqld_safe
mysql 28301 27595 3 Nov03 pts/1 00:33:42 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera_smm.so --log-error=/var/log/mysql/mysql-error.log --open-files-limit=65535 --pid-file=/var/lib/mysql/mysql.pid --socket=/var/lib/mysql/mysql.sock --wsrep_start_position=d4497934-60d2-11e5-b80b-f6ba089aaa65:51357276

anyone able to shed any light on this.

thanks

2

Send us output of the following:

dpkg -l | grep -i percona

How did you install the packages and can you paste last 50 lines from the error log?

3

Hi and thank you for the reply.
I installed it from apt-get

dpkg -l | grep -i percona
ii percona-xtrabackup 2.2.12-1.trusty amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-56 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster with Galera
ii percona-xtradb-cluster-client-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-3 3.12.2-1.trusty amd64 Metapackage for latest version of galera3.
ii percona-xtradb-cluster-galera-3.x 3.12.2-1.trusty amd64 Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database server binaries

4

Forgot the error log. So it does look like its working, I think the see if the service is running daemon just didn’t wait long enough before giving up on it.

2015-11-03 17:34:33 28301 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 537a7737-81fd-11e5-8cc7-f3f60367334b
2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: sent state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b
2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 0 (PERCONA-CLUST-002)
2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 1 (PERCONA-CLUST-001)
2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 2 (PERCONA-CLUST-004)
2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 3 (PERCONA-CLUST-005)
2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 4 (PERCONA-CLUST-003)
2015-11-03 17:34:33 28301 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 169,
members = 4/5 (joined/total),
act_id = 51359073,
last_appl. = -1,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = d4497934-60d2-11e5-b80b-f6ba089aaa65
2015-11-03 17:34:33 28301 [Note] WSREP: Flow-control interval: [36, 36]
2015-11-03 17:34:33 28301 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 51359073)
2015-11-03 17:34:33 28301 [Note] WSREP: State transfer required:
Group state: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359073
Local state: 00000000-0000-0000-0000-000000000000:-1
2015-11-03 17:34:33 28301 [Note] WSREP: New cluster view: global state: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359073, view# 170: Primary, number of nodes: 5, my index: 0, protocol version 2
2015-11-03 17:34:33 28301 [Note] WSREP: closing client connections for protocol change 3 → 2
2015-11-03 17:34:35 28301 [Note] WSREP: (529507d5, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2015-11-03 17:34:35 28301 [Warning] WSREP: Gap in state sequence. Need state transfer.
2015-11-03 17:34:35 28301 [Note] WSREP: Running: 'wsrep_sst_rsync --role ‘joiner’ --address ‘10.150.4.31’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘28301’ ‘’ ’
2015-11-03 17:34:35 28301 [Note] WSREP: Prepared SST request: rsync|10.150.4.31:4444/rsync_sst
2015-11-03 17:34:35 28301 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-11-03 17:34:35 28301 [Note] WSREP: REPL Protocols: 5 (3, 1)
2015-11-03 17:34:35 28301 [Note] WSREP: Service thread queue flushed.
2015-11-03 17:34:35 28301 [Note] WSREP: Assign initial position for certification: 51359073, protocol version: 3
2015-11-03 17:34:35 28301 [Note] WSREP: Service thread queue flushed.
2015-11-03 17:34:35 28301 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (d4497934-60d2-11e5-b80b-f6ba089aaa65): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2015-11-03 17:34:35 28301 [Note] WSREP: Member 0.0 (PERCONA-CLUST-002) requested state transfer from ‘any’. Selected 1.0 (PERCONA-CLUST-001)(SYNCED) as donor.
2015-11-03 17:34:35 28301 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 51359124)
2015-11-03 17:34:35 28301 [Note] WSREP: Requesting state transfer: success, donor: 1
2015-11-03 17:53:10 28301 [Note] WSREP: 1.0 (PERCONA-CLUST-001): State transfer to 0.0 (PERCONA-CLUST-002) complete.
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 28340 (20151103 17:53:11.492)
WSREP_SST: [INFO] Joiner cleanup done. (20151103 17:53:12.006)
2015-11-03 17:53:12 28301 [Note] WSREP: SST complete, seqno: 51359125
2015-11-03 17:53:12 28301 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
2015-11-03 17:53:12 28301 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release
2015-11-03 17:53:12 28301 [Note] Plugin ‘FEDERATED’ is disabled.
2015-11-03 17:53:12 28301 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-11-03 17:53:12 28301 [Note] InnoDB: The InnoDB memory heap is disabled
2015-11-03 17:53:12 28301 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-11-03 17:53:12 28301 [Note] InnoDB: Memory barrier is not used
2015-11-03 17:53:12 28301 [Note] InnoDB: Compressed tables use zlib 1.2.8
2015-11-03 17:53:12 28301 [Note] InnoDB: Using Linux native AIO
2015-11-03 17:53:12 28301 [Note] InnoDB: Using CPU crc32 instructions
2015-11-03 17:53:12 28301 [Note] InnoDB: Initializing buffer pool, size = 6.0G
2015-11-03 17:53:12 28301 [Note] InnoDB: Completed initialization of buffer pool
2015-11-03 17:53:12 28301 [Note] InnoDB: Highest supported file format is Barracuda.
2015-11-03 17:53:12 28301 [Note] InnoDB: Log scan progressed past the checkpoint lsn 612983148972
2015-11-03 17:53:12 28301 [Note] InnoDB: Database was not shutdown normally!
2015-11-03 17:53:12 28301 [Note] InnoDB: Starting crash recovery.
2015-11-03 17:53:12 28301 [Note] InnoDB: Reading tablespace information from the .ibd files…
2015-11-03 17:53:16 28301 [Note] InnoDB: Restoring possible half-written data pages
2015-11-03 17:53:16 28301 [Note] InnoDB: from the doublewrite buffer…
InnoDB: Doing recovery: scanned up to log sequence number 612988391424
InnoDB: Doing recovery: scanned up to log sequence number 612992207621
2015-11-03 17:53:17 28301 [Note] InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2015-11-03 17:53:19 28301 [Warning] InnoDB: Resizing redo log from 23072 to 232768 pages, LSN=612992207621
2015-11-03 17:53:19 28301 [Warning] InnoDB: Starting to delete and rewrite log files.
2015-11-03 17:53:19 28301 [Note] InnoDB: Setting log file ./ib_logfile101 size to 512 MB
InnoDB: Progress in MB: 100 200 300 400 500
2015-11-03 17:53:21 28301 [Note] InnoDB: Setting log file ./ib_logfile1 size to 512 MB
InnoDB: Progress in MB: 100 200 300 400 500
2015-11-03 17:53:24 28301 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2015-11-03 17:53:24 28301 [Warning] InnoDB: New log files created, LSN=612992207884
2015-11-03 17:53:24 28301 [Note] InnoDB: 128 rollback segment(s) are active.
2015-11-03 17:53:24 28301 [Note] InnoDB: Waiting for purge to start
2015-11-03 17:53:24 28301 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 612992207621
2015-11-03 17:53:24 28301 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2015-11-03 17:53:24 28301 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2015-11-03 17:53:24 28301 [Note] Server hostname (bind-address): ‘*’; port: 3306
2015-11-03 17:53:24 28301 [Note] IPv6 is available.
2015-11-03 17:53:24 28301 [Note] - ‘::’ resolves to ‘::’;
2015-11-03 17:53:24 28301 [Note] Server socket created on IP: ‘::’.
2015-11-03 17:53:24 28301 [ERROR] /usr/sbin/mysqld: Table ‘./mysql/user’ is marked as crashed and should be repaired
2015-11-03 17:53:24 28301 [Warning] Checking table: ‘./mysql/user’
2015-11-03 17:53:24 28301 [ERROR] 1 client is using or hasn’t closed the table properly
2015-11-03 17:53:24 28301 [ERROR] /usr/sbin/mysqld: Table ‘./mysql/db’ is marked as crashed and should be repaired
2015-11-03 17:53:24 28301 [Warning] Checking table: ‘./mysql/db’
2015-11-03 17:53:24 28301 [ERROR] 1 client is using or hasn’t closed the table properly
2015-11-03 17:53:24 28301 [Warning] ‘user’ entry ‘root@percona-clust-001’ ignored in --skip-name-resolve mode.
2015-11-03 17:53:24 28301 [Warning] ‘proxies_priv’ entry ‘@ root@percona-clust-001’ ignored in --skip-name-resolve mode.
2015-11-03 17:53:24 28301 [Note] Event Scheduler: Loaded 0 events
2015-11-03 17:53:24 28301 [Note] WSREP: Signalling provider to continue.
2015-11-03 17:53:24 28301 [Note] WSREP: Initialized wsrep sidno 2
2015-11-03 17:53:24 28301 [Note] WSREP: SST received: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359125
2015-11-03 17:53:24 28301 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.6.26-74.0-56-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel74.0, Revision 624ef81, WSREP version 25.12, wsrep_25.12
2015-11-03 17:53:24 28301 [Note] WSREP: 0.0 (PERCONA-CLUST-002): State transfer from 1.0 (PERCONA-CLUST-001) complete.
2015-11-03 17:53:24 28301 [Note] WSREP: Shifting JOINER → JOINED (TO: 51394795)
2015-11-03 17:54:59 28301 [Note] WSREP: Member 1.0 (PERCONA-CLUST-001) synced with group.
2015-11-03 17:55:00 28301 [Note] WSREP: Member 0.0 (PERCONA-CLUST-002) synced with group.
2015-11-03 17:55:00 28301 [Note] WSREP: Shifting JOINED → SYNCED (TO: 51398066)
2015-11-03 17:55:00 28301 [Note] WSREP: Synchronized with group, ready for connections
2015-11-03 17:55:00 28301 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-11-04 16:47:28 28301 [ERROR] /usr/sbin/mysqld: Table ‘./mysql/proc’ is marked as crashed and should be repaired
2015-11-04 16:47:28 28301 [Warning] Checking table: ‘./mysql/proc’