# Some Help with Percona Cluster please **URL:** \ **Category:** Other MySQL® Questions\ **Created:** [November 3, 2015, 7:04pm UTC](https://forums.percona.com/t/some-help-with-percona-cluster-please/4492 "2015-11-03T19:04:53Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Lee232](https://avatars.discourse-cdn.com/v4/letter/l/f475e1/32.png) [@Lee232](https://forums.percona.com/u/Lee232)\ **Post date:** [November 3, 2015, 7:04pm UTC](https://forums.percona.com/t/some-help-with-percona-cluster-please/4492/1 "2015-11-03T19:04:53Z")
## Hi Guys I seem to have an issue with one of my nodes in a cluster, it wouldn’t start then it started to replicate and I am not sure if it is actually running now. So firstly this node is mysql --version mysql Ver 14.14 Distrib 5.6.26-74.0, for debian-linux-gnu (x86\_64) using readline 6.3 ## The other nodes are all mysql Ver 14.14 Distrib 5.6.15, for Linux (x86\_64) using EditLine wrapper When I look at the node I see service mysql status - MySQL (Percona XtraDB Cluster) is stopped. Check log But when I check with ps -ef | grep mysql ldadmin 15010 14537 0 10:25 pts/0 00:00:00 grep --color=auto mysql root 27595 1 0 Nov03 pts/1 00:00:00 /bin/sh /usr/bin/mysqld\_safe mysql 28301 27595 3 Nov03 pts/1 00:33:42 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera\_smm.so --log-error=/var/log/mysql/mysql-error.log --open-files-limit=65535 --pid-file=/var/lib/mysql/mysql.pid --socket=/var/lib/mysql/mysql.sock --wsrep\_start\_position=d4497934-60d2-11e5-b80b-f6ba089aaa65:51357276 anyone able to shed any light on this. thanks ---
**Author:** ![jrivera](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jrivera/32/13_2.png) [@jrivera](https://forums.percona.com/u/jrivera)\ **Post date:** [November 4, 2015, 1:23am UTC](https://forums.percona.com/t/some-help-with-percona-cluster-please/4492/2 "2015-11-04T01:23:17Z")
Send us output of the following: > dpkg -l | grep -i percona How did you install the packages and can you paste last 50 lines from the error log? ---
**Author:** ![Lee232](https://avatars.discourse-cdn.com/v4/letter/l/f475e1/32.png) [@Lee232](https://forums.percona.com/u/Lee232)\ **Post date:** [November 4, 2015, 4:37pm UTC](https://forums.percona.com/t/some-help-with-percona-cluster-please/4492/3 "2015-11-04T16:37:03Z")
Hi and thank you for the reply. I installed it from apt-get dpkg -l | grep -i percona ii percona-xtrabackup 2.2.12-1.trusty amd64 Open source backup tool for InnoDB and XtraDB ii percona-xtradb-cluster-56 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster with Galera ii percona-xtradb-cluster-client-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database client binaries ii percona-xtradb-cluster-common-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf) ii percona-xtradb-cluster-galera-3 3.12.2-1.trusty amd64 Metapackage for latest version of galera3. ii percona-xtradb-cluster-galera-3.x 3.12.2-1.trusty amd64 Galera components of Percona XtraDB Cluster ii percona-xtradb-cluster-server-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database server binaries ---
**Author:** ![Lee232](https://avatars.discourse-cdn.com/v4/letter/l/f475e1/32.png) [@Lee232](https://forums.percona.com/u/Lee232)\ **Post date:** [November 4, 2015, 7:40pm UTC](https://forums.percona.com/t/some-help-with-percona-cluster-please/4492/4 "2015-11-04T19:40:36Z")
Forgot the error log. So it does look like its working, I think the see if the service is running daemon just didn’t wait long enough before giving up on it. 2015-11-03 17:34:33 28301 [Note] WSREP: STATE\_EXCHANGE: sent state UUID: 537a7737-81fd-11e5-8cc7-f3f60367334b 2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: sent state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b 2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 0 (PERCONA-CLUST-002) 2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 1 (PERCONA-CLUST-001) 2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 2 (PERCONA-CLUST-004) 2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 3 (PERCONA-CLUST-005) 2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 4 (PERCONA-CLUST-003) 2015-11-03 17:34:33 28301 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf\_id = 169, members = 4/5 (joined/total), act\_id = 51359073, last\_appl. = -1, protocols = 0/5/2 (gcs/repl/appl), group UUID = d4497934-60d2-11e5-b80b-f6ba089aaa65 2015-11-03 17:34:33 28301 [Note] WSREP: Flow-control interval: [36, 36] 2015-11-03 17:34:33 28301 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 51359073) 2015-11-03 17:34:33 28301 [Note] WSREP: State transfer required: Group state: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359073 Local state: 00000000-0000-0000-0000-000000000000:-1 2015-11-03 17:34:33 28301 [Note] WSREP: New cluster view: global state: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359073, view# 170: Primary, number of nodes: 5, my index: 0, protocol version 2 2015-11-03 17:34:33 28301 [Note] WSREP: closing client connections for protocol change 3 → 2 2015-11-03 17:34:35 28301 [Note] WSREP: (529507d5, ‘tcp://0.0.0.0:4567’) turning message relay requesting off 2015-11-03 17:34:35 28301 [Warning] WSREP: Gap in state sequence. Need state transfer. 2015-11-03 17:34:35 28301 [Note] WSREP: Running: 'wsrep\_sst\_rsync --role ‘joiner’ --address ‘10.150.4.31’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘28301’ ‘’ ’ 2015-11-03 17:34:35 28301 [Note] WSREP: Prepared SST request: rsync|10.150.4.31:4444/rsync\_sst 2015-11-03 17:34:35 28301 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2015-11-03 17:34:35 28301 [Note] WSREP: REPL Protocols: 5 (3, 1) 2015-11-03 17:34:35 28301 [Note] WSREP: Service thread queue flushed. 2015-11-03 17:34:35 28301 [Note] WSREP: Assign initial position for certification: 51359073, protocol version: 3 2015-11-03 17:34:35 28301 [Note] WSREP: Service thread queue flushed. 2015-11-03 17:34:35 28301 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (d4497934-60d2-11e5-b80b-f6ba089aaa65): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():482. IST will be unavailable. 2015-11-03 17:34:35 28301 [Note] WSREP: Member 0.0 (PERCONA-CLUST-002) requested state transfer from ‘_any_’. Selected 1.0 (PERCONA-CLUST-001)(SYNCED) as donor. 2015-11-03 17:34:35 28301 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 51359124) 2015-11-03 17:34:35 28301 [Note] WSREP: Requesting state transfer: success, donor: 1 2015-11-03 17:53:10 28301 [Note] WSREP: 1.0 (PERCONA-CLUST-001): State transfer to 0.0 (PERCONA-CLUST-002) complete. WSREP\_SST: [INFO] Joiner cleanup. rsync PID: 28340 (20151103 17:53:11.492) WSREP\_SST: [INFO] Joiner cleanup done. (20151103 17:53:12.006) 2015-11-03 17:53:12 28301 [Note] WSREP: SST complete, seqno: 51359125 2015-11-03 17:53:12 28301 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 2015-11-03 17:53:12 28301 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release 2015-11-03 17:53:12 28301 [Note] Plugin ‘FEDERATED’ is disabled. 2015-11-03 17:53:12 28301 [Note] InnoDB: Using atomics to ref count buffer pool pages 2015-11-03 17:53:12 28301 [Note] InnoDB: The InnoDB memory heap is disabled 2015-11-03 17:53:12 28301 [Note] InnoDB: Mutexes and rw\_locks use GCC atomic builtins 2015-11-03 17:53:12 28301 [Note] InnoDB: Memory barrier is not used 2015-11-03 17:53:12 28301 [Note] InnoDB: Compressed tables use zlib 1.2.8 2015-11-03 17:53:12 28301 [Note] InnoDB: Using Linux native AIO 2015-11-03 17:53:12 28301 [Note] InnoDB: Using CPU crc32 instructions 2015-11-03 17:53:12 28301 [Note] InnoDB: Initializing buffer pool, size = 6.0G 2015-11-03 17:53:12 28301 [Note] InnoDB: Completed initialization of buffer pool 2015-11-03 17:53:12 28301 [Note] InnoDB: Highest supported file format is Barracuda. 2015-11-03 17:53:12 28301 [Note] InnoDB: Log scan progressed past the checkpoint lsn 612983148972 2015-11-03 17:53:12 28301 [Note] InnoDB: Database was not shutdown normally! 2015-11-03 17:53:12 28301 [Note] InnoDB: Starting crash recovery. 2015-11-03 17:53:12 28301 [Note] InnoDB: Reading tablespace information from the .ibd files… 2015-11-03 17:53:16 28301 [Note] InnoDB: Restoring possible half-written data pages 2015-11-03 17:53:16 28301 [Note] InnoDB: from the doublewrite buffer… InnoDB: Doing recovery: scanned up to log sequence number 612988391424 InnoDB: Doing recovery: scanned up to log sequence number 612992207621 2015-11-03 17:53:17 28301 [Note] InnoDB: Starting an apply batch of log records to the database… InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed 2015-11-03 17:53:19 28301 [Warning] InnoDB: Resizing redo log from 2_3072 to 2_32768 pages, LSN=612992207621 2015-11-03 17:53:19 28301 [Warning] InnoDB: Starting to delete and rewrite log files. 2015-11-03 17:53:19 28301 [Note] InnoDB: Setting log file ./ib\_logfile101 size to 512 MB InnoDB: Progress in MB: 100 200 300 400 500 2015-11-03 17:53:21 28301 [Note] InnoDB: Setting log file ./ib\_logfile1 size to 512 MB InnoDB: Progress in MB: 100 200 300 400 500 2015-11-03 17:53:24 28301 [Note] InnoDB: Renaming log file ./ib\_logfile101 to ./ib\_logfile0 2015-11-03 17:53:24 28301 [Warning] InnoDB: New log files created, LSN=612992207884 2015-11-03 17:53:24 28301 [Note] InnoDB: 128 rollback segment(s) are active. 2015-11-03 17:53:24 28301 [Note] InnoDB: Waiting for purge to start 2015-11-03 17:53:24 28301 [Note] InnoDB: Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 5.6.26-74.0 started; log sequence number 612992207621 2015-11-03 17:53:24 28301 [Note] RSA private key file not found: /var/lib/mysql//private\_key.pem. Some authentication plugins will not work. 2015-11-03 17:53:24 28301 [Note] RSA public key file not found: /var/lib/mysql//public\_key.pem. Some authentication plugins will not work. 2015-11-03 17:53:24 28301 [Note] Server hostname (bind-address): ‘\*’; port: 3306 2015-11-03 17:53:24 28301 [Note] IPv6 is available. 2015-11-03 17:53:24 28301 [Note] - ‘::’ resolves to ‘::’; 2015-11-03 17:53:24 28301 [Note] Server socket created on IP: ‘::’. 2015-11-03 17:53:24 28301 [ERROR] /usr/sbin/mysqld: Table ‘./mysql/user’ is marked as crashed and should be repaired 2015-11-03 17:53:24 28301 [Warning] Checking table: ‘./mysql/user’ 2015-11-03 17:53:24 28301 [ERROR] 1 client is using or hasn’t closed the table properly 2015-11-03 17:53:24 28301 [ERROR] /usr/sbin/mysqld: Table ‘./mysql/db’ is marked as crashed and should be repaired 2015-11-03 17:53:24 28301 [Warning] Checking table: ‘./mysql/db’ 2015-11-03 17:53:24 28301 [ERROR] 1 client is using or hasn’t closed the table properly 2015-11-03 17:53:24 28301 [Warning] ‘user’ entry ‘root@percona-clust-001’ ignored in --skip-name-resolve mode. 2015-11-03 17:53:24 28301 [Warning] ‘proxies\_priv’ entry ‘@ root@percona-clust-001’ ignored in --skip-name-resolve mode. 2015-11-03 17:53:24 28301 [Note] Event Scheduler: Loaded 0 events 2015-11-03 17:53:24 28301 [Note] WSREP: Signalling provider to continue. 2015-11-03 17:53:24 28301 [Note] WSREP: Initialized wsrep sidno 2 2015-11-03 17:53:24 28301 [Note] WSREP: SST received: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359125 2015-11-03 17:53:24 28301 [Note] /usr/sbin/mysqld: ready for connections. Version: ‘5.6.26-74.0-56-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel74.0, Revision 624ef81, WSREP version 25.12, wsrep\_25.12 2015-11-03 17:53:24 28301 [Note] WSREP: 0.0 (PERCONA-CLUST-002): State transfer from 1.0 (PERCONA-CLUST-001) complete. 2015-11-03 17:53:24 28301 [Note] WSREP: Shifting JOINER → JOINED (TO: 51394795) 2015-11-03 17:54:59 28301 [Note] WSREP: Member 1.0 (PERCONA-CLUST-001) synced with group. 2015-11-03 17:55:00 28301 [Note] WSREP: Member 0.0 (PERCONA-CLUST-002) synced with group. 2015-11-03 17:55:00 28301 [Note] WSREP: Shifting JOINED → SYNCED (TO: 51398066) 2015-11-03 17:55:00 28301 [Note] WSREP: Synchronized with group, ready for connections 2015-11-03 17:55:00 28301 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 2015-11-04 16:47:28 28301 [ERROR] /usr/sbin/mysqld: Table ‘./mysql/proc’ is marked as crashed and should be repaired 2015-11-04 16:47:28 28301 [Warning] Checking table: ‘./mysql/proc’