Not the answer you need?
Register and ask your own question!

Some Help with Percona Cluster please

Lee232Lee232 EntrantInactive User Role Beginner
Hi Guys
I seem to have an issue with one of my nodes in a cluster, it wouldn’t start then it started to replicate and I am not sure if it is actually running now.
So firstly this node is
mysql --version
mysql Ver 14.14 Distrib 5.6.26-74.0, for debian-linux-gnu (x86_64) using readline 6.3

The other nodes are all
mysql Ver 14.14 Distrib 5.6.15, for Linux (x86_64) using EditLine wrapper

When I look at the node I see

service mysql status
* MySQL (Percona XtraDB Cluster) is stopped. Check log

But when I check with

ps -ef | grep mysql
ldadmin 15010 14537 0 10:25 pts/0 00:00:00 grep --color=auto mysql
root 27595 1 0 Nov03 pts/1 00:00:00 /bin/sh /usr/bin/mysqld_safe
mysql 28301 27595 3 Nov03 pts/1 00:33:42 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera_smm.so --log-error=/var/log/mysql/mysql-error.log --open-files-limit=65535 --pid-file=/var/lib/mysql/mysql.pid --socket=/var/lib/mysql/mysql.sock --wsrep_start_position=d4497934-60d2-11e5-b80b-f6ba089aaa65:51357276

anyone able to shed any light on this.

thanks

Comments

  • jriverajrivera Percona Support Engineer Percona Staff Role
    Send us output of the following:

    > dpkg -l | grep -i percona

    How did you install the packages and can you paste last 50 lines from the error log?
  • Lee232Lee232 Entrant Inactive User Role Beginner
    Hi and thank you for the reply.
    I installed it from apt-get

    dpkg -l | grep -i percona
    ii percona-xtrabackup 2.2.12-1.trusty amd64 Open source backup tool for InnoDB and XtraDB
    ii percona-xtradb-cluster-56 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster with Galera
    ii percona-xtradb-cluster-client-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database client binaries
    ii percona-xtradb-cluster-common-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
    ii percona-xtradb-cluster-galera-3 3.12.2-1.trusty amd64 Metapackage for latest version of galera3.
    ii percona-xtradb-cluster-galera-3.x 3.12.2-1.trusty amd64 Galera components of Percona XtraDB Cluster
    ii percona-xtradb-cluster-server-5.6 5.6.26-25.12-1.trusty amd64 Percona XtraDB Cluster database server binaries
  • Lee232Lee232 Entrant Inactive User Role Beginner
    Forgot the error log. So it does look like its working, I think the see if the service is running daemon just didn’t wait long enough before giving up on it.

    2015-11-03 17:34:33 28301 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 537a7737-81fd-11e5-8cc7-f3f60367334b
    2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: sent state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b
    2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 0 (PERCONA-CLUST-002)
    2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 1 (PERCONA-CLUST-001)
    2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 2 (PERCONA-CLUST-004)
    2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 3 (PERCONA-CLUST-005)
    2015-11-03 17:34:33 28301 [Note] WSREP: STATE EXCHANGE: got state msg: 537a7737-81fd-11e5-8cc7-f3f60367334b from 4 (PERCONA-CLUST-003)
    2015-11-03 17:34:33 28301 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 169,
    members = 4/5 (joined/total),
    act_id = 51359073,
    last_appl. = -1,
    protocols = 0/5/2 (gcs/repl/appl),
    group UUID = d4497934-60d2-11e5-b80b-f6ba089aaa65
    2015-11-03 17:34:33 28301 [Note] WSREP: Flow-control interval: [36, 36]
    2015-11-03 17:34:33 28301 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 51359073)
    2015-11-03 17:34:33 28301 [Note] WSREP: State transfer required:
    Group state: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359073
    Local state: 00000000-0000-0000-0000-000000000000:-1
    2015-11-03 17:34:33 28301 [Note] WSREP: New cluster view: global state: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359073, view# 170: Primary, number of nodes: 5, my index: 0, protocol version 2
    2015-11-03 17:34:33 28301 [Note] WSREP: closing client connections for protocol change 3 -> 2
    2015-11-03 17:34:35 28301 [Note] WSREP: (529507d5, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2015-11-03 17:34:35 28301 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2015-11-03 17:34:35 28301 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.150.4.31' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '28301' '' '
    2015-11-03 17:34:35 28301 [Note] WSREP: Prepared SST request: rsync|10.150.4.31:4444/rsync_sst
    2015-11-03 17:34:35 28301 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2015-11-03 17:34:35 28301 [Note] WSREP: REPL Protocols: 5 (3, 1)
    2015-11-03 17:34:35 28301 [Note] WSREP: Service thread queue flushed.
    2015-11-03 17:34:35 28301 [Note] WSREP: Assign initial position for certification: 51359073, protocol version: 3
    2015-11-03 17:34:35 28301 [Note] WSREP: Service thread queue flushed.
    2015-11-03 17:34:35 28301 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (d4497934-60d2-11e5-b80b-f6ba089aaa65): 1 (Operation not permitted)
    at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
    2015-11-03 17:34:35 28301 [Note] WSREP: Member 0.0 (PERCONA-CLUST-002) requested state transfer from '*any*'. Selected 1.0 (PERCONA-CLUST-001)(SYNCED) as donor.
    2015-11-03 17:34:35 28301 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 51359124)
    2015-11-03 17:34:35 28301 [Note] WSREP: Requesting state transfer: success, donor: 1
    2015-11-03 17:53:10 28301 [Note] WSREP: 1.0 (PERCONA-CLUST-001): State transfer to 0.0 (PERCONA-CLUST-002) complete.
    WSREP_SST: [INFO] Joiner cleanup. rsync PID: 28340 (20151103 17:53:11.492)
    WSREP_SST: [INFO] Joiner cleanup done. (20151103 17:53:12.006)
    2015-11-03 17:53:12 28301 [Note] WSREP: SST complete, seqno: 51359125
    2015-11-03 17:53:12 28301 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
    2015-11-03 17:53:12 28301 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release
    2015-11-03 17:53:12 28301 [Note] Plugin 'FEDERATED' is disabled.
    2015-11-03 17:53:12 28301 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2015-11-03 17:53:12 28301 [Note] InnoDB: The InnoDB memory heap is disabled
    2015-11-03 17:53:12 28301 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2015-11-03 17:53:12 28301 [Note] InnoDB: Memory barrier is not used
    2015-11-03 17:53:12 28301 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2015-11-03 17:53:12 28301 [Note] InnoDB: Using Linux native AIO
    2015-11-03 17:53:12 28301 [Note] InnoDB: Using CPU crc32 instructions
    2015-11-03 17:53:12 28301 [Note] InnoDB: Initializing buffer pool, size = 6.0G
    2015-11-03 17:53:12 28301 [Note] InnoDB: Completed initialization of buffer pool
    2015-11-03 17:53:12 28301 [Note] InnoDB: Highest supported file format is Barracuda.
    2015-11-03 17:53:12 28301 [Note] InnoDB: Log scan progressed past the checkpoint lsn 612983148972
    2015-11-03 17:53:12 28301 [Note] InnoDB: Database was not shutdown normally!
    2015-11-03 17:53:12 28301 [Note] InnoDB: Starting crash recovery.
    2015-11-03 17:53:12 28301 [Note] InnoDB: Reading tablespace information from the .ibd files...
    2015-11-03 17:53:16 28301 [Note] InnoDB: Restoring possible half-written data pages
    2015-11-03 17:53:16 28301 [Note] InnoDB: from the doublewrite buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 612988391424
    InnoDB: Doing recovery: scanned up to log sequence number 612992207621
    2015-11-03 17:53:17 28301 [Note] InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
    InnoDB: Apply batch completed
    2015-11-03 17:53:19 28301 [Warning] InnoDB: Resizing redo log from 2*3072 to 2*32768 pages, LSN=612992207621
    2015-11-03 17:53:19 28301 [Warning] InnoDB: Starting to delete and rewrite log files.
    2015-11-03 17:53:19 28301 [Note] InnoDB: Setting log file ./ib_logfile101 size to 512 MB
    InnoDB: Progress in MB: 100 200 300 400 500
    2015-11-03 17:53:21 28301 [Note] InnoDB: Setting log file ./ib_logfile1 size to 512 MB
    InnoDB: Progress in MB: 100 200 300 400 500
    2015-11-03 17:53:24 28301 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
    2015-11-03 17:53:24 28301 [Warning] InnoDB: New log files created, LSN=612992207884
    2015-11-03 17:53:24 28301 [Note] InnoDB: 128 rollback segment(s) are active.
    2015-11-03 17:53:24 28301 [Note] InnoDB: Waiting for purge to start
    2015-11-03 17:53:24 28301 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 612992207621
    2015-11-03 17:53:24 28301 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
    2015-11-03 17:53:24 28301 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
    2015-11-03 17:53:24 28301 [Note] Server hostname (bind-address): '*'; port: 3306
    2015-11-03 17:53:24 28301 [Note] IPv6 is available.
    2015-11-03 17:53:24 28301 [Note] - '::' resolves to '::';
    2015-11-03 17:53:24 28301 [Note] Server socket created on IP: '::'.
    2015-11-03 17:53:24 28301 [ERROR] /usr/sbin/mysqld: Table './mysql/user' is marked as crashed and should be repaired
    2015-11-03 17:53:24 28301 [Warning] Checking table: './mysql/user'
    2015-11-03 17:53:24 28301 [ERROR] 1 client is using or hasn't closed the table properly
    2015-11-03 17:53:24 28301 [ERROR] /usr/sbin/mysqld: Table './mysql/db' is marked as crashed and should be repaired
    2015-11-03 17:53:24 28301 [Warning] Checking table: './mysql/db'
    2015-11-03 17:53:24 28301 [ERROR] 1 client is using or hasn't closed the table properly
    2015-11-03 17:53:24 28301 [Warning] 'user' entry '[email protected]' ignored in --skip-name-resolve mode.
    2015-11-03 17:53:24 28301 [Warning] 'proxies_priv' entry '@ [email protected]' ignored in --skip-name-resolve mode.
    2015-11-03 17:53:24 28301 [Note] Event Scheduler: Loaded 0 events
    2015-11-03 17:53:24 28301 [Note] WSREP: Signalling provider to continue.
    2015-11-03 17:53:24 28301 [Note] WSREP: Initialized wsrep sidno 2
    2015-11-03 17:53:24 28301 [Note] WSREP: SST received: d4497934-60d2-11e5-b80b-f6ba089aaa65:51359125
    2015-11-03 17:53:24 28301 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.6.26-74.0-56-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel74.0, Revision 624ef81, WSREP version 25.12, wsrep_25.12
    2015-11-03 17:53:24 28301 [Note] WSREP: 0.0 (PERCONA-CLUST-002): State transfer from 1.0 (PERCONA-CLUST-001) complete.
    2015-11-03 17:53:24 28301 [Note] WSREP: Shifting JOINER -> JOINED (TO: 51394795)
    2015-11-03 17:54:59 28301 [Note] WSREP: Member 1.0 (PERCONA-CLUST-001) synced with group.
    2015-11-03 17:55:00 28301 [Note] WSREP: Member 0.0 (PERCONA-CLUST-002) synced with group.
    2015-11-03 17:55:00 28301 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 51398066)
    2015-11-03 17:55:00 28301 [Note] WSREP: Synchronized with group, ready for connections
    2015-11-03 17:55:00 28301 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2015-11-04 16:47:28 28301 [ERROR] /usr/sbin/mysqld: Table './mysql/proc' is marked as crashed and should be repaired
    2015-11-04 16:47:28 28301 [Warning] Checking table: './mysql/proc'
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.