root owner for all files in mysql_datadir on second node

MySQL & MariaDB Percona XtraDB Cluster 5.x
#1

Hi!
I have installed percona-xtradb-cluster-56 (5.6.25-25.12-1.jessie) on the fresh copies of debian 8.2v.
When i started first node with the command:

/etc/init.d/mysql bootstrap-pxc

Everything is ok and bootstrap the cluster have done.

But when i tried to start the second node, i see this errors in log file:

2015-09-28 14:19:48 9432 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode
2015-09-28 14:19:48 9432 [ERROR] InnoDB: The system tablespace must be writable!
2015-09-28 14:19:48 9432 [ERROR] Plugin 'InnoDB' init function returned error.
2015-09-28 14:19:48 9432 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-09-28 14:19:48 9432 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-09-28 14:19:48 9432 [ERROR] Aborting

And i see, that owner on files in mysql_data_dir is root, but it must be mysql


-rw------- 1 mysql mysql 129M Sep 28 16:33 galera.cache
-rw-rw---- 1 mysql mysql 104 Sep 28 16:33 grastate.dat
-rw-rw---- 1 root root 48M Sep 28 16:34 ib_logfile0
-rw-rw---- 1 root root 48M Sep 28 16:34 ib_logfile1
-rw-rw---- 1 root root 12M Sep 28 16:34 ibdata1
-rw-rw---- 1 root root 20K Sep 28 16:34 innobackup.move.log
-rw-rw---- 1 root root 5.3K Sep 28 16:34 innobackup.prepare.log
drwxrwx--x 2 root root 4.0K Sep 28 16:34 mysql
-rw-r----- 1 mysql mysql 6.9K Sep 28 15:26 mysql-cluster-02.err
drwxrwx--x 2 root root 4.0K Sep 28 16:34 performance_schema
drwxrwx--x 2 root root 19 Sep 28 16:34 test
-rw-rw---- 1 root root 38 Sep 28 16:34 xtrabackup_galera_info
-rw-rw---- 1 root root 671 Sep 28 16:34 xtrabackup_info

and during the sst process i see that all files in temporary dir has root as owner

/var/lib/mysql# ls -lah .sst/
total 126M
drwxrwx--x 5 root root 4.0K Sep 28 17:12 .
drwx------ 3 mysql mysql 4.0K Sep 28 17:12 ..
-rw-rw---- 1 root root 358 Sep 28 17:12 backup-my.cnf
-rw-rw---- 1 root root 48M Sep 28 17:12 ib_logfile1
-rw-rw---- 1 root root 48M Sep 28 17:12 ib_logfile101
-rw-rw---- 1 root root 12M Sep 28 17:12 ibdata1
-rw-rw---- 1 root root 4.7K Sep 28 17:12 innobackup.prepare.log
drwx------ 2 root root 4.0K Sep 28 17:12 mysql
drwx------ 2 root root 4.0K Sep 28 17:12 performance_schema
drwx------ 2 root root 19 Sep 28 17:12 test
-rw-rw---- 1 root root 89 Sep 28 17:12 xtrabackup_checkpoints
-rw-rw---- 1 root root 38 Sep 28 17:12 xtrabackup_galera_info
-rw-rw---- 1 root root 671 Sep 28 17:12 xtrabackup_info
-rw-rw---- 1 root root 2.0M Sep 28 17:12 xtrabackup_logfile

Please, anybody, tell me how can i fix it?

Some lines from my.cnf on both nodes:

[mysqld]
user = mysql
datadir = /var/lib/mysql

# Authentication for SST method
wsrep_sst_auth="cluster:password"

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# SST method
wsrep_sst_method=xtrabackup-v2

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
#2

Dear Community. Please help me with my problem. I suffer second day. Just drop a hint at list…

#3

Have you helped anyone?

#4

Can you paste here the full error log section when you start MySQL on the second node until the end.

#5

rolls, revin, thank you for your answers!!!

This is mysq log.

Part 1:


2015-09-29 16:21:55 0 [Note] /usr/sbin/mysqld (mysqld 5.6.25-73.1-56) starting as process 12720 ...
2015-09-29 16:21:55 12720 [Note] WSREP: Read nil XID from storage engines, skipping position init
2015-09-29 16:21:55 12720 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
2015-09-29 16:21:55 12720 [Note] WSREP: wsrep_load(): Galera 3.12(r9921e73) by Codership Oy <info&#64;codership.com> loaded successfully.
2015-09-29 16:21:55 12720 [Note] WSREP: CRC-32C: using hardware acceleration.
2015-09-29 16:21:55 12720 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2015-09-29 16:21:55 12720 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.3.48; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum =
2015-09-29 16:21:55 12720 [Note] WSREP: Service thread queue flushed.
2015-09-29 16:21:55 12720 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2015-09-29 16:21:55 12720 [Note] WSREP: wsrep_sst_grab()
2015-09-29 16:21:55 12720 [Note] WSREP: Start replication
2015-09-29 16:21:55 12720 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2015-09-29 16:21:55 12720 [Note] WSREP: protonet asio version 0
2015-09-29 16:21:55 12720 [Note] WSREP: Using CRC-32C for message checksums.
2015-09-29 16:21:55 12720 [Note] WSREP: backend: asio
2015-09-29 16:21:55 12720 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2015-09-29 16:21:55 12720 [Note] WSREP: restore pc from disk failed
2015-09-29 16:21:55 12720 [Note] WSREP: GMCast version 0
2015-09-29 16:21:55 12720 [Note] WSREP: (0dd2ccfa, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2015-09-29 16:21:55 12720 [Note] WSREP: (0dd2ccfa, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2015-09-29 16:21:55 12720 [Note] WSREP: EVS version 0
2015-09-29 16:21:55 12720 [Note] WSREP: gcomm: connecting to group 'group', peer '192.168.3.49:,192.168.3.48:'
2015-09-29 16:21:55 12720 [Warning] WSREP: (0dd2ccfa, 'tcp://0.0.0.0:4567') address 'tcp://192.168.3.48:4567' points to own listening address, blacklisting
2015-09-29 16:21:55 12720 [Note] WSREP: (0dd2ccfa, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2015-09-29 16:21:55 12720 [Note] WSREP: declaring 6f1395d2 at tcp://192.168.3.49:4567 stable
2015-09-29 16:21:55 12720 [Note] WSREP: Node 6f1395d2 state prim
2015-09-29 16:21:55 12720 [Note] WSREP: view(view_id(PRIM,0dd2ccfa,4) memb {
0dd2ccfa,0
6f1395d2,0
} joined {
} left {
} partitioned {
})
2015-09-29 16:21:55 12720 [Note] WSREP: save pc into disk
2015-09-29 16:21:55 12720 [Note] WSREP: gcomm: connected
2015-09-29 16:21:55 12720 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2015-09-29 16:21:55 12720 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2015-09-29 16:21:55 12720 [Note] WSREP: Opened channel 'group'
2015-09-29 16:21:55 12720 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2015-09-29 16:21:55 12720 [Note] WSREP: Waiting for SST to complete.
2015-09-29 16:21:55 12720 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0e22d11e-66ad-11e5-8bab-fa1f2d1cf12b
2015-09-29 16:21:55 12720 [Note] WSREP: STATE EXCHANGE: sent state msg: 0e22d11e-66ad-11e5-8bab-fa1f2d1cf12b
2015-09-29 16:21:55 12720 [Note] WSREP: STATE EXCHANGE: got state msg: 0e22d11e-66ad-11e5-8bab-fa1f2d1cf12b from 1 (mysql-cluster-01)
2015-09-29 16:21:55 12720 [Note] WSREP: STATE EXCHANGE: got state msg: 0e22d11e-66ad-11e5-8bab-fa1f2d1cf12b from 0 (mysql-cluster-02)
2015-09-29 16:21:55 12720 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 3,
members = 1/2 (joined/total),
act_id = 3,
last_appl. = -1,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 662ed9a7-66a0-11e5-a831-fa74c0e8b135
2015-09-29 16:21:55 12720 [Note] WSREP: Flow-control interval: [23, 23]
2015-09-29 16:21:55 12720 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 3)
2015-09-29 16:21:55 12720 [Note] WSREP: State transfer required: 
Group state: 662ed9a7-66a0-11e5-a831-fa74c0e8b135:3
Local state: 00000000-0000-0000-0000-000000000000:-1
2015-09-29 16:21:55 12720 [Note] WSREP: New cluster view: global state: 662ed9a7-66a0-11e5-a831-fa74c0e8b135:3, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3
2015-09-29 16:21:55 12720 [Warning] WSREP: Gap in state sequence. Need state transfer.
2015-09-29 16:21:55 12720 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.3.48' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '12720' '' '
WSREP_SST: [INFO] Streaming with xbstream (20150929 16:21:56.384)
WSREP_SST: [INFO] Using socat as streamer (20150929 16:21:56.391)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[&#64;]} ) (20150929 16:21:56.448)
2015-09-29 16:21:56 12720 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.3.48:4444/xtrabackup_sst//1
2015-09-29 16:21:56 12720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-09-29 16:21:56 12720 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-09-29 16:21:56 12720 [Note] WSREP: Service thread queue flushed.
2015-09-29 16:21:56 12720 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
2015-09-29 16:21:56 12720 [Note] WSREP: Service thread queue flushed.
2015-09-29 16:21:56 12720 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (662ed9a7-66a0-11e5-a831-fa74c0e8b135): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2015-09-29 16:21:56 12720 [Note] WSREP: Member 0.0 (mysql-cluster-02) requested state transfer from '*any*'. Selected 1.0 (mysql-cluster-01)(SYNCED) as donor.
2015-09-29 16:21:56 12720 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3)
2015-09-29 16:21:56 12720 [Note] WSREP: Requesting state transfer: success, donor: 1
WSREP_SST: [INFO] Proceeding with SST (20150929 16:21:58.683)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[&#64;]} ) (20150929 16:21:58.684)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20150929 16:21:58.690)
2015-09-29 16:21:58 12720 [Note] WSREP: (0dd2ccfa, 'tcp://0.0.0.0:4567') turning message relay requesting off
#6

Part 2:


removed '/var/lib/mysql/ibdata1'
removed '/var/lib/mysql/xtrabackup_info'
removed '/var/lib/mysql/mysql/tables_priv.frm'
removed '/var/lib/mysql/mysql/db.frm'
removed '/var/lib/mysql/mysql/time_zone_name.frm'
removed '/var/lib/mysql/mysql/proc.MYI'
removed '/var/lib/mysql/mysql/help_topic.MYD'
removed '/var/lib/mysql/mysql/time_zone.MYD'
removed '/var/lib/mysql/mysql/help_topic.MYI'
removed '/var/lib/mysql/mysql/time_zone_leap_second.MYD'
removed '/var/lib/mysql/mysql/func.MYD'
removed '/var/lib/mysql/mysql/time_zone.MYI'
removed '/var/lib/mysql/mysql/slave_relay_log_info.frm'
removed '/var/lib/mysql/mysql/help_relation.frm'
removed '/var/lib/mysql/mysql/help_category.frm'
removed '/var/lib/mysql/mysql/func.MYI'
removed '/var/lib/mysql/mysql/help_relation.MYD'
removed '/var/lib/mysql/mysql/general_log.CSM'
removed '/var/lib/mysql/mysql/help_category.MYD'
removed '/var/lib/mysql/mysql/help_category.MYI'
removed '/var/lib/mysql/mysql/ndb_binlog_index.MYD'
removed '/var/lib/mysql/mysql/procs_priv.MYD'
removed '/var/lib/mysql/mysql/procs_priv.MYI'
removed '/var/lib/mysql/mysql/innodb_index_stats.frm'
removed '/var/lib/mysql/mysql/time_zone_transition_type.frm'
removed '/var/lib/mysql/mysql/servers.MYI'
removed '/var/lib/mysql/mysql/user.MYI'
removed '/var/lib/mysql/mysql/innodb_table_stats.ibd'
removed '/var/lib/mysql/mysql/event.frm'
removed '/var/lib/mysql/mysql/time_zone_leap_second.MYI'
removed '/var/lib/mysql/mysql/user.MYD'
removed '/var/lib/mysql/mysql/columns_priv.MYD'
removed '/var/lib/mysql/mysql/proxies_priv.MYD'
removed '/var/lib/mysql/mysql/time_zone_transition.frm'
removed '/var/lib/mysql/mysql/slow_log.CSV'
removed '/var/lib/mysql/mysql/servers.MYD'
removed '/var/lib/mysql/mysql/time_zone_transition_type.MYD'
removed '/var/lib/mysql/mysql/help_keyword.MYI'
removed '/var/lib/mysql/mysql/event.MYD'
removed '/var/lib/mysql/mysql/help_keyword.MYD'
removed '/var/lib/mysql/mysql/help_relation.MYI'
removed '/var/lib/mysql/mysql/proc.MYD'
removed '/var/lib/mysql/mysql/proc.frm'
removed '/var/lib/mysql/mysql/columns_priv.MYI'
removed '/var/lib/mysql/mysql/general_log.frm'
removed '/var/lib/mysql/mysql/user.frm'
removed '/var/lib/mysql/mysql/time_zone_leap_second.frm'
removed '/var/lib/mysql/mysql/func.frm'
removed '/var/lib/mysql/mysql/time_zone_name.MYD'
removed '/var/lib/mysql/mysql/slave_master_info.frm'
removed '/var/lib/mysql/mysql/columns_priv.frm'
removed '/var/lib/mysql/mysql/ndb_binlog_index.MYI'
removed '/var/lib/mysql/mysql/slave_worker_info.frm'
removed '/var/lib/mysql/mysql/tables_priv.MYI'
removed '/var/lib/mysql/mysql/slave_worker_info.ibd'
removed '/var/lib/mysql/mysql/proxies_priv.frm'
removed '/var/lib/mysql/mysql/slave_master_info.ibd'
removed '/var/lib/mysql/mysql/servers.frm'
removed '/var/lib/mysql/mysql/ndb_binlog_index.frm'
removed '/var/lib/mysql/mysql/procs_priv.frm'
removed '/var/lib/mysql/mysql/help_topic.frm'
removed '/var/lib/mysql/mysql/time_zone_transition_type.MYI'
removed '/var/lib/mysql/mysql/event.MYI'
removed '/var/lib/mysql/mysql/time_zone.frm'
removed '/var/lib/mysql/mysql/plugin.frm'
removed '/var/lib/mysql/mysql/slow_log.CSM'
removed '/var/lib/mysql/mysql/slave_relay_log_info.ibd'
removed '/var/lib/mysql/mysql/time_zone_transition.MYD'
removed '/var/lib/mysql/mysql/tables_priv.MYD'
removed '/var/lib/mysql/mysql/help_keyword.frm'
removed '/var/lib/mysql/mysql/time_zone_transition.MYI'
removed '/var/lib/mysql/mysql/innodb_table_stats.frm'
removed '/var/lib/mysql/mysql/time_zone_name.MYI'
removed '/var/lib/mysql/mysql/plugin.MYI'
removed '/var/lib/mysql/mysql/innodb_index_stats.ibd'
removed '/var/lib/mysql/mysql/db.MYI'
removed '/var/lib/mysql/mysql/db.MYD'
removed '/var/lib/mysql/mysql/plugin.MYD'
removed '/var/lib/mysql/mysql/proxies_priv.MYI'
removed '/var/lib/mysql/mysql/general_log.CSV'
removed '/var/lib/mysql/mysql/slow_log.frm'
removed directory: '/var/lib/mysql/mysql'
removed '/var/lib/mysql/performance_schema/events_statements_summary_global_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_statements_history.frm'
removed '/var/lib/mysql/performance_schema/host_cache.frm'
removed '/var/lib/mysql/performance_schema/events_stages_summary_by_account_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_stages_history_long.frm'
removed '/var/lib/mysql/performance_schema/events_stages_summary_by_host_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/file_instances.frm'
removed '/var/lib/mysql/performance_schema/events_waits_history.frm'
removed '/var/lib/mysql/performance_schema/setup_instruments.frm'
removed '/var/lib/mysql/performance_schema/file_summary_by_instance.frm'
removed '/var/lib/mysql/performance_schema/events_waits_summary_by_host_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/table_io_waits_summary_by_table.frm'
removed '/var/lib/mysql/performance_schema/socket_summary_by_instance.frm'
removed '/var/lib/mysql/performance_schema/events_waits_summary_by_account_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/session_connect_attrs.frm'
removed '/var/lib/mysql/performance_schema/setup_timers.frm'
removed '/var/lib/mysql/performance_schema/mutex_instances.frm'
removed '/var/lib/mysql/performance_schema/file_summary_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/socket_summary_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_statements_summary_by_thread_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_stages_summary_by_user_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/rwlock_instances.frm'
removed '/var/lib/mysql/performance_schema/table_lock_waits_summary_by_table.frm'
removed '/var/lib/mysql/performance_schema/events_waits_history_long.frm'
removed '/var/lib/mysql/performance_schema/table_io_waits_summary_by_index_usage.frm'
removed '/var/lib/mysql/performance_schema/setup_objects.frm'
removed '/var/lib/mysql/performance_schema/db.opt'
removed '/var/lib/mysql/performance_schema/threads.frm'
removed '/var/lib/mysql/performance_schema/session_account_connect_attrs.frm'
removed '/var/lib/mysql/performance_schema/setup_consumers.frm'
removed '/var/lib/mysql/performance_schema/hosts.frm'
removed '/var/lib/mysql/performance_schema/socket_instances.frm'
removed '/var/lib/mysql/performance_schema/events_waits_summary_global_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/users.frm'
removed '/var/lib/mysql/performance_schema/events_stages_summary_by_thread_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_statements_summary_by_account_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_statements_current.frm'
removed '/var/lib/mysql/performance_schema/events_stages_history.frm'
removed '/var/lib/mysql/performance_schema/accounts.frm'
removed '/var/lib/mysql/performance_schema/events_waits_summary_by_instance.frm'
removed '/var/lib/mysql/performance_schema/events_waits_summary_by_user_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/objects_summary_global_by_type.frm'
removed '/var/lib/mysql/performance_schema/performance_timers.frm'
removed '/var/lib/mysql/performance_schema/events_statements_summary_by_host_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_statements_history_long.frm'
removed '/var/lib/mysql/performance_schema/cond_instances.frm'
removed '/var/lib/mysql/performance_schema/events_waits_current.frm'
removed '/var/lib/mysql/performance_schema/setup_actors.frm'
removed '/var/lib/mysql/performance_schema/events_stages_current.frm'
removed '/var/lib/mysql/performance_schema/events_statements_summary_by_digest.frm'
removed '/var/lib/mysql/performance_schema/events_statements_summary_by_user_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_waits_summary_by_thread_by_event_name.frm'
removed '/var/lib/mysql/performance_schema/events_stages_summary_global_by_event_name.frm'
removed directory: '/var/lib/mysql/performance_schema'
removed '/var/lib/mysql/test/db.opt'
removed directory: '/var/lib/mysql/test'
removed '/var/lib/mysql/ib_logfile1'
removed '/var/lib/mysql/ib_logfile0'
#7

Part 3:


WSREP_SST: [INFO] Waiting for SST streaming to complete! (20150929 16:21:59.190)
2015-09-29 16:22:13 12720 [Note] WSREP: 1.0 (mysql-cluster-01): State transfer to 0.0 (mysql-cluster-02) complete.
2015-09-29 16:22:13 12720 [Note] WSREP: Member 1.0 (mysql-cluster-01) synced with group.
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20150929 16:22:13.568)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20150929 16:22:13.574)
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20150929 16:22:20.547)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20150929 16:22:20.553)
WSREP_SST: [INFO] Move successful, removing /var/lib/mysql//.sst (20150929 16:22:21.326)
WSREP_SST: [INFO] Galera co-ords from recovery: 662ed9a7-66a0-11e5-a831-fa74c0e8b135:3 (20150929 16:22:21.362)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20150929 16:22:21.370)
WSREP_SST: [INFO] Removing the sst_in_progress file (20150929 16:22:21.376)
2015-09-29 16:22:21 12720 [Note] WSREP: SST complete, seqno: 3
2015-09-29 16:22:21 12720 [Note] Plugin 'FEDERATED' is disabled.
/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13 - Permission denied)
2015-09-29 16:22:21 12720 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2015-09-29 16:22:21 12720 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-09-29 16:22:21 12720 [Note] InnoDB: The InnoDB memory heap is disabled
2015-09-29 16:22:21 12720 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-09-29 16:22:21 12720 [Note] InnoDB: Memory barrier is not used
2015-09-29 16:22:21 12720 [Note] InnoDB: Compressed tables use zlib 1.2.8
2015-09-29 16:22:21 12720 [Note] InnoDB: Using Linux native AIO
2015-09-29 16:22:21 12720 [Note] InnoDB: Using CPU crc32 instructions
2015-09-29 16:22:21 12720 [Note] InnoDB: Initializing buffer pool, size = 50.0G
2015-09-29 16:22:25 12720 [Note] InnoDB: Completed initialization of buffer pool
2015-09-29 16:22:26 12720 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode
2015-09-29 16:22:26 12720 [ERROR] InnoDB: The system tablespace must be writable!
2015-09-29 16:22:26 12720 [ERROR] Plugin 'InnoDB' init function returned error.
2015-09-29 16:22:26 12720 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-09-29 16:22:26 12720 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-09-29 16:22:26 12720 [ERROR] Aborting

2015-09-29 16:22:28 12720 [Note] WSREP: Closing send monitor...
2015-09-29 16:22:28 12720 [Note] WSREP: Closed send monitor.
2015-09-29 16:22:28 12720 [Note] WSREP: gcomm: terminating thread
2015-09-29 16:22:28 12720 [Note] WSREP: gcomm: joining thread
2015-09-29 16:22:28 12720 [Note] WSREP: gcomm: closing backend
2015-09-29 16:22:29 12720 [Note] WSREP: view(view_id(NON_PRIM,0dd2ccfa,4) memb {
0dd2ccfa,0
} joined {
} left {
} partitioned {
6f1395d2,0
})
2015-09-29 16:22:29 12720 [Note] WSREP: view((empty))
2015-09-29 16:22:29 12720 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-09-29 16:22:29 12720 [Note] WSREP: gcomm: closed
2015-09-29 16:22:29 12720 [Note] WSREP: Flow-control interval: [16, 16]
2015-09-29 16:22:29 12720 [Note] WSREP: Received NON-PRIMARY.
2015-09-29 16:22:29 12720 [Note] WSREP: Shifting JOINER -> OPEN (TO: 3)
2015-09-29 16:22:29 12720 [Note] WSREP: Received self-leave message.
2015-09-29 16:22:29 12720 [Note] WSREP: Flow-control interval: [0, 0]
2015-09-29 16:22:29 12720 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2015-09-29 16:22:29 12720 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 3)
2015-09-29 16:22:29 12720 [Note] WSREP: RECV thread exiting 0: Success
2015-09-29 16:22:29 12720 [Note] WSREP: recv_thread() joined.
2015-09-29 16:22:29 12720 [Note] WSREP: Closing replication queue.
2015-09-29 16:22:29 12720 [Note] WSREP: Closing slave action queue.
2015-09-29 16:22:29 12720 [Note] WSREP: Service disconnected.
2015-09-29 16:22:29 12720 [Note] WSREP: rollbacker thread exiting
2015-09-29 16:22:30 12720 [Note] WSREP: Some threads may fail to exit.
2015-09-29 16:22:30 12720 [Note] Binlog end
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'partition'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'ARCHIVE'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_METRICS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_CMP'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'INNODB_TRX'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'XTRADB_RSEG'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'BLACKHOLE'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'MyISAM'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'CSV'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'MEMORY'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'MRG_MYISAM'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'sha256_password'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'mysql_old_password'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'mysql_native_password'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'wsrep'
2015-09-29 16:22:30 12720 [Note] Shutting down plugin 'binlog'
2015-09-29 16:22:31 12720 [Note] /usr/sbin/mysqld: Shutdown complete
#8

Make mysql user owner of /var/lib/mysql with #chown -R mysql.mysql /var/lib/mysql

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2021 Percona LLC. All rights reserved.