xtrabackup failing sst (donor can't find binlog)

MySQL & MariaDB Percona XtraBackup
#1

Trying to create a cluster, the second and third nodes won’t join if I use xtrabackup-v2. If I use rsync it works fine.
On the donor, after the tables are streamed, there is an error about not being able to find a bin log, and then it all shuts down.

-innobackupex-backup: [00] error: cannot open file /mysql_datadir//mariadb-bin.000363

Here are the donor logs:

...
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] Streaming ./mysql/help_relation.frm to <STDOUT>
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] ...done
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] Streaming ./mysql/time_zone_transition.frm to <STDOUT>
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] ...done
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] Streaming ./mysql/time_zone.frm to <STDOUT>
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] ...done
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] Streaming ./plugin/db.opt to <STDOUT>
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [01] ...done
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 Finished backing up non-InnoDB tables and files
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [00] Streaming xtrabackup_galera_info
Mar 15 09:26:32 avc2-wus1-db08 -innobackupex-backup: 160315 09:26:32 [00] ...done
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: Operating system error number 2 in a file operation.
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: The error means the system cannot find the path specified.
Mar 15 09:26:33 avc2-wus1-db08[B] -innobackupex-backup: [00] error: cannot open file /mysql_datadir//mariadb-bin.000363[/B]
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: Operating system error number 9 in a file operation.
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: Error number 9 means 'Bad file descriptor'
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: File (unknown): 'close' returned OS error 109. Cannot continue operation
Mar 15 09:26:33 avc2-wus1-db08 -innobackupex-backup: InnoDB: Cannot continue operation.
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Warning] Aborted connection 45 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: resuming provider at 11
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Provider resumed.
Mar 15 09:26:33 avc2-wus1-db08 -wsrep-sst-donor: innobackupex finished with error: 3. Check /mysql_datadir//innobackup.backup.log
Mar 15 09:26:33 avc2-wus1-db08 -wsrep-sst-donor: Cleanup after exit with status:22
Mar 15 09:26:33 avc2-wus1-db08 -wsrep-sst-donor: Cleaning up temporary directories
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.150.28.26:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/mysql_datadir/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --binlog '/var/log/mysql/mariadb-bin' --gtid '37178575-308f-11e5-a8b2-6af2fcef5c43:1397289'
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.150.28.26:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/mysql_datadir/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --binlog '/var/log/mysql/mariadb-bin' --gtid '37178575-308f-11e5-a8b2-6af2fcef5c43:1397289': 22 (Invalid argument)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.150.28.26:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/mysql_datadir/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --binlog '/var/log/mysql/mariadb-bin' --gtid '37178575-308f-11e5-a8b2-6af2fcef5c43:1397289'
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Warning] WSREP: 0.0 (avc2-wus1-db08): State transfer to 1.0 (avc2-wus1-db09) failed: -22 (Invalid argument)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 1397290)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: declaring a88f0db7 at tcp://10.150.28.24:4567 stable
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: forgetting 9f75855a (tcp://10.150.28.26:4567)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Member 0.0 (avc2-wus1-db08) synced with group.
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 1397290)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Synchronized with group, ready for connections
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Node 51b6359d state prim
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: view(view_id(PRIM,51b6359d,26) memb {
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #01151b6359d,0
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011a88f0db7,0
Mar 15 09:26:33 avc2-wus1-db08 mysqld: } joined {
Mar 15 09:26:33 avc2-wus1-db08 mysqld: } left {
Mar 15 09:26:33 avc2-wus1-db08 mysqld: } partitioned {
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #0119f75855a,0
Mar 15 09:26:33 avc2-wus1-db08 mysqld: })
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: save pc into disk
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: forgetting 9f75855a (tcp://10.150.28.26:4567)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: STATE_EXCHANGE: sent state UUID: ae2b229f-eaca-11e5-b86f-8e27339ebc6b
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: STATE EXCHANGE: sent state msg: ae2b229f-eaca-11e5-b86f-8e27339ebc6b
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: STATE EXCHANGE: got state msg: ae2b229f-eaca-11e5-b86f-8e27339ebc6b from 0 (avc2-wus1-db08)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: STATE EXCHANGE: got state msg: ae2b229f-eaca-11e5-b86f-8e27339ebc6b from 1 (avc2-wus1-db07)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Quorum results:
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011version = 3,
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011component = PRIMARY,
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011conf_id = 23,
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011members = 2/2 (joined/total),
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011act_id = 1397290,
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011last_appl. = 0,
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011protocols = 0/7/3 (gcs/repl/appl),
Mar 15 09:26:33 avc2-wus1-db08 mysqld: #011group UUID = 37178575-308f-11e5-a8b2-6af2fcef5c43
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Flow-control interval: [23, 23]
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: New cluster view: global state: 37178575-308f-11e5-a8b2-6af2fcef5c43:1397290, view# 24: Primary, number of nodes: 2, my index: 0, protocol version 3
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: REPL Protocols: 7 (3, 2)
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Service thread queue flushed.
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Assign initial position for certification: 1397290, protocol version: 3
Mar 15 09:26:33 avc2-wus1-db08 mysqld: 160315 9:26:33 [Note] WSREP: Service thread queue flushed.
Mar 15 09:26:38 avc2-wus1-db08 mysqld: 160315 9:26:38 [Note] WSREP: cleaning up 9f75855a (tcp://10.150.28.26:4567)

The bin logs are actually located in /var/log/mysql/
How do I get xtrabackup to look there instead?

#2

The joiner reports invalid argument, then shuts down:

...
Mar 15 09:26:08 avc2-wus1-db09 mysqld: 160315 9:26:08 [Note] WSREP: save pc into disk
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: gcomm: connected
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Opened channel 'dev_cluster'
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Waiting for SST to complete.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: STATE EXCHANGE: sent state msg: 9fc40048-eaca-11e5-b1b5-6a90c2c3dd39
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: STATE EXCHANGE: got state msg: 9fc40048-eaca-11e5-b1b5-6a90c2c3dd39 from 0 (avc2-wus1-db08)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: STATE EXCHANGE: got state msg: 9fc40048-eaca-11e5-b1b5-6a90c2c3dd39 from 2 (avc2-wus1-db07)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: STATE EXCHANGE: got state msg: 9fc40048-eaca-11e5-b1b5-6a90c2c3dd39 from 1 (avc2-wus1-db09)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Quorum results:
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011version = 3,
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011component = PRIMARY,
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011conf_id = 22,
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011members = 2/3 (joined/total),
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011act_id = 1397289,
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011last_appl. = -1,
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011protocols = 0/7/3 (gcs/repl/appl),
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011group UUID = 37178575-308f-11e5-a8b2-6af2fcef5c43
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Flow-control interval: [28, 28]
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1397289)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: State transfer required:
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011Group state: 37178575-308f-11e5-a8b2-6af2fcef5c43:1397289
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: New cluster view: global state: 37178575-308f-11e5-a8b2-6af2fcef5c43:1397289, view# 23: Primary, number of nodes: 3, my index: 1, protocol version 3
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Warning] WSREP: Gap in state sequence. Need state transfer.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.150.28.26' --datadir '/mysql_datadir/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '34662' --binlog '/var/log/mysql/mariadb-bin' '
Mar 15 09:26:09 avc2-wus1-db09 mysqld: WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20160315 09:26:09.610)
Mar 15 09:26:09 avc2-wus1-db09 -wsrep-sst-joiner: Streaming with xbstream
Mar 15 09:26:09 avc2-wus1-db09 -wsrep-sst-joiner: Using socat as streamer
Mar 15 09:26:09 avc2-wus1-db09 -wsrep-sst-joiner: Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[&#64;]} )
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.150.28.26:4444/xtrabackup_sst//1
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: REPL Protocols: 7 (3, 2)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Service thread queue flushed.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Assign initial position for certification: 1397289, protocol version: 3
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Service thread queue flushed.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (37178575-308f-11e5-a8b2-6af2fcef5c43): 1 (Operation not permitted)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Member 1.0 (avc2-wus1-db09) requested state transfer from '*any*'. Selected 0.0 (avc2-wus1-db08)(SYNCED) as donor.
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1397289)
Mar 15 09:26:09 avc2-wus1-db09 mysqld: 160315 9:26:09 [Note] WSREP: Requesting state transfer: success, donor: 0
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: Proceeding with SST
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[&#64;]} )
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: Cleaning the existing datadir and innodb-data/log directories
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: removed â/mysql_datadir/aria_log_controlâ
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: removed â/mysql_datadir/aria_log.00000001â
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: removed â/mysql_datadir/ibdata1â
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: removed â/mysql_logdir/ib_logfile1â
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: removed â/mysql_logdir/ib_logfile0â
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: Cleaning the binlog directory /var/log/mysql as well
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: removed â/var/log/mysql/mariadb-bin.000001â
Mar 15 09:26:10 avc2-wus1-db09 -wsrep-sst-joiner: Waiting for SST streaming to complete!
Mar 15 09:26:11 avc2-wus1-db09 mysqld: 160315 9:26:11 [Note] WSREP: (9f75855a, 'tcp://0.0.0.0:4567') turning message relay requesting off
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Warning] WSREP: 0.0 (avc2-wus1-db08): State transfer to 1.0 (avc2-wus1-db09) failed: -22 (Invalid argument)
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: gcomm: terminating thread
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: gcomm: joining thread
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: gcomm: closing backend
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: view(view_id(NON_PRIM,51b6359d,25) memb {
Mar 15 09:26:33 avc2-wus1-db09 mysqld: #0119f75855a,0
Mar 15 09:26:33 avc2-wus1-db09 mysqld: } joined {
Mar 15 09:26:33 avc2-wus1-db09 mysqld: } left {
Mar 15 09:26:33 avc2-wus1-db09 mysqld: } partitioned {
Mar 15 09:26:33 avc2-wus1-db09 mysqld: #01151b6359d,0
Mar 15 09:26:33 avc2-wus1-db09 mysqld: #011a88f0db7,0
Mar 15 09:26:33 avc2-wus1-db09 mysqld: })
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: view((empty))
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: gcomm: closed
Mar 15 09:26:33 avc2-wus1-db09 mysqld: 160315 9:26:33 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Mar 15 09:26:35 avc2-wus1-db09 -wsrep-sst-joiner: xtrabackup_checkpoints missing, failed innobackupex/SST on donor
Mar 15 09:26:35 avc2-wus1-db09 -wsrep-sst-joiner: Cleanup after exit with status:2
Mar 15 09:26:35 avc2-wus1-db09 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
M
#3

I just added the [sst] section to my.cnf file and added option sst_special_dirs = 1 to both the donor and the joiner, but that didn’t seem to help.
Here is my my.cnf file, which is the same for both the joiner and donor except for the local IPs in the cluster settings:


[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[sst]
option sst_special_dirs = 1

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /mysql_datadir
tmpdir = /run/mysqld
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
skip_name_resolve
default-time-zone='+0:00'
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 0.0.0.0
#
# * Fine Tuning
#
max_connections = 1000
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 1280
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover = BACKUP
key_buffer_size = 18M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 52M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit = 128K
query_cache_size = 0
query_cache_type = OFF
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
log_warnings = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[={0|1}]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
#log_slow_rate_limit = 1000
log_slow_verbosity = query_plan

#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#report_host = master1
#auto_increment_increment = 2
#auto_increment_offset = 1
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog = 1
expire_logs_days = 10
max_binlog_size = 100M
binlog_format = ROW

# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine = InnoDB
# you can't just change log file size, requires special procedure
innodb_log_file_size = 512M
innodb_buffer_pool_size = 10G
innodb_log_buffer_size = 128M
innodb_file_per_table = 1
innodb_open_files = 4000
innodb_io_capacity = 800
innodb_flush_method = O_DIRECT

#performance settings
############################
#limit the number of threads, so we don't thrash around
innodb_thread_concurrency = 32
innodb_doublewrite = 0
innodb_use_fallocate = 1
innodb_buffer_pool_instances = 10
innodb_write_io_threads = 32
innodb_read_io_threads = 32
innodb_file_format = Barracuda
innodb_lru_scan_depth = 2000
innodb_log_group_home_dir = /mysql_logdir
innodb_data_home_dir = /mysql_datadir



#clustering settings
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 0
table_definition_cache = 1000
performance_schema = 1

########################### cluster
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_provider_options = "gmcast.segment=0; gcache.size=1G; evs.suspect_timeout = PT90S; evs.inactive_timeout = PT5M; evs.install_timeout = PT5M; evs.join_retrans_period=PT1.0S"
wsrep_cluster_name = "dev_cluster"
wsrep_cluster_address = "gcomm://10.150.28.24,10.150.28.25"
wsrep_restart_slave = 1
wsrep_sst_method = xtrabackup-v2
wsrep_sst_auth = root:************

#our node address
wsrep_node_address = "10.150.28.26"
########################### cluster


[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

Any help solving this would be great!

Thanks,
​Aaron

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2021 Percona LLC. All rights reserved.