XtraDB Cluster and Docker

Hi,

i want run XtraDB Cluster in docker (under Marathon/Messos). But i have problem if i want connect from docker container to exist cluster. If i connect to docker container from native node, everything is ok. Do you have any idea?

For example i have navite node BDB and DOCK. I want connect DOCK to exist cluster in BDB.

Error message from BDB /var/lib/mysql/innobackup.backup.log after start mysql in DOCK

...
innobackupex: Created backup directory /tmp/tmp.srM5t2EAga
^Gxbstream: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
innobackupex: 'xbstream -c' returned with exit code 1.
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 4882
main::backup_file_via_stream('/tmp', 'backup-my.cnf') called at /usr//bin/innobackupex line 4931
main::backup_file('/tmp', 'backup-my.cnf', '/tmp/backup-my.cnf') called at /usr//bin/innobackupex line 4955
main::write_to_backup_file('/tmp/backup-my.cnf', '# This MySQL options file was generated by innobackupex.\x{a}\x{a}# T...') called at /usr//bin/innobackupex line 3762
main::write_backup_config_file('/tmp/backup-my.cnf') called at /usr//bin/innobackupex line 3689
main::init() called at /usr//bin/innobackupex line 1566
innobackupex: Error: Failed to stream '/tmp/backup-my.cnf': 1 at /usr//bin/innobackupex line 4882.

If i try add to configuration wsrep_sst_method = xtrabackup, so innobackupex: completed OK! but i got message in DOCK.

...
2015-02-18 16:28:58 1043 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4)
2015-02-18 16:28:58 1043 [Note] WSREP: Requesting state transfer: success, donor: 0
2015-02-18 16:29:00 1043 [Note] WSREP: (daf27d21, 'tcp://0.0.0.0:4567') turning message relay requesting off
2015-02-18 16:29:02 1043 [Note] WSREP: 0.0 (bdb1): State transfer to 1.0 (f73fff52892b) complete.
2015-02-18 16:29:02 1043 [Note] WSREP: Member 0.0 (bdb1) synced with group.
WSREP_SST: [ERROR] xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info' (20150218 16:29:02.388)
WSREP_SST: [INFO] Contents of datadir (20150218 16:29:02.393)
WSREP_SST: [INFO] -rw-rw---- 1 mysql mysql 8820 Feb 18 15:14 /var/lib/mysql//mysql/columns_priv.frm
-rw-rw---- 1 mysql mysql 0 Feb 18 15:14 /var/lib/mysql//mysql/columns_priv.MYD
-rw-rw---- 1 mysql mysql 4096 Feb 18 15:14 /var/lib/mysql//mysql/columns_priv.MYI
...
-rw-rw---- 1 mysql mysql 8676 Feb 18 15:14 /var/lib/mysql//performance_schema/users.frm
-rw-rw---- 1 mysql mysql 0 Feb 18 15:14 /var/lib/mysql//test/db.opt (20150218 16:29:02.424)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150218 16:29:02.429)
WSREP_SST: [INFO] Removing the sst_in_progress file (20150218 16:29:02.434)
2015-02-18 16:29:02 1043 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '10.22.1.29' --auth 'sst:xxx' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '1043' '' : 32 (Broken pipe)
2015-02-18 16:29:02 1043 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-02-18 16:29:02 1043 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-02-18 16:29:02 1043 [ERROR] Aborting

2015-02-18 16:29:04 1043 [Note] WSREP: Closing send monitor...
2015-02-18 16:29:04 1043 [Note] WSREP: Closed send monitor.
2015-02-18 16:29:04 1043 [Note] WSREP: gcomm: terminating thread
2015-02-18 16:29:04 1043 [Note] WSREP: gcomm: joining thread
2015-02-18 16:29:04 1043 [Note] WSREP: gcomm: closing backend
2015-02-18 16:29:05 1043 [Note] WSREP: view(view_id(NON_PRIM,864bbb23,6) memb {
daf27d21,0
} joined {
} left {
} partitioned {
864bbb23,0
})
2015-02-18 16:29:05 1043 [Note] WSREP: view((empty))
2015-02-18 16:29:05 1043 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-02-18 16:29:05 1043 [Note] WSREP: gcomm: closed
2015-02-18 16:29:05 1043 [Note] WSREP: Flow-control interval: [16, 16]
2015-02-18 16:29:05 1043 [Note] WSREP: Received NON-PRIMARY.
2015-02-18 16:29:05 1043 [Note] WSREP: Shifting JOINER -> OPEN (TO: 4)
2015-02-18 16:29:05 1043 [Note] WSREP: Received self-leave message.
2015-02-18 16:29:05 1043 [Note] WSREP: Flow-control interval: [0, 0]
2015-02-18 16:29:05 1043 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2015-02-18 16:29:05 1043 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 4)
2015-02-18 16:29:05 1043 [Note] WSREP: RECV thread exiting 0: Success
2015-02-18 16:29:05 1043 [Note] WSREP: recv_thread() joined.
2015-02-18 16:29:05 1043 [Note] WSREP: Closing replication queue.
2015-02-18 16:29:05 1043 [Note] WSREP: Closing slave action queue.
2015-02-18 16:29:05 1043 [Note] WSREP: Service disconnected.
2015-02-18 16:29:05 1043 [Note] WSREP: rollbacker thread exiting
2015-02-18 16:29:06 1043 [Note] WSREP: Some threads may fail to exit.
2015-02-18 16:29:06 1043 [Note] Binlog end
2015-02-18 16:29:06 1043 [Note] /usr/sbin/mysqld: Shutdown complete

Error in my_thread_global_end(): 1 threads didn't exit
150218 16:29:11 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

BDB my.cnf

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

binlog_format=ROW
wsrep_cluster_address=gcomm://
wsrep_sst_auth=sst:xxx
wsrep_provider=/usr/lib/galera3/libgalera_smm.so

bind-address = *

DOCK my.cnf

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

binlog_format=ROW
wsrep_cluster_address=gcomm://bdb1
wsrep_sst_auth=sst:xxx
wsrep_provider=/usr/lib/galera3/libgalera_smm.so
wsrep_sst_receive_address=10.22.1.29

bind-address = *

docker run (HOSTIP i use for wsrep_sst_receive_address in container)

docker run --name galera-1 -i -t -e HOSTIP=`hostname -i` -p 4568:4568 -p 3306:3306 -p 4444:4444 -p 4567:4567 -d szn-percona-cluster

Do you have the same versions of Percona XtraDB Cluster packages installed on BDB and DOCK? Are you sure both BDB and DOCK can see each other? If you are using recent versions of Percona XtraDB Cluster can you try to use wsrep_sst_method=xtrabackup-v2. Also provide us SHOW GRANTS for sst user.

Yes both servers have same version. Host from DOCK container listen and forward ports 3306,4444,4567,4568. If i use xtrabackup-v2 so i get first error message in this topic.

It is weird behavior, because if i ran on DOCK new cluster (wsrep_cluster_address=gcomm://) and after this i try connect DBD to this, everything is ok. But i try this for opposite i got error messages ^^^.

Grants i create like this

CREATE USER 'sst'@'localhost' IDENTIFIED BY 'xxx';
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost';
FLUSH PRIVILEGES;

Problem solved

If i change source for my Dockerfile to ubuntu:trusty everything is ok. If i find out diff between my source and ubuntu i will write here.