# problem replicating my pcx 5.7 on an AWS instance

**URL:** <https://forums.percona.com/t/problem-replicating-my-pcx-5-7-on-an-aws-instance/5755>\
**Category:** Percona XtraDB Cluster 5.x\
**Created:** [July 20, 2017, 9:21am UTC](https://forums.percona.com/t/problem-replicating-my-pcx-5-7-on-an-aws-instance/5755 "2017-07-20T09:21:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bdelmedico](https://avatars.discourse-cdn.com/v4/letter/b/ec9cab/32.png) [@bdelmedico](https://forums.percona.com/u/bdelmedico)\
**Post date:** [July 20, 2017, 9:21am UTC](https://forums.percona.com/t/problem-replicating-my-pcx-5-7-on-an-aws-instance/5755/1 "2017-07-20T09:21:09Z")

</div>

Good morning guys

I am having a problem replicating my pcx 5.7 on an AWS instance.

I’ve tried replicating over the internet and the VPC, the problems continue. ☹

I can not get past these timeout errors

Does anyone have any ideas ?

Follow the logs.

Packets Pings  
Loss% Snt Last Avg Best Wrst StDev  
0.5% 3662 3.8 3.4 3.1 266.1 6.1

Transfer is at 15Mb.

Servers without firewall

### joiner

/var/lib/mysql# du -h  
1.5G .

### joiner log

# service mysql start

2017-07-20T13:39:01.620686Z 2 [Note] WSREP: State gap can’t be serviced using IST. Switching to SST  
2017-07-20T13:39:01.627005Z 0 [Note] WSREP: Member 0.0 (ip-x.x.x.x) requested state transfer from ‘_any_’. Selected 1.0 (teste-mha-01)(SYNCED) as donor.  
2017-07-20T13:39:01.627024Z 0 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 55690)  
2017-07-20T13:39:01.627046Z 2 [Note] WSREP: Requesting state transfer: success, donor: 1  
2017-07-20T13:39:01.627070Z 2 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) → new(a8d37d7e-5c3b-11e7-9fbf-36d50cc17d69:55690)  
2017-07-20T13:39:04.034731Z 0 [Note] WSREP: (c9e807ea, ‘tcp://0.0.0.0:4567’) connection to peer c9e807ea with addr tcp://x.x.x.x:4567 timed out, no messages seen in PT3S  
2017-07-20T13:39:04.534737Z 0 [Note] WSREP: (c9e807ea, ‘tcp://0.0.0.0:4567’) turning message relay requesting off  
Job for mysql.service failed because a timeout was exceeded. See “systemctl status mysql.service” and “journalctl -xe” for details.  
root@ip-x.x.x.x:/var/lib/mysql# systemctl status mysql.service  
● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon  
Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)  
Active: failed (Result: timeout) since Thu 2017-07-20 13:44:00 UTC; 8min ago  
Docs: man:systemd-sysv-generator(8)  
Process: 1211 ExecStart=/etc/init.d/mysql start (code=killed, signal=TERM)  
Tasks: 15  
Memory: 1.5G  
CPU: 26.864s  
CGroup: /system.slice/mysql.service  
├─1253 /bin/sh /usr/bin/mysqld\_safe  
├─2243 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera\_smm.so --log-error=/var/log/mysql/mysql-error.log --open-files-limit=10  
├─2254 sh -c wsrep\_sst\_rsync --role ‘joiner’ --address ‘x.x.x.x’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --defaults-group-suffix ‘’ --parent ‘2243’ ‘’  
├─2255 /bin/bash -ue /usr/bin/wsrep\_sst\_rsync --role joiner --address x.x.x.x --datadir /var/lib/mysql/ --defaults-file /etc/mysql/my.cnf --defaults-group-suffix --parent 2243  
├─2339 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync\_sst.conf  
├─4403 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync\_sst.conf  
├─4409 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync\_sst.conf  
└─6973 sleep 1

Jul 20 13:45:07 ip-x.x.x.x rsyncd[4398]: receiving file list  
Jul 20 13:45:07 ip-x.x.x.x rsyncd[4402]: receiving file list  
Jul 20 13:45:07 ip-x.x.x.x rsyncd[4403]: receiving file list  
Jul 20 13:45:07 ip-x.x.x.x rsyncd[4401]: receiving file list  
Jul 20 13:45:08 ip-x.x.x.x rsyncd[4404]: receiving file list  
Jul 20 13:45:10 ip-x.x.x.x rsyncd[4431]: connect from ip-10-10-10-101.sa-east-1.compute.internal (10.10.10.101)  
Jul 20 13:45:10 ip-x.x.x.x rsyncd[4431]: rsync to rsync\_sst/./performance\_schema from ip-10-10-10-101.sa-east-1.compute.internal (10.10.10.101)  
Jul 20 13:45:10 ip-x.x.x.x rsyncd[4431]: receiving file list  
Jul 20 13:45:37 ip-x.x.x.x rsyncd[4401]: rsync: connection unexpectedly closed (10869 bytes received so far) [generator]  
Jul 20 13:45:37 ip-x.x.x.x rsyncd[4401]: rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.1]

#### joiner conf

# cat /etc/mysql/my.cnf

[mysql]

# CLIENT

port = 3306  
socket = /var/run/mysqld/mysqld.sock

[mysqld]  
federated  
performance\_schema

innodb\_force\_recovery=0

pxc\_strict\_mode = PERMISSIVE

# GENERAL

user = mysql  
default\_storage\_engine = InnoDB  
socket = /var/run/mysqld/mysqld.sock  
pid\_file = /var/run/mysqld/mysqld.pid  
bind\_address = 0.0.0.0  
skip-name-resolve

# DATA STORAGE

datadir = /var/lib/mysql/  
tmpdir = /var/lib/mysql/tmp

# MyISAM

key\_buffer\_size = 32M  
myisam\_recover\_options = FORCE,BACKUP

# Path to Galera library

wsrep\_provider = /usr/lib/libgalera\_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep\_cluster\_address = gcomm://10.10.10.101,x.x.x.x

# In order for Galera to work correctly binlog format should be ROW

binlog\_format = ROW

# MyISAM storage engine has only experimental support

default\_storage\_engine = InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb\_autoinc\_lock\_mode = 2

# Node #1 address

wsrep\_node\_address = x.x.x.x

# SST method

wsrep\_sst\_method = rsync

# INNODB

innodb\_flush\_method = O\_DIRECT  
innodb\_log\_files\_in\_group = 2  
innodb\_log\_file\_size = 265Mb  
innodb\_flush\_log\_at\_trx\_commit = 0  
innodb\_file\_per\_table = 1  
innodb\_buffer\_pool\_size = 4G  
thread\_pool\_size = 36  
innodb\_doublewrite = 1  
innodb\_thread\_concurrency = 0  
innodb\_max\_dirty\_pages\_pct = 80  
innodb\_thread\_concurrency = 0  
innodb\_buffer\_pool\_instances = 20  
innodb\_log\_buffer\_size = 8M  
innodb\_lock\_wait\_timeout = 120  
thread\_pool\_idle\_timeout = 3600

binlog\_row\_event\_max\_size = 16M  
binlog\_cache\_size = 16M

# Gcache

#wsrep\_provider\_options = “gcache.size=300M; gcache.page\_size=300M; gmcast.segment=2; evs.send\_window=512; evs.user\_send\_window=256; gcs.fc\_limit=128; evs.inactive\_timeout = PT90S; evs.suspect\_timeout = PT30S”  
wsrep\_provider\_options = " gcs.max\_packet\_size=1048576; evs.send\_window=512; evs.user\_send\_window=256; evs.inactive\_timeout = PT100S; evs.suspect\_timeout = PT60S"

#myisam  
#wsrep\_replicate\_myisam = 1

# Cluster name

wsrep\_cluster\_name = ev-cluster-teste

# Node Name

wsrep\_node\_name = ip-x-x-x-x

# Set to number of cpu cores.

wsrep\_slave\_threads = 8

# to enable debug level logging, set this to 1

wsrep\_debug = 0

#innodb\_print\_all\_deadlock = 1

# how many times to retry deadlocked autocommits

wsrep\_retry\_autocommit = 1

# convert locking sessions into transactions

wsrep\_convert\_LOCK\_to\_trx = 1

# Generate fake primary keys for non-PK tables (required for multi-master and parallel applying operation)

wsrep\_certify\_nonPK = 1

#### Required for Galera

innodb\_locks\_unsafe\_for\_binlog = 1

# BINARY LOGGING

log\_bin = /var/lib/mysql/mysql-bin  
expire\_logs\_days = 5  
sync\_binlog = 1  
server\_id = 1

# SAFETY

max\_allowed\_packet = 16M  
max\_connect\_errors = 1000000  
skip\_name\_resolve  
sql-mode = STRICT\_TRANS\_TABLES,ERROR\_FOR\_DIVISION\_BY\_ZERO,NO\_AUTO\_CREATE\_USER,NO\_AUTO\_VALUE\_ON\_ZERO,NO\_ENGINE\_SUBSTITUTION,NO\_ZERO\_DATE,NO\_ZERO\_IN\_DATE #,ONLY\_FULL\_GROUP\_BY  
sysdate\_is\_now = 1  
innodb = FORCE  
innodb\_strict\_mode = 1

# CACHES AND LIMITS

tmp\_table\_size = 256M  
max\_heap\_table\_size = 256M  
query\_cache\_type = 0  
query\_cache\_size = 0  
max\_connections = 100  
thread\_cache\_size = 120  
open\_files\_limit = 100000  
table\_definition\_cache = 4096  
table\_open\_cache = 4096  
read\_buffer\_size = 8M  
query\_cache\_limit = 1M  
join\_buffer\_size = 6M  
#table\_cache = 2000

# LOGGING

log\_error = /var/log/mysql/mysql-error.log  
#log\_queries\_not\_using\_indexes = 1  
slow\_query\_log = 1  
slow\_query\_log\_file = /var/log/mysql/mysql-slow.log

general\_log\_file = /var/log/mysql/mysql-general.log  
general\_log = 1

#### donor

/var/lib/mysql# du -h  
64G .

#### donor log

# 

2017-07-20T14:07:53.707628Z 2 [Note] WSREP: DONOR thread signaled with 0  
2017-07-20T14:07:53.787297Z 0 [Note] WSREP: Flushing tables for SST…  
2017-07-20T14:07:53.794344Z 0 [Note] WSREP: Provider paused at a8d37d7e-5c3b-11e7-9fbf-36d50cc17d69:55690 (5)  
2017-07-20T14:07:53.797108Z 0 [Note] WSREP: Table flushing completed.  
2017-07-20T14:07:54.004707Z WSREP\_SST: [INFO] Starting rsync of data-dir…  
2017-07-20T14:07:55.810592Z 0 [Note] WSREP: (c4b642ee, ‘tcp://0.0.0.0:4567’) turning message relay requesting off  
rsync: read error: Connection timed out (110)  
rsync error: error in socket IO (code 10) at io.c(802) [sender=3.1.0]  
rsync: read error: Connection timed out (110)  
rsync error: error in socket IO (code 10) at io.c(802) [sender=3.1.0]  
rsync: read error: Connection timed out (110)  
rsync error: error in socket IO (code 10) at io.c(802) [sender=3.1.0]

### network

net.core.somaxconn = 1000  
net.core.netdev\_max\_backlog = 5000  
net.core.rmem\_max = 16777216  
net.core.wmem\_max = 16777216  
net.ipv4.tcp\_wmem = 4096 12582912 16777216  
net.ipv4.tcp\_rmem = 4096 12582912 16777216  
net.ipv4.tcp\_max\_syn\_backlog = 8096  
net.ipv4.tcp\_slow\_start\_after\_idle = 0  
net.ipv4.tcp\_tw\_reuse = 1  
net.ipv4.ip\_local\_port\_range = 10240 65535

fs.file-max=200000  
kernel.sem=250 32000 100 1024  
kernel.shmmax=4294967295

###### 

net.ipv4.tcp\_retries2 = 2  
net.ipv4.tcp\_keepalive\_time = 12  
net.ipv4.tcp\_keepalive\_intvl = 2  
net.ipv4.tcp\_keepalive\_probes = 9

vm.swappiness = 0  
vm.dirty\_ratio = 80  
vm.dirty\_background\_ratio = 5  
vm.dirty\_expire\_centisecs = 12000

---

<div class="post-metadata">

**Author:** ![bdelmedico](https://avatars.discourse-cdn.com/v4/letter/b/ec9cab/32.png) [@bdelmedico](https://forums.percona.com/u/bdelmedico)\
**Post date:** [July 20, 2017, 9:33am UTC](https://forums.percona.com/t/problem-replicating-my-pcx-5-7-on-an-aws-instance/5755/2 "2017-07-20T09:33:38Z")

</div>

# dpkg -l | grep percona

ii percona-release 0.1-4.xenial all Package to install Percona gpg key and APT repo  
ii percona-xtrabackup-24 2.4.7-2.xenial amd64 Open source backup tool for InnoDB and XtraDB  
ii percona-xtradb-cluster-5.7-dbg 5.7.18-29.20-1.xenial amd64 Debugging package for Percona XtraDB Cluster  
ii percona-xtradb-cluster-client-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database client binaries  
ii percona-xtradb-cluster-common-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)  
ii percona-xtradb-cluster-full-57 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster with Galera  
ii percona-xtradb-cluster-garbd-5.7 5.7.18-29.20-1.xenial amd64 Garbd components of Percona XtraDB Cluster  
ii percona-xtradb-cluster-garbd-debug-5.7 5.7.18-29.20-1.xenial amd64 Debugging package for Percona XtraDB Cluster Garbd.  
ii percona-xtradb-cluster-server-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database server binaries  
ii percona-xtradb-cluster-server-debug-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database server UNIV\_DEBUG binaries  
ii percona-xtradb-cluster-test-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database test suite