In AWS ec2, applications are only allowed to listen private IP rather than public IP. Public IP is not visible on the host. Anything send to public IP will be forward to corresponding private IP. IST receiver will try to bind the address specified by wsrep_node_address. For galera clusters across data centers, we must set wsrep_node_address to node public IP address.
Here’s the problem:
No NIC has public IP.
Setting wsrep_node_address to public IP address will make IST receiver tries to bind that IP and fails:
2016-08-04 06:06:16 19852 [Warning] WSREP: Failed to prepare for incremental state transfer: Failed to open IST listener at tcp://52.207.12.34:4568’, asio error ‘Cannot assign requested address’: 99 (Cannot assign requested address) at galera/src/ist.cpp:prepare():326. IST will be unavailable.
Setting wsrep_node_address to private IP address will cause the donor node tries to access joiner node via private IP address, which is not possible.
We need to separate the listening address (private) and access adresss (public). Setting wsrep_node_incoming_address is not helping in this case. The donor still tries to access joiner via wsrep_node_address (private).