-98 (Address already in use) - cannot start node

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

When trying to restart the 3rd node in our cluster, we are getting:

2022-11-01T11:30:00.721346Z 0 [Warning] WSREP: error while trying to listen ‘ssl://0.0.0.0:4567?socket.non_blocking=1’, asio error ‘bind: Address already in use’
2022-11-01T11:30:00.721385Z 0 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen ‘ssl://0.0.0.0:4567?socket.non_blocking=1’, asio error ‘bind: Address already in use’: 98 (Address already in use)
at gcomm/src/asio_tcp.cpp:listen():928
2022-11-01T11:30:00.721392Z 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -98 (Address already in use)
2022-11-01T11:30:00.721437Z 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1514: Failed to open channel ‘morpheusdb-cluster’ at ‘gcomm://xxx.xxx.xxx.xxx’: -98 (Address already in use)
2022-11-01T11:30:00.721442Z 0 [ERROR] WSREP: gcs connect failed: Address already in use
2022-11-01T11:30:00.721446Z 0 [ERROR] WSREP: Provider/Node (gcomm://xxx.xxx.xxx.xxx) failed to establish connection with cluster (reason: 7)
2022-11-01T11:30:00.721449Z 0 [ERROR] Aborting

I am not sure where to go from here.

1 Like
2

Hi @jimbo881

Thank you for being an active member of our community. Did you check if there is already an
instance of MySQL running? Usually this kind of errors appear when you try to start a process
that is already running, and has the port allocated.

Check this:
netstat -anpe | grep ":4567" | grep "LISTEN"

The results for me (I started a nc session to listen on that port… this are not the results you will see)

[jplasanz@pc103orchestratorapp-01 ~]$ sudo netstat -anpe | grep ":4567" | grep "LISTEN"
tcp        0      0 0.0.0.0:4567            0.0.0.0:*               LISTEN      705643     290161493  10163/nc
tcp6       0      0 :::4567                 :::*                    LISTEN      705643     290161492  10163/nc

The last column reports the process identified and the executable program that is running and has the port allocated.

Thank you.

Pep

1 Like
4

The problem is on the donor node:

2022-11-01T15:33:12.049288Z 57 [Note] Access denied for user ‘sstuser’@‘localhost’ (using password: YES)
2022-11-01T15:33:12.052292Z 58 [Note] Access denied for user ‘sstuser’@‘localhost’ (using password: YES)

I cannot get it to take the sstuser password no matter what I try…

This is v5.7 and the wsrep_sst_auth=sstuser: is set in the /etc/my.cnf file correctly…

1 Like
5

This first error indicates you are running multiple PXC instances on the same host -

This is a new error - can you verify that the sst user can log in with the password from the my.cnf config?

1 Like
6

The first error is from the down node 3. The reason for it is the second error from the donor node 2 which is the access denied error. I verified I can log in with the password from the my.cnf file on the joiner node on the donor node…

1 Like
7

Hi @jimbo881
Can you share the full error logs again? I don’t see the link here between “port already in use” error and “access denied for sstuser” error

1 Like
8

It came down to:

set global wsrep_sst_auth=“sstuser:newpass”;

Being set correctly on the donor node.

Thanks for all the responses.

1 Like