How to grant permission in a Galera Cluster

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

Hi all,
I have a 3 nodes Mysql(Ver 8.0.35-27.1) Galera Cluster.
Im trying to grant permission to a user in Galera Cluster, then it failed and the node that I run the command is down.

The command is:

GRANT INSERT, UPDATE, DELETE ON `my-database`.tablename TO 'user'@'%';

Then there is an error:

2024-03-05T01:23:51.444419Z 0 [Note] [MY-000000] [Galera] Member 2(192.168.101.13) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566,c23895a64a070af1:  GRANT ... AS is currently supported only for global privileges., Error_code: 3835;
2024-03-05T01:23:51.444728Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 3835. New vote baebdb9c4478d8d4 will be used for further steps. Old Vote: c23895a64a070af1
2024-03-05T01:23:51.444775Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566:
   baebdb9c4478d8d4:   1/3
Waiting for more votes.
2024-03-05T01:23:51.444915Z 0 [Note] [MY-000000] [Galera] Member 0(192.168.101.11) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566,c23895a64a070af1:  GRANT ... AS is currently supported only for global privileges., Error_code: 3835;
2024-03-05T01:23:51.444914Z 10 [Note] [MY-000000] [Galera] Got vote request for seqno df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566
2024-03-05T01:23:51.445348Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 3835. New vote baebdb9c4478d8d4 will be used for further steps. Old Vote: c23895a64a070af1
2024-03-05T01:23:51.445381Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566:
   baebdb9c4478d8d4:   2/3
Winner: baebdb9c4478d8d4
2024-03-05T01:23:51.447213Z 0 [Note] [MY-000000] [Galera] Recovering vote result from history: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566,baebdb9c4478d8d4
2024-03-05T01:23:51.447344Z 10 [ERROR] [MY-000000] [Galera] Vote 0 (success) on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:82659566 is inconsistent with group. Leaving cluster.
2024-03-05T01:23:51.449495Z 10 [Note] [MY-000000] [Galera] Closing send monitor...
2024-03-05T01:23:51.449529Z 10 [Note] [MY-000000] [Galera] Closed send monitor.
2024-03-05T01:23:51.449554Z 10 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-03-05T01:23:51.449582Z 10 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-03-05T01:23:51.450561Z 10 [Note] [MY-000000] [Galera] gcomm: closing backend
2024-03-05T01:23:51.451201Z 10 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,726e9f57-a0c1,13)
memb {
        add8ab5b-94a7,0
        }
joined {
        }
left {
        }
partitioned {
        726e9f57-a0c1,0
        ece40ef0-952e,0
        }
)
2024-03-05T01:23:51.451281Z 10 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2024-03-05T01:23:51.451309Z 10 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-03-05T01:23:51.451729Z 10 [Note] [MY-000000] [Galera] gcomm: closed
2024-03-05T01:23:51.451782Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-03-05T01:23:51.451891Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-03-05T01:23:51.451917Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2024-03-05T01:23:51.451932Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> OPEN (TO: 82659566)
2024-03-05T01:23:51.451968Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2024-03-05T01:23:51.452001Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2024-03-05T01:23:51.452026Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2024-03-05T01:23:51.452040Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 82659566)
2024-03-05T01:23:51.452062Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2024-03-05T01:23:51.452385Z 10 [Note] [MY-000000] [Galera] recv_thread() joined.
2024-03-05T01:23:51.452413Z 10 [Note] [MY-000000] [Galera] Closing replication queue.
2024-03-05T01:23:51.452460Z 10 [Note] [MY-000000] [Galera] Closing slave action queue.

How to correctly run the GRANT command ?

Thank you all.

2

Hello @hung_le,
The GRANT you used is correct. It looks like you might have data differences on some of your nodes. 2/3 nodes voted to evict node3 because node3 data was different. Use pt-table-checksum — Percona Toolkit Documentation to verify all tables are the same between nodes for the mysql database.

3

Hi @matthewb ,
Thank you, but the in the error log, it says that:

GRANT ... AS is currently supported only for global privileges., Error_code: 3835;

Seems like my command is not correct

4

According to the manual, https://dev.mysql.com/doc/refman/8.0/en/grant.html#grant-as, the error message is correct that GRANT .. AS is only allowed for global privs. But according to your post, that’s not what you are entering. You are entering a simple GRANT … ON … TO user@‘%’

I had no issues doing this in my 8.0.35 PXC

node1 [localhost:29536] {root} ((none)) > CREATE DATABASE `my-database`;
Query OK, 1 row affected (0.04 sec)

node1 [localhost:29536] {root} ((none)) > CREATE TABLE `my-database`.tablenamefoo (id int);
Query OK, 0 rows affected (0.07 sec)

node1 [localhost:29536] {root} ((none)) > CREATE USER 'user'@'%' IDENTIFIED BY 'foobarbaz';
Query OK, 0 rows affected (0.06 sec)

node1 [localhost:29536] {root} ((none)) > GRANT INSERT, UPDATE, DELETE ON `my-database`.tablenamefoo TO 'user'@'%';
Query OK, 0 rows affected (0.02 sec)

node1 [localhost:29536] {root} ((none)) > ^DBye

node1$ ./use -uuser -pfoobarbaz
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 8.0.35-27.1 Percona XtraDB Cluster binary (GPL) 8.0.35, Revision 84d9464, WSREP version 26.1.4.3

node1 [localhost:29536] {user} ((none)) > SELECT CURRENT_USER();
+----------------+
| CURRENT_USER() |
+----------------+
| user@%         |
+----------------+
1 row in set (0.00 sec)
5

Hi @matthewb,
I tried this:

CREATE DATABASE `my-database`;
CREATE TABLE `my-database`.tablenamefoo (id int);
CREATE USER 'user'@'%' IDENTIFIED BY 'foobarbaz';
GRANT INSERT, UPDATE, DELETE ON `my-database`.tablenamefoo TO 'user'@'%';

Then it failed, and this is the error:

2024-03-07T10:38:04.207388Z 0 [Note] [MY-000000] [Galera] Member 1(192.168.102.13) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888,a169df732a709847:  GRANT ... AS is currently supported only for global privileges., Error_code: 3835;
2024-03-07T10:38:04.207634Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 3835. New vote b74fc7830c822810 will be used for further steps. Old Vote: a169df732a709847
2024-03-07T10:38:04.207684Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888:
   b74fc7830c822810:   1/3
Waiting for more votes.
2024-03-07T10:38:04.207742Z 0 [Note] [MY-000000] [Galera] Member 2(192.168.102.11) initiates vote on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888,a169df732a709847:  GRANT ... AS is currently supported only for global privileges., Error_code: 3835;
2024-03-07T10:38:04.207775Z 2 [Note] [MY-000000] [Galera] Got vote request for seqno df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888
2024-03-07T10:38:04.207867Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 3835. New vote b74fc7830c822810 will be used for further steps. Old Vote: a169df732a709847
2024-03-07T10:38:04.207905Z 0 [Note] [MY-000000] [Galera] Votes over df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888:
   b74fc7830c822810:   2/3
Winner: b74fc7830c822810
2024-03-07T10:38:04.208314Z 0 [Note] [MY-000000] [Galera] Recovering vote result from history: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888,b74fc7830c822810
2024-03-07T10:38:04.208389Z 2 [ERROR] [MY-000000] [Galera] Vote 0 (success) on df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888 is inconsistent with group. Leaving cluster.
2024-03-07T10:38:04.209743Z 2 [Note] [MY-000000] [Galera] Closing send monitor...
2024-03-07T10:38:04.209775Z 2 [Note] [MY-000000] [Galera] Closed send monitor.
2024-03-07T10:38:04.209796Z 2 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-03-07T10:38:04.209819Z 2 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-03-07T10:38:04.209990Z 2 [Note] [MY-000000] [Galera] gcomm: closing backend
2024-03-07T10:38:04.210877Z 2 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,74d8d4d2-91bd,21)
memb {
        74d8d4d2-91bd,0
        }
joined {
        }
left {
        }
partitioned {
        ad3801e0-aaeb,0
        b5c77fc5-a894,0
        }
)
2024-03-07T10:38:04.210943Z 2 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2024-03-07T10:38:04.210970Z 2 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-03-07T10:38:04.211275Z 2 [Note] [MY-000000] [Galera] gcomm: closed
2024-03-07T10:38:04.211332Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-03-07T10:38:04.211397Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-03-07T10:38:04.211437Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2024-03-07T10:38:04.211459Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> OPEN (TO: 83071888)
2024-03-07T10:38:04.211493Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2024-03-07T10:38:04.211539Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2024-03-07T10:38:04.211567Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2024-03-07T10:38:04.211597Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 83071888)
2024-03-07T10:38:04.211624Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2024-03-07T10:38:04.211703Z 2 [Note] [MY-000000] [Galera] recv_thread() joined.
2024-03-07T10:38:04.211743Z 2 [Note] [MY-000000] [Galera] Closing replication queue.
2024-03-07T10:38:04.211759Z 2 [Note] [MY-000000] [Galera] Closing slave action queue.
2024-03-07T10:38:04.211815Z 10 [Note] [MY-000000] [Galera] ================================================
View:
  id: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 74d8d4d2-dc6c-11ee-91bd-1641b790a0b5, 192.168.102.12
=================================================
2024-03-07T10:38:04.211871Z 10 [Note] [MY-000000] [Galera] Non-primary view
2024-03-07T10:38:04.211896Z 10 [Note] [MY-000000] [WSREP] Server status change synced -> connected
2024-03-07T10:38:04.211947Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-07T10:38:04.211995Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-07T10:38:04.212046Z 10 [Note] [MY-000000] [Galera] ================================================
View:
  id: df948d8d-5994-11ec-91b5-ff5ae1e8c4ba:83071888
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================
2024-03-07T10:38:04.212069Z 10 [Note] [MY-000000] [Galera] Non-primary view
2024-03-07T10:38:04.212083Z 10 [Note] [MY-000000] [WSREP] Server status change connected -> disconnected
2024-03-07T10:38:04.212094Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-07T10:38:04.212107Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-07T10:38:04.212137Z 10 [Note] [MY-000000] [Galera] Waiting 600 seconds for 4 receivers to finish
2024-03-07T10:38:04.221654Z 11 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-03-07T10:38:04.221660Z 12 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-03-07T10:38:04.221718Z 11 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 11
2024-03-07T10:38:04.221892Z 2 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-03-07T10:38:04.221921Z 12 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 12
2024-03-07T10:38:04.221989Z 2 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 2
2024-03-07T10:38:04.223090Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-03-07T10:38:04.223156Z 10 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
2024-03-07T10:38:04.223185Z 10 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 0
2024-03-07T10:38:04.223201Z 10 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 0 thd: 10

It seems that cannot grant on a specific table.

6

Even I tried to install a new Cluster then run same commands, it still failed with the same error message.

7

@hung_le,
Please stop 2 nodes, leaving 1 node online in PRIMARY state. Then repeat the exercise above. Does that work?

8

Hi @matthewb,
I stopped Node 2, Node 3 then run the same commands in Node 1.
The log in Node 1 seems not any error, but when I start Node 2 again, then there is error in Node 2:

2024-03-08T03:00:10.202300Z 10 [Note] [MY-000000] [Galera] ================================================
View:
  id: 49c05c69-dcf4-11ee-bb3a-9f01679837bd:13
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: -1
  members(1):
        0: 118513ec-dcf5-11ee-87da-a734e6da907b, 192.168.102.11
=================================================
2024-03-08T03:00:10.202374Z 0 [Note] [MY-000000] [Galera] ####### Passing IST CC 16, must_apply: 1, preload: true
2024-03-08T03:00:10.202371Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:10.202513Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060
2024-03-08T03:00:10.202627Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.35-27.1'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Percona XtraDB Cluster (GPL), Release rel27, Revision 84d9464, WSREP version 26.1.4.3.
2024-03-08T03:00:10.207368Z 12 [ERROR] [MY-010584] [Repl] Replica SQL: Error 'GRANT ... AS is currently supported only for global privileges.' on query. Default database: ''. Query: 'GRANT INSERT, UPDATE, DELETE ON `my-database`.`tablenamefoo` TO 'user'@'%' AS 'root'@'localhost' WITH ROLE NONE', Error_code: MY-003835
2024-03-08T03:00:10.207458Z 12 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno 14
2024-03-08T03:00:10.210095Z 12 [ERROR] [MY-000000] [Galera] Inconsistency detected: Failed on preordered 49c05c69-dcf4-11ee-bb3a-9f01679837bd:14: inconsistency.
         at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.35-27/percona-xtradb-cluster-galera/galera/src/replicator_smm.cpp:process_apply_error():1454
2024-03-08T03:00:10.211372Z 12 [Note] [MY-000000] [Galera] Closing send monitor...
2024-03-08T03:00:10.211397Z 12 [Note] [MY-000000] [Galera] Closed send monitor.
2024-03-08T03:00:10.211411Z 12 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-03-08T03:00:10.211433Z 12 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-03-08T03:00:10.211536Z 12 [Note] [MY-000000] [Galera] gcomm: closing backend
2024-03-08T03:00:11.214343Z 12 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,118513ec-87da,10)
memb {
        f7ae3c87-a95a,0
        }
joined {
        }
left {
        }
partitioned {
        118513ec-87da,0
        }
)
2024-03-08T03:00:11.214417Z 12 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2024-03-08T03:00:11.214438Z 12 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-03-08T03:00:11.214641Z 12 [Note] [MY-000000] [Galera] gcomm: closed
2024-03-08T03:00:11.214686Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-03-08T03:00:11.214743Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-03-08T03:00:11.214760Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2024-03-08T03:00:11.214772Z 0 [Note] [MY-000000] [Galera] Shifting JOINER -> OPEN (TO: 16)
2024-03-08T03:00:11.214786Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2024-03-08T03:00:11.214806Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2024-03-08T03:00:11.214819Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2024-03-08T03:00:11.214830Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 16)
2024-03-08T03:00:11.214842Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2024-03-08T03:00:11.214912Z 12 [Note] [MY-000000] [Galera] recv_thread() joined.
2024-03-08T03:00:11.214931Z 12 [Note] [MY-000000] [Galera] Closing replication queue.
2024-03-08T03:00:11.214943Z 12 [Note] [MY-000000] [Galera] Closing slave action queue.
2024-03-08T03:00:11.215502Z 11 [ERROR] [MY-010584] [Repl] Replica SQL: Error 'GRANT ... AS is currently supported only for global privileges.' on query. Default database: ''. Query: 'GRANT INSERT, UPDATE, DELETE ON `my-database`.`tablenamefoo` TO 'user'@'%' AS 'root'@'localhost' WITH ROLE NONE', Error_code: MY-003835
2024-03-08T03:00:11.215559Z 11 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno 15
2024-03-08T03:00:11.218581Z 0 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2024-03-08T03:00:11.218615Z 0 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 15 -> 16
2024-03-08T03:00:11.218662Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-03-08T03:00:11.218713Z 0 [Note] [MY-000000] [Galera] Recording CC from ist: 16
2024-03-08T03:00:11.218732Z 0 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from ist: 8
2024-03-08T03:00:11.218744Z 0 [Note] [MY-000000] [Galera] Min available from gcache for CC from ist: 2
2024-03-08T03:00:11.218802Z 0 [Note] [MY-000000] [Galera] Receiving IST...100.0% (9/9 events) complete.
2024-03-08T03:00:11.218812Z 1 [Note] [MY-000000] [Galera] ================================================
View:
  id: 49c05c69-dcf4-11ee-bb3a-9f01679837bd:16
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 1
  members(2):
        0: 118513ec-dcf5-11ee-87da-a734e6da907b, 192.168.102.11
        1: f7ae3c87-dcf7-11ee-a95a-9388344592a1, 192.168.102.12
=================================================
2024-03-08T03:00:11.218863Z 1 [Note] [MY-000000] [WSREP] Server status change initialized -> joined
2024-03-08T03:00:11.218883Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:11.218907Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:11.222592Z 1 [Note] [MY-000000] [Galera] Draining apply monitors after IST up to 16
2024-03-08T03:00:11.222627Z 1 [Note] [MY-000000] [Galera] IST received: 49c05c69-dcf4-11ee-bb3a-9f01679837bd:16
2024-03-08T03:00:11.222667Z 1 [Warning] [MY-000000] [Galera] Failed to JOIN the cluster after SST gcs_join(49c05c69-dcf4-11ee-bb3a-9f01679837bd:16) failed: 77 (File descriptor in bad state)
         at /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/percona-xtradb-cluster-8.0.35-27/percona-xtradb-cluster-galera/galera/src/galera_gcs.hpp:join():220
2024-03-08T03:00:11.222686Z 1 [Note] [MY-000000] [Galera] Recording CC from sst: 16
2024-03-08T03:00:11.222698Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from sst: 8
2024-03-08T03:00:11.222709Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from sst: 2
2024-03-08T03:00:11.222756Z 1 [Note] [MY-000000] [Galera] ================================================
View:
  id: 49c05c69-dcf4-11ee-bb3a-9f01679837bd:16
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: f7ae3c87-dcf7-11ee-a95a-9388344592a1, 192.168.102.12
=================================================
2024-03-08T03:00:11.222773Z 1 [Note] [MY-000000] [Galera] Non-primary view
2024-03-08T03:00:11.222784Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> connected
2024-03-08T03:00:11.222795Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:11.222809Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:11.222835Z 1 [Note] [MY-000000] [Galera] ================================================
View:
  id: 49c05c69-dcf4-11ee-bb3a-9f01679837bd:16
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================
2024-03-08T03:00:11.222849Z 1 [Note] [MY-000000] [Galera] Non-primary view
2024-03-08T03:00:11.222859Z 1 [Note] [MY-000000] [WSREP] Server status change connected -> disconnected
2024-03-08T03:00:11.222868Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:11.222886Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-03-08T03:00:11.222901Z 1 [Note] [MY-000000] [Galera] Waiting 600 seconds for 4 receivers to finish
2024-03-08T03:00:11.228975Z 12 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-03-08T03:00:11.228990Z 10 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-03-08T03:00:11.228979Z 11 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 6
2024-03-08T03:00:11.229006Z 12 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 12
2024-03-08T03:00:11.229028Z 10 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 10
2024-03-08T03:00:11.229055Z 11 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 6 thd: 11
2024-03-08T03:00:11.229392Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-03-08T03:00:11.229440Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
2024-03-08T03:00:11.229465Z 1 [Note] [MY-000000] [Galera] Slave thread exit. Return code: 0
2024-03-08T03:00:11.229478Z 1 [Note] [MY-000000] [WSREP] Applier thread exiting ret: 0 thd: 1
9

Hi @matthewb,
Can you recreate this issue ?
I installed my cluster using ClusterControl.
Do you need any configuration ?

Thank you.

10

@hung_le, I cannot recreate this issue. I am not using ClusterControl. That is a product made by a 3rd party. I suggest you start node2 manually and test to see if clusterControl is causing the problem. You might also want to reach out to the makers of CC and see what they say.