Gardb FATAL Group requested gcs_proto ver 2

I’m trying to get an arbitrator node up but not having any luck.

I’m using centos 8, with Percona-XtraDB-Cluster-garbd-57.

$ garbd --name=fdb103.loc --group=fdbclust01 --address=gcomm://10.0.0.100,10.0.0.101,10.0.0.103

10.0.0.100 and 101 are already in a cluster. 10.0.0.103 is the garb node.

FATAL: gcs/src/gcs_group.cpp:group_check_proto_ver():261: Group requested gcs_proto_ver: 2, max supported by this node: 0.Upgrade the node before joining this group.Need to abort.
2022-05-20 20:58:28.150 INFO: garbd: Terminated. Aborted

I’ve been searching for hours with no luck what so ever on what to do with this.

In /etc/sysconfig/garb
Man says set the config file as follows but that doesn't work.

             # garbd config file
              daemon=1
              address="gcomm://192.168.90.3:4567"
              group="my_cluster"
              options="gmcast.segment=4;"
              log="/tmp/garb.log"

My current version is;

# Copyright (C) 2012 Codership Oy
# This config file is to be sourced by garb service script.

# REMOVE THIS AFTER CONFIGURATION
# daemon=1
# address="gcomm://10.0.0.100,10.0.0.101,10.0.0.103"
# group="fdbclust01"
# options="gmcast.segment=4;"
# log="/var/log/garbd.log"

#The above nor below seem to make any diff.

# A comma-separated list of node addresses (address[:port]) in the cluster
GALERA_NODES="10.0.0.100,10.0.0.101,10.0.0.103"

# Galera cluster name, should be the same as on the rest of the nodes.
GALERA_GROUP="fdbclust01"

# Optional Galera internal options string (e.g. SSL settings)
# see http://galeracluster.com/documentation-webpages/galeraparameters.html
# GALERA_OPTIONS=""

# Log file for garbd. Optional, by default logs to syslog
# Deprecated for CentOS7, use journalctl to query the log for garbd
LOG_FILE="/var/log/garbd.log"

Cannot find anything that helps me to get this working. I do see the other nodes logs entries when I try the above.

Might as well share the log from one of the nodes;

2022-05-20 21:09:24 0 [Note] WSREP: (03511ad3-9350, 'tcp://0.0.0.0:4567') connection established to adfbe363-bd65 tcp://10.0.0.103:4567
2022-05-20 21:09:24 0 [Note] WSREP: (03511ad3-9350, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2022-05-20 21:09:25 0 [Note] WSREP: EVS version downgrade 1 -> 0
2022-05-20 21:09:25 0 [Note] WSREP: declaring 1e7c1acf-bf40 at tcp://10.0.0.100:4567 stable
2022-05-20 21:09:25 0 [Note] WSREP: declaring adfbe363-bd65 at tcp://10.0.0.103:4567 stable
2022-05-20 21:09:25 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2022-05-20 21:09:25 0 [Note] WSREP: Node 03511ad3-9350 state prim
2022-05-20 21:09:25 0 [Note] WSREP: view(view_id(PRIM,03511ad3-9350,27) memb {
        03511ad3-9350,0
        1e7c1acf-bf40,0
        adfbe363-bd65,0
} joined {
} left {
} partitioned {
})
2022-05-20 21:09:25 0 [Note] WSREP: save pc into disk
2022-05-20 21:09:25 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
2022-05-20 21:09:25 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: ccfacbee-d8bb-11ec-9212-367eaefa427e
2022-05-20 21:09:25 0 [Note] WSREP: STATE EXCHANGE: sent state msg: ccfacbee-d8bb-11ec-9212-367eaefa427e
2022-05-20 21:09:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: ccfacbee-d8bb-11ec-9212-367eaefa427e from 0 (fdb101.loc)
2022-05-20 21:09:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: ccfacbee-d8bb-11ec-9212-367eaefa427e from 1 (fdb100.loc)
2022-05-20 21:09:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: ccfacbee-d8bb-11ec-9212-367eaefa427e from 2 (fdb103.loc)
2022-05-20 21:09:25 0 [Note] WSREP: Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 26,
        members    = 2/3 (joined/total),
        act_id     = 3388,
        last_appl. = 3360,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 20f1fd76-d7f9-11ec-b526-3e066e2f6956
2022-05-20 21:09:25 0 [Note] WSREP: Flow-control interval: [28, 28]
2022-05-20 21:09:25 2 [Note] WSREP: ####### processing CC 3389, local, ordered
2022-05-20 21:09:25 2 [Note] WSREP: ####### My UUID: 03511ad3-d8b7-11ec-9350-eac16c3bba13
2022-05-20 21:09:25 2 [Note] WSREP: Skipping cert index reset
2022-05-20 21:09:25 2 [Note] WSREP: REPL Protocols: 10 (5)
2022-05-20 21:09:25 2 [Note] WSREP: ####### Adjusting cert position: 3388 -> 3389
2022-05-20 21:09:25 0 [Note] WSREP: Service thread queue flushed.
2022-05-20 21:09:25 2 [Note] WSREP: ================================================
View:
  id: 20f1fd76-d7f9-11ec-b526-3e066e2f6956:3389
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(3):
        0: 03511ad3-d8b7-11ec-9350-eac16c3bba13, fdb101.loc
        1: 1e7c1acf-d8b7-11ec-bf40-a214f72b1eaa, fdb100.loc
        2: adfbe363-d8b6-11ec-bd65-fb872d39e0d5, fdb103.loc
=================================================
2022-05-20 21:09:25 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-05-20 21:09:25 2 [Note] WSREP: Lowest cert index boundary for CC from group: 3361
2022-05-20 21:09:25 2 [Note] WSREP: Min available from gcache for CC from group: 10
2022-05-20 21:09:26 0 [Note] WSREP: (03511ad3-9350, 'tcp://0.0.0.0:4567') reconnecting to adfbe363-bd65 (tcp://10.0.0.103:4567), attempt 0
2022-05-20 21:09:30 0 [Note] WSREP: evs::proto(03511ad3-9350, OPERATIONAL, view_id(REG,03511ad3-9350,27)) suspecting node: adfbe363-bd65
2022-05-20 21:09:30 0 [Note] WSREP: evs::proto(03511ad3-9350, OPERATIONAL, view_id(REG,03511ad3-9350,27)) suspected node without join message, declaring inactive
2022-05-20 21:09:31 0 [Note] WSREP: EVS version upgrade 0 -> 1
2022-05-20 21:09:31 0 [Note] WSREP: declaring 1e7c1acf-bf40 at tcp://10.0.0.100:4567 stable
2022-05-20 21:09:31 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2022-05-20 21:09:31 0 [Note] WSREP: Node 03511ad3-9350 state prim
2022-05-20 21:09:31 0 [Note] WSREP: view(view_id(PRIM,03511ad3-9350,28) memb {
        03511ad3-9350,0
        1e7c1acf-bf40,0
} joined {
} left {
} partitioned {
        adfbe363-bd65,0
})
2022-05-20 21:09:31 0 [Note] WSREP: save pc into disk
2022-05-20 21:09:31 0 [Note] WSREP: forgetting adfbe363-bd65 (tcp://10.0.0.103:4567)
2022-05-20 21:09:31 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-05-20 21:09:31 0 [Note] WSREP: (03511ad3-9350, 'tcp://0.0.0.0:4567') turning message relay requesting off
2022-05-20 21:09:31 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: d0a50e39-d8bb-11ec-a663-423b0c914122
2022-05-20 21:09:31 0 [Note] WSREP: STATE EXCHANGE: sent state msg: d0a50e39-d8bb-11ec-a663-423b0c914122
2022-05-20 21:09:31 0 [Note] WSREP: STATE EXCHANGE: got state msg: d0a50e39-d8bb-11ec-a663-423b0c914122 from 0 (fdb101.loc)
2022-05-20 21:09:31 0 [Note] WSREP: STATE EXCHANGE: got state msg: d0a50e39-d8bb-11ec-a663-423b0c914122 from 1 (fdb100.loc)
2022-05-20 21:09:31 0 [Note] WSREP: Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 27,
        members    = 2/2 (joined/total),
        act_id     = 3390,
        last_appl. = 3360,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 20f1fd76-d7f9-11ec-b526-3e066e2f6956
2022-05-20 21:09:31 0 [Note] WSREP: Flow-control interval: [23, 23]
2022-05-20 21:09:31 2 [Note] WSREP: ####### processing CC 3391, local, ordered
2022-05-20 21:09:31 2 [Note] WSREP: ####### My UUID: 03511ad3-d8b7-11ec-9350-eac16c3bba13
2022-05-20 21:09:31 2 [Note] WSREP: Skipping cert index reset
2022-05-20 21:09:31 2 [Note] WSREP: REPL Protocols: 10 (5)
2022-05-20 21:09:31 2 [Note] WSREP: ####### Adjusting cert position: 3390 -> 3391
2022-05-20 21:09:31 0 [Note] WSREP: Service thread queue flushed.
2022-05-20 21:09:31 2 [Note] WSREP: ================================================
View:
  id: 20f1fd76-d7f9-11ec-b526-3e066e2f6956:3391
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: 03511ad3-d8b7-11ec-9350-eac16c3bba13, fdb101.loc
        1: 1e7c1acf-d8b7-11ec-bf40-a214f72b1eaa, fdb100.loc
=================================================
2022-05-20 21:09:31 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-05-20 21:09:31 2 [Note] WSREP: Lowest cert index boundary for CC from group: 3361
2022-05-20 21:09:31 2 [Note] WSREP: Min available from gcache for CC from group: 10
2022-05-20 21:09:33 0 [Note] WSREP:  cleaning up adfbe363-bd65 (tcp://10.0.0.103:4567)