Galera: change server_id & domain-id on *running* cluster

MySQL & MariaDB MariaDB Server & Utilities
1

Hello,

I’m trying to change those ids on an existing cluster, but can’t manage to make it work properly, and would like some help:

Context:

  • a 3-nodes Galera cluster in production, receiving read/write traffic
  • using MariaDB v10.11
  • config: server_id=10, gtid_domain_id=100, wsrep_gtid_domain_id=1000, wsrep_gtid_mode=ON (on all 3 nodes)
  • no (async) replica at all

Goal

(Long-term goal, not the objective of this message): I want to setup a partial async replication from Galera cluster A to Galera cluster B, in order to perform a no-downtime migration of some tables and reorganize our data.
But both clusters have the same server_id & domain ids, which is a blocker to the replication according to mariadb’s doc.

Thus my current goal of changing server & domain ids to different values on the target cluster (B).
But my tests in an isolated env are not working properly: the generated gtid still use the old ids.

What I already tested

Test 1:

SET GLOBAL wsrep_gtid_domain_id = 2000;   -- on all 3 nodes

=> param accepted, and almost ok:

SHOW GLOBAL VARIABLES WHERE Variable_name IN ('wsrep_gtid_domain_id',
'gtid_current_pos', 'gtid_binlog_pos', 'gtid_binlog_state')"
+----------------------+-------------------------------------+
| Variable_name        | Value                               |
+----------------------+-------------------------------------+
| gtid_binlog_pos      | 100-10-28,1000-10-1928,2000-10-3870 |
| gtid_binlog_state    | 100-10-28,1000-10-1928,2000-10-3870 |
| gtid_current_pos     |                                     |
| wsrep_gtid_domain_id | 2000                                |
+----------------------+-------------------------------------+

=> new (2000) gtid domain added & increases, and the old one (1000) stays there, constant :white_check_mark:
=> only strange thing: the gtid_current_pos is not filled anymore :thinking: Do you know why ? Is it important ?

Test 2:

SET GLOBAL server_id = 20;   -- on all 3 nodes

=> refused: Can't change server_id because wsrep and wsrep_gtid_mode is set. You can set server_id only with wsrep_new_cluster.
=> impossible to change the server_id without re-bootstraping the whole cluster ?

Idea 3:

Not tested yet, but thinking about: disabling wsrep_gtid_mode temporarily, then change server_id & domain-id, and enabling wsrep_gtid_mode back.
Could it be a good idea ?

Conclusion

I welcome any help to guide me into performing those change of ids safely :folded_hands:
Thanks !

2

Bootstrapping a cluster simply means stopping all nodes, changing the .cnf, and starting the cluster again; has nothing to do with resetting/erasing data, so this should take only a couple of minutes to accomplish.

3

Thanks @matthewb for your answer !

By stopping all nodes, we’ll have a downtime for applications, so not the greatest solution at first look :confused:

But let’s say that we accept that, what would be the steps in practice ?
Stop all nodes gracefully, change server-id + domain-ids in config file of all nodes, restart the first one using galera_new_cluster, and then start the other nodes normally ? Won’t it trigger an SST (and thus longer recovery time) because of a new cluster state UUID ?

4

Yes, about 1 minute of downtime, probably less.

I can’t speak for the wsrep_gtid_domain_id as that is a MariaDB-specific feature I’m unfamiliar with. However, server_id doesn’t have anything to do with wsrep, so changing this alone should not trigger an SST.