Deprecated Warnings Causing Nodes to Drop From Cluster

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

Current Percona XtraDB Cluster Version: 8.0.41-32 (2025-03-13)

We have a situation where running invalid SQL queries that produce deprecation warnings causes nodes to be ejected from the cluster. For instance in the example log output below, a migration is trying to create a table that already exists. One node, just returns the ‘table already exists’ error, while another node includes some deprecation warnings about defining Int width as well. My understanding is this causes the hashes they generate with the error for the re-vote to be different, and thus eject one of the nodes that seems to be inconsistent.

So I have two questions. First, we could try and suppress all deprecation warnings, but I haven’t found a straight forward way to do that in MySQL. We can suppress all warnings in general with log_error_verbosity=1, but that seems like overkill, and obviously has the downside of suppressing warnings. Is there a better way to handle this for all deprecation warnings that may get reported differently across nodes?

The second question is, has this problem come up before and is there a more general recommendation from Percona or Galera for handling this? I could not find any discussions related to this issue, but seems like it could be a common problem. I don’t know if this is correct, but ChatGPT was suggesting that this could happen because the original node is parsing the entire statement and generating the deprecation warnings, while other nodes are using “replay a replicated write-set (RBR)” which doesn’t parse the entire statement for deprecation warnings. I believe XtraDB Clustering/Galera aren’t using standard MySQL replication, but something else, so not sure if that’s related or not. Either way, different errors are being generated on different nodes, ultimately causing this problem, and I’d like to understand this best way to address this.

Any help here is greatly appreciated!

Here are the logs indicating the issue:

2025-08-19T20:19:22.477485Z 0 [Note] [MY-000000] [Galera] Member 1(dev-mysql-01) initiates vote on 40edfb85-4247-11f0-a4ca-eed470f8c373:544806,8f520538a60f5195:  Table 'users' already exists, Error_code: 1050;
2025-08-19T20:19:22.477596Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1050. New vote df478899fb6e67d7 will be used for further steps. Old Vote: 8f520538a60f5195
2025-08-19T20:19:22.477610Z 0 [Note] [MY-000000] [Galera] Votes over 40edfb85-4247-11f0-a4ca-eed470f8c373:544806:
   df478899fb6e67d7:   1/2
Waiting for more votes.
2025-08-19T20:19:22.478740Z 0 [Note] [MY-000000] [Galera] Member 0(dev-mysql-03) initiates vote on 40edfb85-4247-11f0-a4ca-eed470f8c373:544806,eeba60c51ddac56a:  Integer display width is deprecated and will be removed in a future release., Error_code: 1681; Integer display width is deprecated and will be removed in a future release., Error_code: 1681; Table 'users' already exists, Error_code: 1050;
2025-08-19T20:19:22.478823Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1050, 1681, 1681. New vote ead16751af531ade will be used for further steps. Old Vote: eeba60c51ddac56a
2025-08-19T20:19:22.478833Z 0 [Note] [MY-000000] [Galera] Votes over 40edfb85-4247-11f0-a4ca-eed470f8c373:544806:
   df478899fb6e67d7:   1/2
   ead16751af531ade:   1/2
Winner: ead16751af531ade
2025-08-19T20:19:22.478903Z 533443 [ERROR] [MY-000000] [Galera] Inconsistency detected: Inconsistent by consensus on 40edfb85-4247-11f0-a4ca-eed470f8c373:544806
         at ../../../../percona-xtradb-cluster-galera/galera/src/replicator_smm.cpp:process_apply_error():1458
...
2

Correct. Percona XtraDB Cluster uses the Galera communication protocol.

If one node returns something different from the other nodes, then you have a larger issue at hand. In PXC, all nodes should be identical clones of each other. Since you are getting different results on the nodes for the same SQL, that says you have nodes which are not correct copies and thus the node-ejection is 100% expected.

You can create safer DDLs by using ‘IF NOT EXISTS’ modifier in your ‘CREATE TABLE’.

Ideally, though, you should first ensure that all nodes are exactly the same. You may need to wipe node2 and node3 and let them get a fresh SST from node1.

3

Thank you for the quick response! I have already confirmed the state of all three nodes matches. There is nothing different between the three nodes we have deployed in the cluster, best I can tell. Definitely not the table in question here - the schemas match exactly. This is a fairly fresh setup, where all servers were setup identically with matching my.cnf configuration files, and generally the cluster shows healthy with no issues on any of the three nodes. I also checked the active global settings for sql_mode and wsrep_OSU_method, and all match. All versions also match.

The logs I posted seem to indicate this is just a difference in error/warning reporting, which seems can vary under certain circumstances (as our logs are indicating). All nodes are appropriately getting the Error_code: 1050 that the table already exists. It’s just that some nodes (probably the initial one executing the query) is also getting deprecation warnings, Error_code:1681, from parsing that portion of the query at execution before returning. If the other nodes parsed the entire query before returning, they should get the same 1681 errors, since they are all the same version with the same level of logging enabled. This would again seem to indicate they just aren’t parsing that part of the query before they terminate the query and return with the error.

4

Does anyone have any suggestions for us here?

The implications of this are that any deprecation warnings that get introduced have the potential to eject nodes in the cluster when paired with queries that generate other errors (x already exists, duplicate column, etc.). This issue is worsened with automation and tasks or applications that would keep retrying the queries where servers would always disagree on the votes.

We can try and address these issues with the SQL queries themselves today, but that doesn’t put our mind at ease if seemingly innocuous SQL queries can damage our cluster. While we are hosting some of our own in-house built applications, we have other use cases that would use different databases on the same cluster, and this situation allows apps to degrade the DB service for other completely unrelated applications. Also, this would still become a problem as new deprecation warnings are introduced.

Is there somewhere I should publish this as a bug? I can’t believe that the intent here is for deprecation warnings to cause re-votes on SQL queries that fail (due to mal-formed queries, duplicate columns, already exists, etc.). Again, all these nodes are failing the problem queries in question for the same reasons, they just don’t realize it because the deprecation warnings are not generated on all of the nodes (probably the nodes where the query is replayed, but unclear at this point).