# No wsrep and general variables after upgrade **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [September 19, 2023, 4:42pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283 "2023-09-19T16:42:37Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 19, 2023, 4:42pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/1 "2023-09-19T16:42:37Z")
Hi all, we upgraded our Percona Galera Xtradb Clusters (2 different cluster, one on CentOS and one on Ubuntu) to the version 5.7.43-31.65-1 and on both servers we got performance scheme wrong structure we fixed the error with mysql\_upgrade (but i had to force it because we use InnoDB engine enforced) and we restarted all the cluster nodes after that on both cluster we have no wsrep variables anymore mysql\> SHOW STATUS LIKE ‘%wsrep%’; Empty set (0.00 sec) mysql\> SHOW GLOBAL STATUS LIKE ‘%wsrep%’; Empty set (0.00 sec) mysql\> SHOW GLOBAL VARIABLES LIKE ‘%wsrep%’; Empty set (0.00 sec) in general also the mysql\> show variables; Empty set (0.00 sec) shows no variables on all nodes our scripts based on them are not working, but the cluster seems to work how i can fix the problem ? We also tried to stop the cluster and bootstraping from the last node, but wsrep and general variables are not available anymore any suggestions ? Claudio ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 19, 2023, 5:22pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/2 "2023-09-19T17:22:14Z")
> [@soprano](#): > > mysql\> SHOW STATUS LIKE ‘%wsrep%’; > Empty set (0.00 sec) Either A) the wsrep plugin did not load; Check error logs, or B) the mysql\_upgrade failed. You should shut down MySQL, disable the wsrep\_provider in my.cnf, start plain non-galera MySQL and run mysql\_upgrade. Restart MySQL. Read the error log. Ensure no errors/warnings about the performance\_schema. Then, enable wsrep\_provider and bootstrap. ---
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 20, 2023, 8:53am UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/3 "2023-09-20T08:53:18Z")
My first question is, the cluster is actually working fine ? is replicating datas ? because the tests i did seems all working good, but to be sure data was consistent my Load Balancer use only one node of the Cluster, so all the writes are going only on one node from the update. For the steps indicated Will i have to do it for each node of the cluster ? or only on the actual working node ? So the steps are: - shutdown the mysql on the actual working node with - disable wsrep\_provider in my.cnf on that node in this way - start mysql normally - run mysql\_upgrade - search the logs for errors - if errors found report them here - if no errors found, enable wsrep\_provider again in my.cnf and bootstrap the node For the rest of the nodes what i have to do ? Do i have to redo the same steps ? and then start mysql normally ? To be sure they will resync from the one is working do i have to delete the grastate.dat file ? Let me know. Claudio ---
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 20, 2023, 1:19pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/4 "2023-09-20T13:19:41Z")
I stopped one of nodes of first cluster (not the actually working) commented the lines of wsrep\_provider started mysql Sep 20 15:14:19 xxxx systemd: Starting Percona XtraDB Cluster… Sep 20 15:14:20 xxxx mysqld\_safe: mysqld\_safe Adding ‘/usr/lib64/libjemalloc.so.1’ to LD\_PRELOAD for mysqld Sep 20 15:14:20 xxxx mysqld\_safe: 2023-09-20T13:14:20.459530Z mysqld\_safe Logging to syslog. Sep 20 15:14:20 xxxx mysqld\_safe: 2023-09-20T13:14:20.496414Z mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql/db Sep 20 15:14:20 xxxx mysqld\_safe: Starting mysqld daemon with databases from /var/lib/mysql/db Sep 20 15:14:20 xxxx mysqld\_safe: 2023-09-20T13:14:20.509504Z mysqld\_safe Skipping wsrep-recover for de5a461b-7059-11e7-9135-b6f59edaa63d:345591159 pair Sep 20 15:14:20 xxxx mysqld\_safe: Skipping wsrep-recover for de5a461b-7059-11e7-9135-b6f59edaa63d:345591159 pair Sep 20 15:14:20 xxxx mysqld\_safe: 2023-09-20T13:14:20.512466Z mysqld\_safe Assigning de5a461b-7059-11e7-9135-b6f59edaa63d:345591159 to wsrep\_start\_position Sep 20 15:14:20 xxxx mysqld\_safe: Assigning de5a461b-7059-11e7-9135-b6f59edaa63d:345591159 to wsrep\_start\_position Sep 20 15:14:21 xxxx mysql-systemd: SUCCESS! Sep 20 15:14:21 xxxx systemd: Started Percona XtraDB Cluster. when i run mysql\_upgrade i get [root@xxxx ~]# mysql\_upgrade -u root Checking if update is needed. This installation of MySQL is already upgraded to 5.7.43-47, use --force if you still need to run mysql\_upgrade do i have to force it ? The strange thing is that [root@xxxx ~]# rpm -qa | grep -i percona Percona-XtraDB-Cluster-client-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-57-5.7.43-31.65.1.el7.x86\_64 percona-release-1.0-27.noarch percona-xtrabackup-24-2.4.28-1.el7.x86\_64 percona-toolkit-3.5.4-2.el7.x86\_64 Percona-XtraDB-Cluster-server-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-compat-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-57-5.7.43-31.65.1.el7.x86\_64 version is 5.7.43-31 why it said 5.7.43-47 on mysql\_upgrade ? Claudio ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 20, 2023, 5:21pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/5 "2023-09-20T17:21:43Z")
Force it. Restart MySQL in NON-PXC mode. Log in. Check if you can SHOW GLOBAL VARIABLES and SHOW GLOBAL STATUS. Check mysql error log. If all is good, shut down. Enable PXC, start. Repeat verification. Check wsrep\_ variables now visible. ---
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 21, 2023, 7:32am UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/6 "2023-09-21T07:32:10Z")
Ok i did this systemctl stop mysql modified /etc/my.cnf #wsrep\_provider = … systemctl start mysql [root@XXXX ~]# mysql\_upgrade -u root --force Checking server version. Running queries to upgrade MySQL server. mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.db] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.user] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.func] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.tables\_priv] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.columns\_priv] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.proc] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.procs\_priv] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.event] mysql\_upgrade: [ERROR] 1050: Table ‘cond\_instances’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_current’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_history’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_history\_long’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_summary\_by\_instance’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_summary\_by\_host\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_summary\_by\_user\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_summary\_by\_account\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_summary\_by\_thread\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_waits\_summary\_global\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘file\_instances’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘file\_summary\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘file\_summary\_by\_instance’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘socket\_instances’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘socket\_summary\_by\_instance’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘socket\_summary\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘host\_cache’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘mutex\_instances’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘objects\_summary\_global\_by\_type’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘performance\_timers’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘rwlock\_instances’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘setup\_actors’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘setup\_consumers’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘setup\_instruments’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘setup\_objects’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘setup\_timers’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘table\_io\_waits\_summary\_by\_index\_usage’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘table\_io\_waits\_summary\_by\_table’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘table\_lock\_waits\_summary\_by\_table’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘threads’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘processlist’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_current’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_history’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_history\_long’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_summary\_by\_thread\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_summary\_by\_host\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_summary\_by\_user\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_summary\_by\_account\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_stages\_summary\_global\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_current’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_history’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_history\_long’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_by\_thread\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_by\_host\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_by\_user\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_by\_account\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_global\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_current’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_history’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_history\_long’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_summary\_by\_thread\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_summary\_by\_host\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_summary\_by\_user\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_summary\_by\_account\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_transactions\_summary\_global\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘hosts’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘users’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘accounts’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘memory\_summary\_global\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘memory\_summary\_by\_thread\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘memory\_summary\_by\_account\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘memory\_summary\_by\_host\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘memory\_summary\_by\_user\_by\_event\_name’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_by\_digest’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘events\_statements\_summary\_by\_program’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘prepared\_statements\_instances’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_connection\_configuration’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_group\_member\_stats’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_group\_members’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_connection\_status’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_applier\_configuration’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_applier\_status’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_applier\_status\_by\_coordinator’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘replication\_applier\_status\_by\_worker’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘pxc\_cluster\_view’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘session\_connect\_attrs’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘session\_account\_connect\_attrs’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘table\_handles’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘metadata\_locks’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘user\_variables\_by\_thread’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘variables\_by\_thread’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘global\_variables’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘session\_variables’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘status\_by\_thread’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘status\_by\_user’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘status\_by\_host’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘status\_by\_account’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘global\_status’ already exists mysql\_upgrade: [ERROR] 1050: Table ‘session\_status’ already exists mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.proxies\_priv] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.tables\_priv] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.columns\_priv] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.user] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.db] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.func] mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.procs\_priv] mysql\_upgrade: [ERROR] 1644: **Unexpected content found in the performance\_schema database**. Checking system database. DB1 OK … Upgrade process completed successfully. Checking if update is needed. so performance schema was not correct we enforce innodb engine started again mysql was showing still [root@XXXX ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 488 Server version: 5.7.43-47-57-log Percona XtraDB Cluster (GPL), Release rel47, Revision 69b5c96, WSREP version 31.65, wsrep\_31.65 Copyright (c) 2009-2023 Percona LLC and/or its affiliates Copyright (c) 2000, 2023, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement. mysql\> show variables; Empty set (0.00 sec) mysql\> show global variables; Empty set (0.00 sec) mysql\> SHOW GLOBAL VARIABLES; Empty set (0.01 sec) mysql\> SHOW GLOBAL STATUS; Empty set (0.00 sec) mysql\> quit ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 21, 2023, 3:48pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/7 "2023-09-21T15:48:46Z")
> [@soprano](#): > > [root@XXXX ~]# mysql\_upgrade -u root --force > Checking server version. > Running queries to upgrade MySQL server. > mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.db] > mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.user] > mysql\_upgrade: [ERROR] 1726: Storage engine ‘InnoDB’ does not support system tables. [mysql.func] This is messed up. Are you somehow using MySQL 5.7 and MySQL 8 on the same data directory? The mysql.\* tables should **NOT** be InnoDB tables in 5.7. I would completely erase this server and let it SST from a good/existing node. ---
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 21, 2023, 4:50pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/8 "2023-09-21T16:50:52Z")
I’m using the Percona XtraDB MYSQL Cluster 5.7 on 3 nodes, then the update comes out i updated all the 3 nodes and this is the result anyway this is that is installed on the 3 ndoes [root@node1 ~]# rpm -qa | grep -i percona Percona-XtraDB-Cluster-client-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-57-5.7.43-31.65.1.el7.x86\_64 percona-release-1.0-27.noarch percona-toolkit-3.5.4-2.el7.x86\_64 Percona-XtraDB-Cluster-server-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-compat-57-5.7.43-31.65.1.el7.x86\_64 percona-xtrabackup-24-2.4.28-1.el7.x86\_64 Percona-XtraDB-Cluster-57-5.7.43-31.65.1.el7.x86\_64 [root@node2 ~]# rpm -qa | grep -i percona Percona-XtraDB-Cluster-client-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-server-57-5.7.43-31.65.1.el7.x86\_64 percona-xtrabackup-24-2.4.28-1.el7.x86\_64 percona-toolkit-3.5.4-2.el7.x86\_64 Percona-XtraDB-Cluster-57-5.7.43-31.65.1.el7.x86\_64 percona-release-1.0-27.noarch Percona-XtraDB-Cluster-shared-compat-57-5.7.43-31.65.1.el7.x86\_64 [root@node3 ~]# rpm -qa | grep -i percona Percona-XtraDB-Cluster-client-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-57-5.7.43-31.65.1.el7.x86\_64 percona-release-1.0-27.noarch percona-xtrabackup-24-2.4.28-1.el7.x86\_64 percona-toolkit-3.5.4-2.el7.x86\_64 Percona-XtraDB-Cluster-server-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-shared-compat-57-5.7.43-31.65.1.el7.x86\_64 Percona-XtraDB-Cluster-57-5.7.43-31.65.1.el7.x86\_64 the problem is that all 3 nodes are screwed so i can’t SST from none. I would have a backup of the night before the yum update, what i have to recover from backup ? I have to fix 2 clusters on both the same problem, this is on Centos 7.9, the other is on Ubuntu LTS 20.04 but both the same errors, same situation, something gone wrong on the update script for sure, all the servers started mysql with no problem, while the performance\_schema had wrong structure error on the logs. Repeat we have enforce\_storage\_engine = InnoDB sql\_mode = ‘’ pxc\_strict\_mode = PERMISSIVE in my.cnf of all nodes on both clusters. I have to fix this cluster, there is a way to recreate the performance\_schema DB ? Else i can take one node down, reinstall it, then take mysqldumps from backup, bootstraping it and then reinstall the other nodes, let me know the fastest way, with minor downtime, i have a lot of stuffs on it that need to be on. Claudio ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 21, 2023, 7:12pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/9 "2023-09-21T19:12:31Z")
You may have messed up everything: > **[Percona Server for MySQL - Enforcing Storage Engine](https://docs.percona.com/percona-server/5.7/management/enforce_engine.html)** > > Percona Server for MySQL has implemented variable which can be used for enforcing the use of a specific storage engine. ```auto If you’re using enforce_storage_engine, you must either disable it before doing mysql_upgrade or perform mysql_upgrade with server started with --skip-grants-tables. ``` I would not recommend the use of this parameter at all. Remove it, then restore your backup to restore the MYI-based system tables. PXC doesn’t work with non-InnoDB engines anyway, so this variable isn’t really needed. ---
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 23, 2023, 10:34am UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/10 "2023-09-23T10:34:53Z")
Anyway i succeded to recover one of two cluster that on Ubuntu 20.04, this is that i did 1. take one node down systemctl stop mysql 1. commented in /etc/mysql/percona-xtradb-cluster.conf.d/wsrep.cnf #wsrep\_provider = … #engine\_enforce\_engine = InnoDB #sql\_mode = ‘’ 1. started mysql systemctl start mysql 1. dropped performance\_schema database mysql -u root -p mysql\> DROP DATABASE performance\_schema; 1. run mysql\_upgrade mysql\_upgrade -u root --force -p 1. restarted mysql systemctl restart mysql 1. verified variables were ok mysql -u root -p mysql\> SHOW VARIABLES; and all variables finally were there 1. stopped mysql systemctl stop mysql 1. enabled wsrep\_provider in /etc/mysql/percona-xtradb-cluster.conf.d/wsrep.cnf wsrep\_provider = … 1. started Percona Xtradb systemctl start mysql and all is ok on this node Repeat all the steps on the second node and third node. Now my main question is, if someone create a table with MyIsam engine it will not be replicated, for this behaviour we enabled enforce\_storage\_engine and sql\_mode, there is a way to deny creation of MyIsam DB or TABLES ? And is it recommended ? Another question is why mysql\_upgrade don’t check that enforce\_engine\_storage is active and warn about it and don’t permit to make disasters when one upgrade the percona cluster ? The other cluster is not working, we tried to make another mysql\_upgrade (someone suggested it) and it screwed the nodes resetting mysql root password and deleting all the DB data files where we executed it, luckily we have a single node working in bootstrapping mode (no varables on it), Next week we’ll make a down and we will repair it, then we will add the nodes again and they would SST all the DB datafiles. Claudio ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [September 23, 2023, 2:26pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/11 "2023-09-23T14:26:29Z")
> [@soprano](#): > > engine\_enforce\_engine engine\_enforce\_engine only works if you also have NO\_ENGINE\_SUBSTUTION enabled for sql\_mode, which you don’t above. You can use this parameter _after_ you have bootstrapped your cluster correctly. Ensure all the system tables are correct _then_ you can implement this parameter. ---
**Author:** ![soprano](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@soprano](https://forums.percona.com/u/soprano)\ **Post date:** [September 25, 2023, 3:51pm UTC](https://forums.percona.com/t/no-wsrep-and-general-variables-after-upgrade/25283/12 "2023-09-25T15:51:59Z")
we have default\_storage\_engine = InnoDB enforce\_storage\_engine = InnoDB sql\_mode = ‘’ the last one would permit engine substution, we want all is InnoDB, no MyIsam or Memory engine is that right ? Claudio