# Galera Cluser hung state with wsrep: preparing for TO isolation

**URL:** <https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359>\
**Category:** Percona XtraDB Cluster\
**Created:** [September 25, 2024, 7:53pm UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359 "2024-09-25T19:53:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chanakya](https://avatars.discourse-cdn.com/v4/letter/c/cab0a1/32.png) [@Chanakya](https://forums.percona.com/u/Chanakya)\
**Post date:** [September 25, 2024, 7:53pm UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359/1 "2024-09-25T19:53:35Z")

</div>

Hi,  
I have a backend database for orchestrator running galera cluster. When the wsrep\_OSU\_method is TOI, whole cluster goes into hung state with all connections waiting on the DDL to complete.

When I change the wsrep\_OSU\_method to NBO, I get errors from orchestrator whenever I run any operation. ERROR Error 1235 (42000): This version of MySQL doesnt yet support this query in wsrep\_OSU\_method NBO

Please help as this is happening too frequently now.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [September 25, 2024, 9:18pm UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359/2 "2024-09-25T21:18:42Z")

</div>

> [@Chanakya](#):
>
> When the wsrep\_OSU\_method is TOI, whole cluster goes into hung state with all connections waiting on the DDL to complete.

This is correct, and expected behavior.

> [@Chanakya](#):
>
> When I change the wsrep\_OSU\_method to NBO, I get errors from orchestrator whenever I run any operation.

What operation are you trying to do? You should not be making any topology changes or other DDLs while an existing DDL is running.

---

<div class="post-metadata">

**Author:** ![Chanakya](https://avatars.discourse-cdn.com/v4/letter/c/cab0a1/32.png) [@Chanakya](https://forums.percona.com/u/Chanakya)\
**Post date:** [September 25, 2024, 9:23pm UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359/3 "2024-09-25T21:23:06Z")

</div>

It appears to be MySQL orchestrator trying to run  
"create database if not exists orchestrator " – This is run often.

Any other alternatives to avoid this behavior @matthewb ?

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [September 25, 2024, 10:38pm UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359/4 "2024-09-25T22:38:44Z")

</div>

> **[Percona XtraDB Cluster - Non-Blocking Operations (NBO)](https://docs.percona.com/percona-xtradb-cluster/8.0/nbo.html)**
>
> An Online Schema Upgrade can be a daily issue in an environment with accelerated development and deployment. The task becomes more difficult as the data grows. An ALTER TABLE statement is a multi-step operation and must run until it is complete....

NBO does not support “CREATE DATABASE …”

You can change the `wsrep_osu_method` on a per-session basis, so keep the global setting in TOI, but change it for the session to NBO to accomplish your long-running DDL.

> [@Chanakya](#):
>
> "create database if not exists orchestrator " – This is run often.

I would investigate why; this should run once and never again. Find a way to turn this off.

---

<div class="post-metadata">

**Author:** ![Chanakya](https://avatars.discourse-cdn.com/v4/letter/c/cab0a1/32.png) [@Chanakya](https://forums.percona.com/u/Chanakya)\
**Post date:** [September 26, 2024, 4:31pm UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359/5 "2024-09-26T16:31:06Z")

</div>

@matthewb I really doubt orchestrator has any such option of doing specific ones per session. Please correct me if I am wrong here.

Also, create database appears to be on this line. Worst case I can may be disable and have the database created prior to starting the orchestrator service.

> <https://github.com/openark/orchestrator/blob/f0d685e0325322ba28f0eb79e3e64eceff241a30/go/db/db.go#L188>

Do you happen to know any option to enable create database to run only once ?

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [September 27, 2024, 2:05am UTC](https://forums.percona.com/t/galera-cluser-hung-state-with-wsrep-preparing-for-to-isolation/33359/6 "2024-09-27T02:05:20Z")

</div>

Are you using your main PXC as both application DB and Orchestrator backend? This is not recommended if that is the case.

Also, looking at the code you provided, it says “first time ever we talk to MySQL” so Orc should only be executing this DDL once. If this is executing often, then you should investigate if Orc is restarting on its own, or otherwise crashing/restarting.
