# Recent issues with "Waiting for bakup lock" **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Tags:** troubleshooting, mysql, percona\ **Created:** [June 27, 2020, 3:14pm UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723 "2020-06-27T15:14:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![tucj7](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@tucj7](https://forums.percona.com/u/tucj7)\ **Post date:** [June 27, 2020, 3:14pm UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/1 "2020-06-27T15:14:59Z")
Hi,Some help please. Over the last couple of days, we’re been having issues with our cluster. When a node goes down for some reason and starts coming back over SST, we’ve been getting a status of “Waiting for backup lock” when a “CREATE” or “DROP” table query is executed. Both are executed on InnoDB tables. ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/b089750343d466abe244f10d556cc117ddc9e283.png "Image: /uploads/percona1/original/1X/b089750343d466abe244f10d556cc117ddc9e283.png")   What this does is locks till the node is completely back up, which can take up to an hour. This renders the whole cluster unavailable during this time.Any ideas what is causing this? I’ve been running this cluster for 4+ years and this is the first time seeing this.Thanks! ---
**Author:** ![vaibhav\_upadhyay40](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vaibhav_upadhyay40/32/9_2.png) [@vaibhav\_upadhyay40](https://forums.percona.com/u/vaibhav_upadhyay40)\ **Post date:** [June 27, 2020, 4:32pm UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/2 "2020-06-27T16:32:32Z")
Hi @tucj7 It would be great if you can share version details of xtrabakup and cluster. ---
**Author:** ![tucj7](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@tucj7](https://forums.percona.com/u/tucj7)\ **Post date:** [June 27, 2020, 4:35pm UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/3 "2020-06-27T16:35:39Z")
Hi, No problem: Server version: 5.7.26-29-57 Percona XtraDB Cluster (GPL), Release rel29, Revision 03540a3, WSREP version 31.37, wsrep\_31.37xtrabackup version 2.4.15 based on MySQL server 5.7.19 Linux (x86\_64) (revision id: 544842a) ---
**Author:** ![tucj7](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@tucj7](https://forums.percona.com/u/tucj7)\ **Post date:** [June 28, 2020, 1:38am UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/4 "2020-06-28T01:38:04Z")
This is what’s running during the SST ![](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/1X/05eac0cc052a341789a004697e54b1837c87c61c.png "Image: /uploads/percona1/original/1X/05eac0cc052a341789a004697e54b1837c87c61c.png") ---
**Author:** ![tucj7](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@tucj7](https://forums.percona.com/u/tucj7)\ **Post date:** [June 28, 2020, 5:20am UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/5 "2020-06-28T05:20:51Z")
It looks like it could be related to this: [https://jira.percona.com/browse/PXC-2365](https://jira.percona.com/browse/PXC-2365 "Link: https://jira.percona.com/browse/PXC-2365")Can someone confirm that this is the same issue? Is there a resolution to this? Or, at least, a workaround? ---
**Author:** ![vaibhav\_upadhyay40](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vaibhav_upadhyay40/32/9_2.png) [@vaibhav\_upadhyay40](https://forums.percona.com/u/vaibhav_upadhyay40)\ **Post date:** [June 28, 2020, 8:43am UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/6 "2020-06-28T08:43:24Z")
Hi @tucj7 Yes you are right. As per my understanding i was expecting this issue on older version, but as you have shared it is not the issue in your case. Looking at above jira it is appears its a bug. May be someone can recommend interim solutions, if any. ---
**Author:** ![tucj7](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@tucj7](https://forums.percona.com/u/tucj7)\ **Post date:** [July 2, 2020, 7:10am UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/7 "2020-07-02T07:10:40Z")
@“lorraine.pocklington” is this something you could check out? ---
**Author:** ![DGB](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dgb/32/79_2.png) [@DGB](https://forums.percona.com/u/DGB)\ **Post date:** [July 14, 2020, 9:12am UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/8 "2020-07-14T09:12:54Z")
Hi Let me start by saying that the behaviour you are experience is expected by design. On 5.7, the xtrabackup command used for SST includes the parameter --lock-ddl ([https://github.com/percona/percona-xtradb-cluster/blob/5.7/scripts/wsrep\_sst\_xtrabackup-v2.sh#L1582](https://github.com/percona/percona-xtradb-cluster/blob/5.7/scripts/wsrep_sst_xtrabackup-v2.sh#L1582)) which will execute the LOCK TABLES FOR BACKUP command ([https://www.percona.com/doc/percona-xtrabackup/LATEST/xtrabackup\_bin/xbk\_option\_reference.html#cmdoption-lock-ddl](https://www.percona.com/doc/percona-xtrabackup/LATEST/xtrabackup_bin/xbk_option_reference.html#cmdoption-lock-ddl))  All this is to guarantee consistency of the backup. The fix is to avoid running DDLs on the cluster during the SST process (or at least at the beginning of it where all the backup lock happens)  ---
**Author:** ![tucj7](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@tucj7](https://forums.percona.com/u/tucj7)\ **Post date:** [July 14, 2020, 9:20am UTC](https://forums.percona.com/t/recent-issues-with-waiting-for-bakup-lock/7723/9 "2020-07-14T09:20:50Z")
Thanks for the feedback - that makes sense. The trouble I have is that when this happens it locks up the DB for the entire SST process, which currently runs around 1.5 hours. We have a bunch of jobs (and client initiated jobs) that run regularly that create/drop temp tables and it’s impossible to know when a node will go down and cause SST. It turns out this was likely happening due to a memory issue on one node. Do you have any recommendations on how to anticipate an SST and then push a change to crons/etc. to prevent DDL during this process? Or is there a better way to get around this. My constant fear is that this happens at critical times or after hours and causes considerable problems.