# Ha\_err\_key\_not\_found

**URL:** <https://forums.percona.com/t/ha-err-key-not-found/6601>\
**Category:** Percona XtraDB Cluster 5.x\
**Created:** [September 28, 2018, 5:54pm UTC](https://forums.percona.com/t/ha-err-key-not-found/6601 "2018-09-28T17:54:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Goldberg](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@Mark\_Goldberg](https://forums.percona.com/u/Mark_Goldberg)\
**Post date:** [September 28, 2018, 5:54pm UTC](https://forums.percona.com/t/ha-err-key-not-found/6601/1 "2018-09-28T17:54:23Z")

</div>

We are running a 3-node Percona XtraDB Cluster Server version 5.6.39-83.1-56-log on Red Hat Enterprise Linux Server release 7.5 (Maipo).  
The cluster’s first node crashed, with the corresponding error messages appearing in the mysqld.err log file:

Retrying 4th time  
2018-09-27 14:04:00 4608 [ERROR] Slave SQL: Could not execute Delete\_rows event on table cgi\_d8\_prod.sessions; Can’t find record in ‘sessions’,  
Error\_code: 1032; handler error HA\_ERR\_KEY\_NOT\_FOUND; the event’s master log FIRST, end\_log\_pos 1300, Error\_code: 1032  
2018-09-27 14:04:00 4608 [Warning] WSREP: RBR event 3 Delete\_rows apply warning: 120, 67045668  
2018-09-27 14:04:00 4608 [ERROR] WSREP: Failed to apply trx: source: e8ae4b07-b7ae-11e8-b74c-8a9f3a425bd0 version: 3 local: 0 state: APPLYING f  
lags: 1 conn\_id: 184715 trx\_id: 3777747218 seqnos (l: 1170734, g: 67045668, s: 67045657, d: 67045533, ts: 1191591817699643)  
2018-09-27 14:04:00 4608 [ERROR] WSREP: Failed to apply trx 67045668 4 times  
2018-09-27 14:04:00 4608 [ERROR] WSREP: Node consistency compromised, aborting…

In addition, I was able to extract the following information for the associated GRA\_ binary log error file:

ERROR: Error in Log\_event::read\_log\_event(): ‘Found invalid event in binary log’, data\_len: 1157, event\_type: 32  
WARNING: The range of printed events ends with a row event or a table map event that does not have the STMT\_END\_F flag set. This might be because the last statement was not fully written to the log, or because you are using a --stop-position or --stop-datetime that refers to an event in the middle of a statement. The event(s) from the partial statement have not been written to output.

Has anyone on this forum encountered this issue, and if so, do you have a solution that you can share with us ?

---

<div class="post-metadata">

**Author:** ![przemek](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/przemek/32/3_2.png) [@przemek](https://forums.percona.com/u/przemek)\
**Post date:** [October 2, 2018, 5:08am UTC](https://forums.percona.com/t/ha-err-key-not-found/6601/2 "2018-10-02T05:08:01Z")

</div>

You should investigate why only this one node aborted. Error suggests that there was data inconsistency here as the row to be deleted did not exist on this node.  
GRA file is not a complete binary log, but rather a single RBR event, without proper header. You may decode by getting first a proper header from any binary log, like:

```auto
head -c 120 /var/lib/mysql/log-bin.000001 > GRA_HEADER
cat GRA_HEADER > gra_X_Y-bin.log
cat GRA_X_Y.log >> gra_X_Y-bin.log
mysqlbinlog -vvv gra_X_Y-bin.log

```

The usual source of data inconsistencies are: sql\_log\_bin=0 used, wsrep\_on=0, RSU method for incompatible DDLs, non-InnoDB tables, etc. If you have binlogs enabled, I suggest you dig through them for the Xid transaction saved with your GRA file and check at what point related rows were changed on the cluster nodes.