Hi @Anisha,
Apologies for making you wait, since I was away.
Please note I am working over your issue with my team. Please allow me sometime. Meanwhile, is it possible for you to provide me with your error logs (you can remove confidential data, if any )? Additionally, how you upgraded to 8.0.30 from 8.0.24 ? Was it in-place or you installed new mysql and then imported the backup ? Can you also confirm if there is any particular time when your server crashes at the time of executing this query or whenever you execute this, mysql crashes. Lastly, can you give me the output from pt-summary It will provide me some details of your system related to memory, for example.
Is it possible for you to use 8.0.31 and try testing on it and see if it crashes there as well ?
Hi Ankit,
Acknowledged. I will share you the error log details and pt-summary. I will get you the timestamp details of mysql crashes and see if any pattern.
It was In-place upgrade from 8.0.23 to 8.0.30 version.
Upgrade timestamp:
2023-01-05T20:16:54.667832Z 4 [System] [MY-013381] [Server] Server upgrade from ‘80023’ to ‘80030’ started.
2023-01-05T20:16:57.864740Z 4 [System] [MY-013381] [Server] Server upgrade from ‘80023’ to ‘80030’ completed.
This was the production database , we can test upgrading in lower environment to 8.0.31 version.
Thanks,
Anisha
acip_errlog.txt (1.4 MB)
Uploaded errorlog details. Could you share the pt-summary steps to collect the details.
Timestamp when server was crashed.
PROD:!:_rn3-mysqlp-ldb01:/home/mysqld/acip/data> egrep “crash” acip_a.err
2022-04-12T14:14:21.353284Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2022-04-12T14:14:21.359664Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-06T15:36:46.234963Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-06T15:36:46.247397Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-06T20:39:09.420796Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-06T20:39:09.431877Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-07T07:45:57.934802Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-07T07:45:57.946111Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-11T07:17:14.825993Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-11T07:17:14.836774Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-12T08:31:07.147543Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-12T08:31:07.153666Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-12T15:09:10.159044Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-12T15:09:10.165809Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-14T10:03:40.297040Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-14T10:03:40.304961Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-14T15:46:38.575495Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-14T15:46:38.581114Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-15T08:22:10.194469Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-15T08:22:10.201712Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
in the manual which will help you identify the cause of the crash.
2023-01-15T11:14:29.436549Z 0 [System] [MY-010229] [Server] Starting XA crash recovery…
2023-01-15T11:14:29.443635Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
Hi @Anisha ,
You can follow this link for pt-summary and please read all the guide and warning before running this.
I still need a test case wherein I can try to reproduce this issue at my end. Can you give me one.
Is it possible to have a webex or zoom to further debug this issue
Hello @Sanjay_Sheoran,
These forums are not official support. They are provided for community assistance and volunteer time by Percona employees. If you would like more dedicated, one-on-one assistance in debugging this matter you will need a Support Contract.
@matthewb It appears to be the problem is with “select limit update …” PT_query_expression::contextualize block.
Upon looking at the source " select->master_query_expression(); " has changed in 8.0.30 for this block compared to 8.0.23.
Since there is no control over app query, is there a workaround to the problem ?
Thanks in advance!
Yes, the workaround is to downgrade to 8.0.23.
Then, submit a repeatable test case to Percona so that we can investigate further, and then submit a repeatable test case to bugs.mysql.com, so that Oracle can investigate further.
Without a repeatable test case, or a support contract, there isn’t much more that can be done via the forums. Oracle will close your bug immediately without a repeatable test case, so be aware.
Hi Matthew,
Can the server be downgraded directly to 8.0.29 or 8.0.23 from 8.0.30 by just replacing the binaries and starting the server?
Most likely no. Look MySQL’s error log from your most recent start. You’ll see the current version of the data dictionary. That is the version you can downgrade to by replacing the binaries. That is not recommended, however. The best practice is to restore the 8.0.23 backup you took before upgrading. You can spin up a replica using 8.0.23 and that backup, connected to your current 8.0.30 source to catch up any changes in data, then do a promotion.
HI Matthew,
Fyi, we saw the DB crashing only with 8.0.30 and it works fine for 8.0.29. So can you help to see if any changes were done that could crash the db on select query with limit update clause in 8.0.30.
@Hussain_Patel I’m not a developer so I can’t tell you the code-level differences between .29 and .30. Thankfully, Percona Server is open source so you can go to github and see the differences if you like. Based on some conversations I’ve had with my fellow consultants, we do recommend you downgrade to .29 for now. This will require either restoring a previous backup to a new .29 install, or do a logical dump from .30 to .29.
Yes we have done rolling back But due to compliance we need to upgrade to 8.0.30 in future too and if this bug is not resolved we cant.
we have filed this bug here
https://jira.percona.com/projects/PS/issues/PS-8573