Unable to run due to rocksdb error

MySQL & MariaDB Percona Server for MySQL 8.0
,
1

hello. I have configured a new server using percona-server:8.0.28.
While migrating table configuration data to rocksdb, slowness (memory full) occurred and the server died.

After that, the server happens as below and doesn’t run.

2022-07-20T05:17:22.820372Z 0 [ERROR] [MY-000000] [Server] Plugin rocksdb reported: ‘Error detected in background, Status Code: 2, Status: Corruption: block checksum mismatch: stored = 2611920852, computed = 1320809697 in ./.rocksdb/070011.sst offset 12569567 size 9721’
2022-07-20T05:17:22.820400Z 0 [ERROR] [MY-000000] [Server] Plugin rocksdb reported: ‘BackgroundErrorReason: 1’
05:17:22 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: 18aaa142649c55cdd3da3a4b5bea39fe4cc9a679
Server Version: 8.0.28-19 Percona Server (GPL), Release 19, Revision 31e88966cd3

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x20cbe11]
/usr/sbin/mysqld(print_fatal_signal(int)+0x323) [0x11506e3]
/usr/sbin/mysqld(handle_fatal_signal+0xbd) [0x11507ad]
/lib64/libpthread.so.0(+0x12cf0) [0x7f34b90a1cf0]
/lib64/libc.so.6(gsignal+0x10f) [0x7f34b7450a4f]
/lib64/libc.so.6(abort+0x127) [0x7f34b7423db5]
/usr/lib64/mysql/plugin/ha_rocksdb.so(myrocks::Rdb_event_listener::OnBackgroundError(rocksdb::BackgroundErrorReason, rocksdb::Status*)+0x1ea) [0x7f346256a2ba]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::EventHelpers::NotifyOnBackgroundError(std::vector<std::shared_ptrrocksdb::EventListener, std::allocator<std::shared_ptrrocksdb::EventListener > > const&, rocksdb::BackgroundErrorReason, rocksdb::Status*, rocksdb::InstrumentedMutex*, bool*)+0x130) [0x7f3462196f30]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::ErrorHandler::SetBGError(rocksdb::Status const&, rocksdb::BackgroundErrorReason)+0x38b) [0x7f3462194e2b]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x1c18) [0x7f346214a0d8]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xea) [0x7f346214acea]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::DBImpl::BGWorkCompaction(void*)+0x3c) [0x7f346214b40c]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x27b) [0x7f346241d38b]
/usr/lib64/mysql/plugin/ha_rocksdb.so(rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x63) [0x7f346241d643]
/lib64/libstdc++.so.6(+0xc2ba3) [0x7f34b7e24ba3]
/lib64/libpthread.so.0(+0x81df) [0x7f34b90971df]
/lib64/libc.so.6(clone+0x43) [0x7f34b743bd83]
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

You may download the Percona Server operations manual by visiting
Percona Server for MySQL - Open Source MySQL Server Alternative. You may find information
in the manual which will help you identify the cause of the crash.

What should I do?

1 Like
2

It seems the crash corrupted the data. Best to restore from backup.

1 Like
4

For me this is not a big deal right now.
This is because the problem occurred during data transfer.

But not being able to recover this is a big problem.
There is no guarantee that this will not happen next time.
Even if some data is corrupted, I would like to try to recover it.

1 Like