Percona Server 5.7 - ERROR 3185 on one DRE encrypted table only; others work fine

pravata_dash · September 9, 2026, 4:12pm

Problem:

Without any changes at Mysql DB OR table level, one DRE(Data At Rest Encryption) table consistently fails where as other encrypted tables are accessible,

SELECT COUNT(*) FROM myschema.user_profile;
-- ERROR 3185 (HY000): Can't find master key from keyring, please check in the server log
-- if a keyring plugin is loaded and initialized successfully.

Other encrypted tables in the same schema work, for example:

SELECT COUNT(*) FROM myschema.account;          -- OK
SELECT COUNT(*) FROM myschema.name;             -- OK
SELECT COUNT(*) FROM myschema.communication;    -- OK

keyring_file shows ACTIVE in SHOW PLUGINS. keyring_file_data points at the expected path.

Error log (relevant lines):

On first access to the failing table:

[ERROR] InnoDB: Failed to find tablespace for table myschema.user_profile in the cache.
        Attempting to load the tablespace with space id 81
[ERROR] InnoDB: Encryption can't find master key, please check the keyring plugin is loaded.
[ERROR] InnoDB: Encryption information in datafile: ./myschema/user_profile.ibd can't be decrypted,
        please check if a keyring plugin is loaded and initialized successfully.

Keyring contents

cat /etc/mysql/conf.d/my.cnf  | grep -i keyring
  early-plugin-load = "keyring_file.so"
  keyring_file_data = "/mnt/keyring"


$ strings /path/to/keyring | grep INNODBKey
INNODBKey-<uuid-a>-2AES
INNODBKey-<uuid-b>-1AES

$ wc -c /path/to/keyring
283

Notes:

Two different server_uuid values in the keyring (uuid-a vs uuid-b).
Key IDs are 1 and 2 on different UUIDs (not simply -1 and -2 on the same instance).
We have not run ALTER INSTANCE ROTATE INNODB MASTER KEY (5.7 — rotation API not available anyway).
Why tmpfs: The live keyring sits on tmpfs (/mnt/keyring) so the master keys are only in memory and are not stored in plaintext on persistent disk with the datadir.

How it loads: That path is empty after a reboot. Before or right after startup, the keyring is rebuilt by decrypting a separate encrypted backup file on disk (e.g. under the datadir) and writing it to /mnt/keyring; MySQL’s keyring_file plugin then reads only from that in-memory file at runtime.

Can you please help us with below questions:

Can one table hit 3185 while others work only because its .ibd references an INNODBKey-<uuid>-<id> missing from the keyring, while working tables reference keys that are present and how this scenario can happen?
What causes two different UUIDs in a single keyring_file without rotation , keyring merge, restore from another host?
On 5.7.39, best way to see which key a failing tablespace expects INNODB_TABLESPACES_ENCRYPTION (table is empty only), or something else?
Can a partial keyring restore after tmpfs loss cause “most tables OK, one table always 3185”?
If the key is missing , is recovery only a full keyring restore? Any way to fix a single table without the original master key?
For partitioned encrypted tables , can one partition fail while others work, or does the whole table fail together?

Tried: Verified plugin/path; restored keyring from backup; tested all encrypted tables explicitly — only one fails.

Looking for: Root causes, detection steps, and how to prevent/mitigate (especially with tmpfs keyring).

Thanks.

hai.nguyen · September 14, 2026, 2:03am

Hi @pravata_dash,

Can one table hit 3185 while others work only because its .ibd references an INNODBKey-- missing from the keyring, while working tables reference keys that are present and how this scenario can happen?

What causes two different UUIDs in a single keyring_file without rotation , keyring merge, restore from another host?

Yes, this is possible. I just tested and this can happen if the server_uuid gets changed (for example delete the auto.cnf file and restart the server). Then a new master encryption key will be created and will be used for new encrypted tables from that point. However, this is not a rotation, and the old existing encrypted tables will continue using the old master encryption key. Since you have keys with different UUIDs in the keyring file, it is likely that this is the case. Somehow the keyring is missing the key that table myschema.user_profile needs, so the server cannot open that table, but other tables are fine.

On 5.7.39, best way to see which key a failing tablespace expects INNODB_TABLESPACES_ENCRYPTION (table is empty only), or something else?

You can use this script to check the encryption information in page0 header of the ibd data file. For example:

# git clone https://github.com/Percona-Lab/eng-scripts.git

# cd eng-scripts

# ./innodb_page_header.sh /var/lib/mysql/test/t1.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd
SPACE_ID of tablespace:          30

input for decode_flags is 8225
FLAGS: 0
ANTELOPE: 1
COMPRESSED: NO
ATOMIC_BLOBS: 1
DATADIR: 0
SHARED_TABLESPACE: 0
TEMPORARY TABLESPACE: 0
ENCRYPTION: 1
SDI: 0
PHYSICAL_PAGE_SIZE: 16384
UNCOMP_PAGE_SIZE  : 16384
ENCRYPTION OFFSET : 10390

------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000001 ( 1 )
SERVER UUID: fc3befa0-afd8-11f1-84e1-3ed3ee773c97
Key:
000028c1: 4612 652e 1c7e da37 a7c3 50df 9c7a c3e3  F.e..~.7..P..z..
000028d1: 8946 2afe 1bbc 413e ba0e 6e87 4608 133f  .F*...A>..n.F..?
iv:
000028e1: 1ba4 44bb 862b b057 2b86 4411 41be 6e47  ..D..+.W+.D.A.nG
000028f1: 7e2d caca eb52 a901 6fd4 7dd2 3cb9 14eb  ~-...R..o.}.<...

The key this table needs is:

# INNODBKey-<SERVER UUID>-<MASTER_KEY_ID>
INNODBKey-fc3befa0-afd8-11f1-84e1-3ed3ee773c97-1

Can a partial keyring restore after tmpfs loss cause “most tables OK, one table always 3185”?

Yes, it is likely that after a server reboot, a stale copy of the keyring file was restored. If that file doesn’t have the key that table myschema.user_profile needs, that table will fail and the rest will be fine.

If the key is missing , is recovery only a full keyring restore? Any way to fix a single table without the original master key?

No, there is no way to fix a table without the master encryption key that it needs. Check which master encryption key this table is using with the mentioned script above and see if you still have a backup of keyring file containing that master encryption key.

For partitioned encrypted tables , can one partition fail while others work, or does the whole table fail together?

Each partition is its own .ibd with its own encryption header. One partition can fail while others work if that partition was created later (e.g. ADD PARTITION after a UUID change) and its master key is missing. If they were all created under the same master key, they fail or work together.

pravata_dash · September 14, 2026, 5:52pm

@hai.nguyen
Thanks for the details, will get those checked.

pravata_dash · September 16, 2026, 10:22am

Hi @hai.nguyen
we root-caused the generation of the new master keyring version in the existing file. The new master keyring version is being created immediately following a single MySQL service restart. We are currently investigating why the restart is triggering this behavior.

On a new cluster, after the first encrypted table was created, the keyring contained only one InnoDB master key:

INNODBKey-<server_uuid>-1

Keyring size was 155 bytes; mysqld had been up ~10 days with no keyring changes.

We then performed a clean MySQL restart (graceful stop/start — same config, same keyring_file_data path, keyring file still present on tmpfs). We did not run ALTER INSTANCE ROTATE INNODB MASTER KEY and did not encrypt any new tables before checking the keyring.

After restart, the keyring contained both keys for the same server_uuid:

INNODBKey-<server_uuid>-2
INNODBKey-<server_uuid>-1

Size grew to 283 bytes; keyring mtime matched startup time.

Further restarts do not add -3, -4, etc. — only this first post-creation restart produced a second master key.

Impact we observed:
Using your earlier sharedinnodb_page_header.sh on encrypted .ibd files, all tables encrypted before that restart reference INNODBKey-<server_uuid>-1 in their tablespace header. Tables encrypted after the restart reference INNODBKey-<server_uuid>-2. Our keyring backup was taken at initial DRE setup and therefore contained only -1. After a keyring restore from that backup, tables still on -1 worked; tables bound to -2 failed with ERROR 3185 (Can't find master key from keyring).

Questions

Is it normal for a clean mysqld restart to create a new InnoDB master key (-2) exactly once (so the keyring ends up with two keys total and later restarts add nothing further), without us running ALTER INSTANCE ROTATE INNODB MASTER KEY in that session?
If yes, what startup condition triggers that one-time creation?
Is there any way we can stop the new master keyring generation part of mysql restart?

hai.nguyen · September 17, 2026, 4:16am

Hi @pravata_dash,

Is it normal for a clean mysqld restart to create a new InnoDB master key (-2) exactly once (so the keyring ends up with two keys total and later restarts add nothing further), without us running ALTER INSTANCE ROTATE INNODB MASTER KEY in that session?

If yes, what startup condition triggers that one-time creation?

I just tested and confirmed this behavior in PS 5.7. If the latest master key is -1, then a restart will always create a new key -2. However, again this is not a rotation so the existing encrypted tables will still be using the old -1 master key, and new encrypted tables will use the new -2 key:

mysql> create table test.t1 (id int auto_increment primary key) encryption='Y';
insert into test.t1 values (1);
Query OK, 0 rows affected (0.05 sec)

Query OK, 1 row affected (0.01 sec)

# strings /var/lib/mysql-keyring/keyring
Keyring file version:1.0
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES

# systemctl restart mysqld

# strings /var/lib/mysql-keyring/keyring
Keyring file version:1.0
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-2AES
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES

mysql> create table test.t2 (id int auto_increment primary key) encryption='Y';
insert into test.t2 values (1);
Query OK, 0 rows affected (0.08 sec)

Query OK, 1 row affected (0.01 sec)


# /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd
...
------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000001 ( 1 )
SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97
...

# /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t2.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t2.ibd
...
------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 )
SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97
...

Is there any way we can stop the new master keyring generation part of mysql restart?

There is no option to stop this behavior. Looking at the source code, this seems to be intended - percona-server/storage/innobase/handler/ha_innodb.cc at Percona-Server-5.7.44-57 · percona/percona-server · GitHub. Note that I have only checked on PS 5.7.

The only workaround I see is to trigger the -2 master key creation yourself by manually rotating the master key (with ALTER INSTANCE ROTATE INNODB MASTER KEY). In this case, this is an actual rotation and the new -2 master key will be used for all encrypted tables (including the existing ones). You don’t want some tables to use a different master key than the others. Then just make sure you back up the keyring file whenever you manually rotate the master key.

mysql> create table test.t1 (id int auto_increment primary key) encryption='Y';
insert into test.t1 values (1);
Query OK, 0 rows affected (0.03 sec)

Query OK, 1 row affected (0.01 sec)

mysql> exit
Bye

# strings /var/lib/mysql-keyring/keyring
Keyring file version:1.0
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES

# /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd
...
------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000001 ( 1 )
SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97
...

mysql> ALTER INSTANCE ROTATE INNODB MASTER KEY;
Query OK, 0 rows affected (0.02 sec)

# strings /var/lib/mysql-keyring/keyring
Keyring file version:1.0
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-2AES
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES

# /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd
...
------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 )
SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97
...

# systemctl restart mysqld

# strings /var/lib/mysql-keyring/keyring
Keyring file version:1.0
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-2AES
INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES


mysql> create table test.t2 (id int auto_increment primary key) encryption='Y';
insert into test.t2 values (1);
Query OK, 0 rows affected (0.04 sec)

Query OK, 1 row affected (0.01 sec)

mysql> exit
Bye

# /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd
...
------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 )
SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97
...

# /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t2.ibd
Reading FSP_FLAGS of /var/lib/mysql/test/t2.ibd
...
------ Encryption ------
ENCRYPTION_KEY_MAGIC: lCC
10393
MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 )
SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97
...

pravata_dash · September 19, 2026, 10:42am

Thanks a lot. We will get the suggestions reviewed.

Topic		Replies	Views
Very undesiderable side effects in case of key vault plugin startup failure Percona Server for MySQL 5.7	6	1252	May 30, 2018
InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. Percona XtraBackup	0	3178	August 22, 2019
How to rotate keys when using component_keyring Uncategorized	10	1956	October 31, 2022
Configuring the Component Keyring in Percona Server and PXC … \| Percona Community Community website	7	240	March 25, 2026
Problem when trying creating backup in encrypted database Percona XtraBackup percona	7	1673	September 10, 2021

Percona Server 5.7 - ERROR 3185 on one DRE encrypted table only; others work fine

Related topics