# Percona Server 5.7 - ERROR 3185 on one DRE encrypted table only; others work fine **URL:** \ **Category:** Percona Server for MySQL 5.7\ **Tags:** percona\ **Created:** [September 9, 2026, 4:12pm UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255 "2026-09-09T16:12:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![pravata\_dash](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pravata_dash/32/12098_2.png) [@pravata\_dash](https://forums.percona.com/u/pravata_dash)\ **Post date:** [September 9, 2026, 4:12pm UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255/1 "2026-09-09T16:12:51Z")
**Problem:** Without any changes at Mysql DB OR table level, one DRE(Data At Rest Encryption) table consistently fails where as other encrypted tables are accessible, ```auto SELECT COUNT(*) FROM myschema.user_profile; -- ERROR 3185 (HY000): Can't find master key from keyring, please check in the server log -- if a keyring plugin is loaded and initialized successfully. ``` Other encrypted tables in the same schema work, for example: ```auto SELECT COUNT(*) FROM myschema.account; -- OK SELECT COUNT(*) FROM myschema.name; -- OK SELECT COUNT(*) FROM myschema.communication; -- OK ``` `keyring_file` shows ACTIVE in `SHOW PLUGINS`. `keyring_file_data` points at the expected path. * * * **Error log (relevant lines):** On first access to the failing table: ```auto [ERROR] InnoDB: Failed to find tablespace for table myschema.user_profile in the cache. Attempting to load the tablespace with space id 81 [ERROR] InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. [ERROR] InnoDB: Encryption information in datafile: ./myschema/user_profile.ibd can't be decrypted, please check if a keyring plugin is loaded and initialized successfully. ``` Keyring contents ```auto cat /etc/mysql/conf.d/my.cnf | grep -i keyring early-plugin-load = "keyring_file.so" keyring_file_data = "/mnt/keyring" $ strings /path/to/keyring | grep INNODBKey INNODBKey--2AES INNODBKey--1AES $ wc -c /path/to/keyring 283 ``` Notes: - Two different `server_uuid` values in the keyring (`uuid-a` vs `uuid-b`). - Key IDs are 1 and 2 on different UUIDs (not simply `-1` and `-2` on the same instance). - We have not run `ALTER INSTANCE ROTATE INNODB MASTER KEY` (5.7 — rotation API not available anyway). - Why tmpfs: The live keyring sits on tmpfs (`/mnt/keyring`) so the master keys are only in memory and are not stored in plaintext on persistent disk with the datadir. * * * Can you please help us with below questions: 1. Can one table hit 3185 while others work only because its `.ibd` references an `INNODBKey--` missing from the keyring, while working tables reference keys that _are_ present and how this scenario can happen? 2. What causes two different UUIDs in a single `keyring_file` without rotation , keyring merge, restore from another host? 3. On 5.7.39, best way to see which key a failing tablespace expects `INNODB_TABLESPACES_ENCRYPTION` (table is empty only), or something else? 4. Can a partial keyring restore after tmpfs loss cause “most tables OK, one table always 3185”? 5. If the key is missing , is recovery only a full keyring restore? Any way to fix a single table without the original master key? 6. For partitioned encrypted tables , can one partition fail while others work, or does the whole table fail together? * * * Tried: Verified plugin/path; restored keyring from backup; tested all encrypted tables explicitly — only one fails. Looking for: Root causes, detection steps, and how to prevent/mitigate (especially with tmpfs keyring). Thanks. ---
**Author:** ![hai.nguyen](https://avatars.discourse-cdn.com/v4/letter/h/41988e/32.png) [@hai.nguyen](https://forums.percona.com/u/hai.nguyen)\ **Post date:** [September 14, 2026, 2:03am UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255/2 "2026-09-14T02:03:00Z")
Hi @pravata_dash, > 1. Can one table hit 3185 while others work only because its .ibd references an INNODBKey-- missing from the keyring, while working tables reference keys that are present and how this scenario can happen? > 2. What causes two different UUIDs in a single keyring\_file without rotation , keyring merge, restore from another host? Yes, this is possible. I just tested and this can happen if the server\_uuid gets changed (for example delete the auto.cnf file and restart the server). Then a new master encryption key will be created and will be used for new encrypted tables from that point. However, this is not a rotation, and the old existing encrypted tables will continue using the old master encryption key. Since you have keys with different UUIDs in the keyring file, it is likely that this is the case. Somehow the keyring is missing the key that table myschema.user\_profile needs, so the server cannot open that table, but other tables are fine. > 1. On 5.7.39, best way to see which key a failing tablespace expects `INNODB_TABLESPACES_ENCRYPTION` (table is empty only), or something else? You can use this [script](https://github.com/Percona-Lab/eng-scripts/blob/master/innodb_page_header.sh) to check the encryption information in page0 header of the ibd data file. For example: ```auto # git clone https://github.com/Percona-Lab/eng-scripts.git # cd eng-scripts # ./innodb_page_header.sh /var/lib/mysql/test/t1.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd SPACE_ID of tablespace: 30 input for decode_flags is 8225 FLAGS: 0 ANTELOPE: 1 COMPRESSED: NO ATOMIC_BLOBS: 1 DATADIR: 0 SHARED_TABLESPACE: 0 TEMPORARY TABLESPACE: 0 ENCRYPTION: 1 SDI: 0 PHYSICAL_PAGE_SIZE: 16384 UNCOMP_PAGE_SIZE : 16384 ENCRYPTION OFFSET : 10390 ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000001 ( 1 ) SERVER UUID: fc3befa0-afd8-11f1-84e1-3ed3ee773c97 Key: 000028c1: 4612 652e 1c7e da37 a7c3 50df 9c7a c3e3 F.e..~.7..P..z.. 000028d1: 8946 2afe 1bbc 413e ba0e 6e87 4608 133f .F*...A>..n.F..? iv: 000028e1: 1ba4 44bb 862b b057 2b86 4411 41be 6e47 ..D..+.W+.D.A.nG 000028f1: 7e2d caca eb52 a901 6fd4 7dd2 3cb9 14eb ~-...R..o.}.<... ``` The key this table needs is: ```auto # INNODBKey-- INNODBKey-fc3befa0-afd8-11f1-84e1-3ed3ee773c97-1 ``` > 1. Can a partial keyring restore after tmpfs loss cause “most tables OK, one table always 3185”? Yes, it is likely that after a server reboot, a stale copy of the keyring file was restored. If that file doesn’t have the key that table myschema.user\_profile needs, that table will fail and the rest will be fine. > 1. If the key is missing , is recovery only a full keyring restore? Any way to fix a single table without the original master key? No, there is no way to fix a table without the master encryption key that it needs. Check which master encryption key this table is using with the mentioned script above and see if you still have a backup of keyring file containing that master encryption key. > 1. For partitioned encrypted tables , can one partition fail while others work, or does the whole table fail together? Each partition is its own .ibd with its own encryption header. One partition can fail while others work if that partition was created later (e.g. ADD PARTITION after a UUID change) and its master key is missing. If they were all created under the same master key, they fail or work together. ---
**Author:** ![pravata\_dash](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pravata_dash/32/12098_2.png) [@pravata\_dash](https://forums.percona.com/u/pravata_dash)\ **Post date:** [September 14, 2026, 5:52pm UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255/3 "2026-09-14T17:52:38Z")
@hai.nguyen Thanks for the details, will get those checked. ---
**Author:** ![pravata\_dash](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pravata_dash/32/12098_2.png) [@pravata\_dash](https://forums.percona.com/u/pravata_dash)\ **Post date:** [September 16, 2026, 10:22am UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255/4 "2026-09-16T10:22:05Z")
Hi @hai.nguyen we root-caused the generation of the new master keyring version in the existing file. The new master keyring version is being created immediately following a single MySQL service restart. We are currently investigating why the restart is triggering this behavior. On a new cluster, after the first encrypted table was created, the keyring contained only one InnoDB master key: ```auto INNODBKey--1 ``` Keyring size was 155 bytes; mysqld had been up ~10 days with no keyring changes. We then performed a clean MySQL restart (graceful stop/start — same config, same `keyring_file_data` path, keyring file still present on tmpfs). We did not run `ALTER INSTANCE ROTATE INNODB MASTER KEY` and did not encrypt any new tables before checking the keyring. After restart, the keyring contained both keys for the same `server_uuid`: ```bash INNODBKey--2 INNODBKey--1 ``` Size grew to 283 bytes; keyring `mtime` matched startup time. Further restarts do not add `-3`, `-4`, etc. — only this first post-creation restart produced a second master key. **Impact we observed:** Using your earlier shared`innodb_page_header.sh` on encrypted `.ibd` files, all tables encrypted before that restart reference `INNODBKey--1` in their tablespace header. Tables encrypted after the restart reference `INNODBKey--2`. Our keyring backup was taken at initial DRE setup and therefore contained only `-1`. After a keyring restore from that backup, tables still on `-1` worked; tables bound to `-2` failed with ERROR 3185 (`Can't find master key from keyring`). **Questions** 1. Is it normal for a clean mysqld restart to create a new InnoDB master key (`-2`) exactly once (so the keyring ends up with two keys total and later restarts add nothing further), without us running `ALTER INSTANCE ROTATE INNODB MASTER KEY` in that session? 2. If yes, what startup condition triggers that one-time creation? 3. Is there any way we can stop the new master keyring generation part of mysql restart? ---
**Author:** ![hai.nguyen](https://avatars.discourse-cdn.com/v4/letter/h/41988e/32.png) [@hai.nguyen](https://forums.percona.com/u/hai.nguyen)\ **Post date:** [September 17, 2026, 4:16am UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255/5 "2026-09-17T04:16:00Z")
Hi @pravata_dash, > 1. Is it normal for a clean mysqld restart to create a new InnoDB master key (`-2`) exactly once (so the keyring ends up with two keys total and later restarts add nothing further), without us running `ALTER INSTANCE ROTATE INNODB MASTER KEY` in that session? > 2. If yes, what startup condition triggers that one-time creation? I just tested and confirmed this behavior in PS 5.7. If the latest master key is -1, then a restart will always create a new key -2. However, again this is not a rotation so the existing encrypted tables will still be using the old -1 master key, and new encrypted tables will use the new -2 key: ```auto mysql> create table test.t1 (id int auto_increment primary key) encryption='Y'; insert into test.t1 values (1); Query OK, 0 rows affected (0.05 sec) Query OK, 1 row affected (0.01 sec) # strings /var/lib/mysql-keyring/keyring Keyring file version:1.0 INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES # systemctl restart mysqld # strings /var/lib/mysql-keyring/keyring Keyring file version:1.0 INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-2AES INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES mysql> create table test.t2 (id int auto_increment primary key) encryption='Y'; insert into test.t2 values (1); Query OK, 0 rows affected (0.08 sec) Query OK, 1 row affected (0.01 sec) # /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd ... ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000001 ( 1 ) SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97 ... # /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t2.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t2.ibd ... ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 ) SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97 ... ``` > 1. Is there any way we can stop the new master keyring generation part of mysql restart? There is no option to stop this behavior. Looking at the source code, this seems to be intended - [percona-server/storage/innobase/handler/ha\_innodb.cc at Percona-Server-5.7.44-57 · percona/percona-server · GitHub](https://github.com/percona/percona-server/blob/Percona-Server-5.7.44-57/storage/innobase/handler/ha_innodb.cc#L4064-L4087). Note that I have only checked on PS 5.7. The only workaround I see is to trigger the -2 master key creation yourself by manually rotating the master key (with `ALTER INSTANCE ROTATE INNODB MASTER KEY`). In this case, this is an actual rotation and the new -2 master key will be used for all encrypted tables (including the existing ones). You don’t want some tables to use a different master key than the others. Then just make sure you back up the keyring file whenever you manually rotate the master key. ```auto mysql> create table test.t1 (id int auto_increment primary key) encryption='Y'; insert into test.t1 values (1); Query OK, 0 rows affected (0.03 sec) Query OK, 1 row affected (0.01 sec) mysql> exit Bye # strings /var/lib/mysql-keyring/keyring Keyring file version:1.0 INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES # /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd ... ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000001 ( 1 ) SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97 ... mysql> ALTER INSTANCE ROTATE INNODB MASTER KEY; Query OK, 0 rows affected (0.02 sec) # strings /var/lib/mysql-keyring/keyring Keyring file version:1.0 INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-2AES INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES # /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd ... ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 ) SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97 ... # systemctl restart mysqld # strings /var/lib/mysql-keyring/keyring Keyring file version:1.0 INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-2AES INNODBKey-fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97-1AES mysql> create table test.t2 (id int auto_increment primary key) encryption='Y'; insert into test.t2 values (1); Query OK, 0 rows affected (0.04 sec) Query OK, 1 row affected (0.01 sec) mysql> exit Bye # /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t1.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t1.ibd ... ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 ) SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97 ... # /eng-scripts/innodb_page_header.sh /var/lib/mysql/test/t2.ibd Reading FSP_FLAGS of /var/lib/mysql/test/t2.ibd ... ------ Encryption ------ ENCRYPTION_KEY_MAGIC: lCC 10393 MASTER_KEY_ID: 00000000000000000000000000000010 ( 2 ) SERVER UUID: fd5ff8bb-afdf-11f1-9e82-3ed3ee773c97 ... ``` ---
**Author:** ![pravata\_dash](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pravata_dash/32/12098_2.png) [@pravata\_dash](https://forums.percona.com/u/pravata_dash)\ **Post date:** [September 19, 2026, 10:42am UTC](https://forums.percona.com/t/percona-server-5-7-error-3185-on-one-dre-encrypted-table-only-others-work-fine/41255/6 "2026-09-19T10:42:44Z")
Thanks a lot. We will get the suggestions reviewed.