Percona Server 5.7 - ERROR 3185 on one DRE encrypted table only; others work fine

MySQL & MariaDB Percona Server for MySQL 5.7
1

Problem:

Without any changes at Mysql DB OR table level, one DRE(Data At Rest Encryption) table consistently fails where as other encrypted tables are accessible,

SELECT COUNT(*) FROM myschema.user_profile;
-- ERROR 3185 (HY000): Can't find master key from keyring, please check in the server log
-- if a keyring plugin is loaded and initialized successfully.

Other encrypted tables in the same schema work, for example:

SELECT COUNT(*) FROM myschema.account;          -- OK
SELECT COUNT(*) FROM myschema.name;             -- OK
SELECT COUNT(*) FROM myschema.communication;    -- OK

keyring_file shows ACTIVE in SHOW PLUGINS. keyring_file_data points at the expected path.

Error log (relevant lines):

On first access to the failing table:

[ERROR] InnoDB: Failed to find tablespace for table myschema.user_profile in the cache.
        Attempting to load the tablespace with space id 81
[ERROR] InnoDB: Encryption can't find master key, please check the keyring plugin is loaded.
[ERROR] InnoDB: Encryption information in datafile: ./myschema/user_profile.ibd can't be decrypted,
        please check if a keyring plugin is loaded and initialized successfully.

Keyring contents

cat /etc/mysql/conf.d/my.cnf  | grep -i keyring
  early-plugin-load = "keyring_file.so"
  keyring_file_data = "/mnt/keyring"


$ strings /path/to/keyring | grep INNODBKey
INNODBKey-<uuid-a>-2AES
INNODBKey-<uuid-b>-1AES

$ wc -c /path/to/keyring
283

Notes:

  • Two different server_uuid values in the keyring (uuid-a vs uuid-b).

  • Key IDs are 1 and 2 on different UUIDs (not simply -1 and -2 on the same instance).

  • We have not run ALTER INSTANCE ROTATE INNODB MASTER KEY (5.7 — rotation API not available anyway).

  • Why tmpfs: The live keyring sits on tmpfs (/mnt/keyring) so the master keys are only in memory and are not stored in plaintext on persistent disk with the datadir.

    How it loads: That path is empty after a reboot. Before or right after startup, the keyring is rebuilt by decrypting a separate encrypted backup file on disk (e.g. under the datadir) and writing it to /mnt/keyring; MySQL’s keyring_file plugin then reads only from that in-memory file at runtime.

Can you please help us with below questions:

  1. Can one table hit 3185 while others work only because its .ibd references an INNODBKey-<uuid>-<id> missing from the keyring, while working tables reference keys that are present and how this scenario can happen?

  2. What causes two different UUIDs in a single keyring_file without rotation , keyring merge, restore from another host?

  3. On 5.7.39, best way to see which key a failing tablespace expects INNODB_TABLESPACES_ENCRYPTION (table is empty only), or something else?

  4. Can a partial keyring restore after tmpfs loss cause “most tables OK, one table always 3185”?

  5. If the key is missing , is recovery only a full keyring restore? Any way to fix a single table without the original master key?

  6. For partitioned encrypted tables , can one partition fail while others work, or does the whole table fail together?

Tried: Verified plugin/path; restored keyring from backup; tested all encrypted tables explicitly — only one fails.

Looking for: Root causes, detection steps, and how to prevent/mitigate (especially with tmpfs keyring).

Thanks.