Is there any way to debug keyring_vault plugin in pxc operator instance?

MySQL & MariaDB Percona Operator for MySQL
1

I’ve worked pxc (from helm chart 0.1.14) cluster (percona-kubernetes-operator 0.1.10). Currently I’m training to test a disaster recovery plan. I’ve created new cluster, restore Vault (HA Raft) from snapshot, but pxc can’t start:

[4] prod.pxc-db-pxc-0.mysqld-error.log: [1615882257.953232529, {"log"=>"2021-03-16T08:10:57.953042Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'Could not retrieve list of keys from Vault.'"}]
[5] prod.pxc-db-pxc-0.mysqld-error.log: [1615882257.953342834, {"log"=>"2021-03-16T08:10:57.953267Z 0 [ERROR] [MY-011371] [Server] Plugin keyring_vault reported: 'Error while loading keyring content. The keyring might be malformed'"}]
[6] prod.pxc-db-pxc-0.mysqld-error.log: [1615882257.953395061, {"log"=>"2021-03-16T08:10:57.953336Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'"}]

I’ve checked Vault token and permission from another sidecar container:

curl -H "X-Vault-Token: XXXXX" --cacert /tmp/vault.ca https://vault.prod.svc.cluster.local:8200/v1/pxc-secret/?list=true | jq

return

{
  "request_id": "9e5caeac-c463-dfc3-bf0c-129ad6b3e8dc",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "keys": [
      "MTRfcGVyY29uYV9yZWRvOjAwXw==",
      "NDhfSU5OT0RCS2V5LTE4NDc0OGY1LTYzZTQtMTFlYi05NDE2LTFhNGVjZWJjYjliMi0xMF8=",
      "NDhfSU5OT0RCS2V5LTE4NDc0OGY1LTYzZTQtMTFlYi05NDE2LTFhNGVjZWJjYjliMi0yMF8=",
      "NDhfSU5OT0RCS2V5LTRiMmEwNWRkLTYzZTQtMTFlYi1hY2VhLTEyYmQ0MjZjZTA4Yi0xMF8=",
...

Thanks

1 Like
2

Hello @Oleksandr_Slubskyi ,

could you pls check if the secret for Vault is present?

Is the secret correct?

What about the config itself - is it correctly restored? /etc/mysql/vault-keyring-secret/keyring_vault.conf

1 Like