Hello everyone,
I’m working on integrating MySQL (Percona XtraDB Cluster 8.4.4-4.1) with HashiCorp Vault using the keyring_vault component. I’ve successfully used other keyring components in the cluster previously.
I’ve followed these steps to configure keyring_vault:
-
Manifest File (mysqld.my):
I created the mysqld.my file and placed it in the directory where the mysqld executable is located (i.e., /usr/sbin/).
mysqld.my content:
{
"components": "file://component_keyring_vault"
}
-
Component Configuration File (component_keyring_vault.cnf):
I created this file in the MySQL plugin directory (e.g., /usr/lib/mysql/plugin/).
component_keyring_vault.cnf content:
{
"read_local_config": false,
"vault_url": "http://10.169.100.127:8200",
"secret_mount_point": "secrets/mysql_vault",
"token": "hvs.mytokenhere", // Replaced with my actual token
"secret_mount_point_version": "AUTO",
"timeout": 15
}
Note: I’m intentionally using HTTP (no TLS) for initial debugging.
-
Vault Setup:
On the HashiCorp Vault side, I’ve successfully created a KV Secrets Engine version 2 (kv-v2) at the path secrets/mysql_vault and ensured the token has all necessary permissions (create, read, update, delete, list) for that path.
The Problem:
When I try to start the MySQL service (sudo systemctl start mysql), it fails to start. The most concerning part is that the MySQL error log (/var/log/mysql/error.log) remains completely empty, with no new entries whatsoever after the startup attempt. I can’t seem to figure out why MySQL isn’t even attempting to log errors.
It appears MySQL is crashing at a very early stage, before it even begins its logging process. Could I have misplaced the manifest file, or is there another extremely early point of failure that prevents log writing?
Any thoughts or guidance on how to debug this situation would be greatly appreciated!
Thanks in advance.
Hello Community! Unfortunately, I have the same issue with setting up keyring_vault. Could you please help to resolve it?
Simple verfication: mv /usr/sbin/mysqld.my /tmp/, then start MySQL. If it starts correctly, then yea, need to open bug report. Additionally, if replacing mysqld.my using keyring_file component works, yea, bug on vault component.
Do you have coredumps enabled on your OS? Try to configure this then start MySQL with vault. Hopefully a coredump is generated that will help our developers.
Hi, totally working fine without keyring_vault. Can u using in cnf file http? Or i must using https and crt? Mb some problems with http?
As a test, use your original mysqld.my file for Vault. In another terminal, use ngrep, or tcpdump to listen on the vault port. Then start mysql. See if there is any traffic. If none, and mysql silently crashes, edit mysqld.my and change to https and try again. If you see traffic, and mysql starts, then we know the issue has to do with http vs https, and probably needs to be documented that only https is supported.
I just did some rudimentary strace on Percona 8.4.4 using component_keyring_vault, and it crashes without any messages in mysql error log. What I see in the strace output is SIGSEV address mapping violations, whatever that means.
I will report this to our developers.
Hey there @Denis_Keldanovich,
Your component .cnf is invalid JSON. // comments are not allowed. The component plugin is crashing right after reading the file contents. Using ltrace I found the error message, and then searched through the source code. The error is only found in 1 location.
After removing the comment, mysql still doesn’t start for me, but it does output to the error log now. I’m assuming it’s because I don’t have vault setup. Can you remove the comment and try on your side?
Hello @Denis_Keldanovich,
I was able to get component_keyring_vault working in PS 8.4.5. Here is network sniff during a call from mysql, shown below. Vault is running inside a docker container, bound to port 8200, in dev mode (no ssl).
$ ngrep -d docker0 port 8200
interface: docker0 (172.17.0.0/255.255.0.0)
filter: ( port 8200 ) and ((ip || ip6) || (vlan && (ip || ip6)))
####
T 10.10.10.203:52328 -> 172.17.0.8:8200 [AP] #4
POST /v1/secrets/mysql_vault/data/N19BRVNfa2V5MThfbXNhbmRib3hAbG9jYWxob3N0 HTTP/1.1..Host: 10.10.10.203:8200..Accept: */*..X-Vault-Token:hvs.CAESIBJVHdPyLhRaBt2p9X4ucjY
y8hBAsXgVexYhKq8IgAMYGh4KHGh2cy5LcE5wOTM4OXUyNU5qMGZ5dmc1dTJETWs..Content-Type: application/json..Content-Length: 54....{"data":{"type":"AES","value":"U2VjcmV0IHN0cmluZ
w=="}}
##
mysql [localhost:8405] {msandbox} (test) > SELECT keyring_key_store('AES_key', 'AES', 'Secret string');
+------------------------------------------------------+
| keyring_key_store('AES_key', 'AES', 'Secret string') |
+------------------------------------------------------+
| 1 |
+------------------------------------------------------+
$ cat ~/dbdeployer/opt/mysql/ps8.4.5/lib/plugin/component_keyring_vault.cnf
{
"read_local_config": false,
"vault_url": "http://192.168.22.33:8200",
"secret_mount_point": "secrets/mysql_vault",
"token": "hvs.blahblahtokentoken",
"secret_mount_point_version": "AUTO",
"timeout": 15
}
$ docker exec -it vault sh
# vault secrets enable -version=2 -path=secrets/mysql_vault kv
# cat mysql.hcl
path "secrets/mysql_vault/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# vault policy write mysql-policy mysql.hcl
# vault token create -policy=mysql-policy >mysql_token.txt
# cat mysql_token.txt
Key Value
--- -----
token hvs.blahblahtokentoken
token_accessor UM8ZyzvPgAXLKZo9iaQmEpOZ
token_duration 768h
token_renewable true
token_policies ["default" "mysql-policy"]
identity_policies []
policies ["default" "mysql-policy"]
1 Like
Hi @matthewb , will check that and be back with feedback.
Hi, confirmed. Thats working, but cant understand whats wrong in first attempt((( Just changed Vault container to another(((