How does rotation of the root CA handled if performed by cert-manager?

MySQL & MariaDB Percona Operator for MySQL
1

Hello everyone,

I have a doubt related to certificates rotation, specifically to the root CA rotation. I’ve read the documentation on transport encryption

The doc states the following:

  • If a cert-manager is used, it should take care of updating the certificates. … This allows to reissue TLS certificates automatically on schedule and without downtime.

  • If you don’t use cert-manager and have created certificates manually, you can follow the next steps to perform a no-downtime update of these certificates if they are still valid. (followed by the steps)

Now, the steps suggested to rotate the certificates manually suggest to combine the old CA with a newly generated one, so that when the servers restart, they trust both new and old certificates since they have combined CAs bundled together, which totally makes sense.

But, if using Cert Manager, how does it work? Cert Manager will not bundle the old and new CAs in a combined way. It will directly replace the old secret with the new CA. The old will be gone.

What happens after the CA is rotated, forcing rotation of the certificates generated from it? How does the operator and PXC ndoes handle such a situation? When one of PXC node will be restarted, the CA of that node will be different from the others I think.

Or will the operator somehow perform some operations to avoid this situation?

2

Maybe I’ve found an answer to my own question.

When cert-manager renews the CA, it makes sure to use the same rsa private key of the previous one. So I think this means that each node certificate is valid both with the old and the new CAs, so during the rollout restart of the database pods, they will not have issues even if they are using different CAs.

3

Hi,

Let me try to say it with different words and hopefully that will help you.

Let’s assume this is “proper” and not adhoq. First off, you need a certificate authority. This certificate authority will sign a certificate authority request, CSR, for a newly created key for OpenBao.

FYI: when I work on this stuff internally I cheat by using the OpenBao key and certificate which is created by default when using Ubuntu as the CA in order to sign the certificate request for the openbao server. The reason is that openbao must identify itself in the CSR by IP address!

Here’s an example set of command creating both sets of key and signing the crt request.

echo "
[req]
default_bits=2048
prompt = no
default_md = sha256
distinguished_name = req_distinguished_name
req_extensions = v3_req
x509_extensions = v3_ca

[req_distinguished_name]
C = US
ST = Washington
L = Seattle
O = Percona
CN = pg1

[v3_ca]
basicConstraints = critical,CA:true

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
" > /opt/openbao/tls/openssl.cnf

# generate new key for openbao
openssl genrsa -out server.key 4096

# create openbao certificate request, CSR
openssl req -new -key server.key -out server.csr -config openssl.cnf

# sign CSR by OpenBao tls.key which is pre-existing
openssl x509 -req -in server.csr \
        -CA tls.crt -CAkey tls.key \
        -out server.crt -days 500 -sha256 \
        -extfile openssl.cnf -extensions v3_req

It’s understood that openbao uses server.key and server.crt in openbao.hcl.

echo "
ui = true

storage \"file\" {
  path = \"/opt/openbao/data\"
}

# HTTP listener
listener \"tcp\" {
  address = \"0.0.0.0:8200\"
  tls_cert_file = \"/opt/openbao/tls/server.crt\"
  tls_key_file  = \"/opt/openbao/tls/server.key\"
}
" > /etc/openbao/openbao.hcl

Now postgres can use the signing certificate that already exists i.e. tls.crt. Note that postgres and openbao are both on the same server in this example:

        select * from pg_tde_add_database_key_provider_vault_v2 (
            'percona02',
            'https://127.0.0.1:8200',
            'percona',
            '/var/lib/postgresql/.vault_token',
            '/var/lib/postgresql/tls.crt'
        );

Of course you need to refer to the OpenSSL documentation to get a true appreciation of the requirements.

Hope this helps.