SSL problem after upgrading PXC from 8.0.20-11.3 to 8.0.21-12.1

MySQL & MariaDB Percona Operator for MySQL
1

Our company uses the PXC Operator and it’s truly amazing!

Recently, we upgraded from 1.6.0 (MySQL image 8.0.20-11.3) to 1.7.0 (MySQL image 8.0.21-12.1) and noticed something strange in some of our applications.

I’ll be using a shortened way to refer to the base images for simplicity’s sake.

Whenever we used SSL in 8.0.20 (generated by cert-manager) everything worked and we could follow any and all MySQL documentation.

Now, once the cluster start using 8.0.21 we notice the SSL variables disappear from performance_schema.session_status…

We started thinking it was nothing because our users are required to login with X509 so the applications kept working and all was fine until some different support was needed and people started port-forwarding to the cluster’s service and/or directly to one of the replicas (as in pxc-0 and so on).

Some desktop applications, like MySQL Shell, never complain, but other applications like Heidi and another custom tool we use yield the error SSL Connection Error: unknown error number.

We tried a lot of things before deciding to write this post, but once we recreated the cluster with the old 1.6.0 & 8.0.20 images (keeping every single configuration unchanged, even SSL!) and all worked again we started getting frustrated to what even would cause the SSL configuration to differ between the newer version of the operator and the old one…

As far as our limited understating of PXC goes, the difference is the lack of any indication SSL is even enabled in the session (even if the connection goes through using MySQL Shell).

Not knowing if this is a bug or misconfiguration, we decided to ask here instead of opening a bug report over Jira…

2 Likes
2

This is the output from SELECT * FROM performance_schema.session_status WHERE VARIABLE_NAME LIKE '%ssl%'\G.

Please notice the lack of variables such as Ssl_version, Ssl_cipher_list, and others. We suspect the client errors are related to the missing configuration…

*************************** 1. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_accepts
VARIABLE_VALUE: 0
*************************** 2. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_active
VARIABLE_VALUE: 
*************************** 3. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_cipher
VARIABLE_VALUE: 
*************************** 4. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_cipher_list
VARIABLE_VALUE: 
*************************** 5. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_ctx_verify_depth
VARIABLE_VALUE: 18446744073709551615
*************************** 6. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_ctx_verify_mode
VARIABLE_VALUE: 5
*************************** 7. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_finished_accepts
VARIABLE_VALUE: 0
*************************** 8. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_server_not_after
VARIABLE_VALUE: May 16 18:34:53 2021 GMT
*************************** 9. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_server_not_before
VARIABLE_VALUE: Feb 15 18:34:53 2021 GMT
*************************** 10. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_verify_depth
VARIABLE_VALUE: 
*************************** 11. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_verify_mode
VARIABLE_VALUE: 
*************************** 12. row ***************************
 VARIABLE_NAME: Mysqlx_ssl_version
VARIABLE_VALUE: 
12 rows in set (0.0141 sec)
2 Likes
3

Hi @davi - thanks for your interest in the PXC Operator! I can confirm on PXC 8.0.21 in my own 1.7 environment that we do not set the mysqlx_* values.

 mysql> select @@version;
 +-------------+
 | @@version   |
 +-------------+
 | 8.0.21-12.1 |
 +-------------+
 1 row in set (0.00 sec)
 
 mysql> SELECT * FROM performance_schema.session_status WHERE VARIABLE_NAME LIKE '%ssl%';
 +------------------------------+--------------------------+
 | VARIABLE_NAME                | VARIABLE_VALUE           |
 +------------------------------+--------------------------+
 | Mysqlx_ssl_accepts           | 0                        |
 | Mysqlx_ssl_active            |                          |
 | Mysqlx_ssl_cipher            |                          |
 | Mysqlx_ssl_cipher_list       |                          |
 | Mysqlx_ssl_ctx_verify_depth  | 18446744073709551615     |
 | Mysqlx_ssl_ctx_verify_mode   | 5                        |
 | Mysqlx_ssl_finished_accepts  | 0                        |
 | Mysqlx_ssl_server_not_after  | Dec 31 23:59:59 9999 GMT |
 | Mysqlx_ssl_server_not_before | Feb  9 20:44:55 2021 GMT |
 | Mysqlx_ssl_verify_depth      |                          |
 | Mysqlx_ssl_verify_mode       |                          |
 | Mysqlx_ssl_version           |                          |
 +------------------------------+--------------------------+
 12 rows in set (0.00 sec)

If you still have access to your 1.6 environment can you paste the output from the same SELECT * FROM PERFORMANCE_SCHEMA.SESSION_STATUS statement?

1 Like
4

As you can see, in this cluster all SSL variables are there and for some reason they are not present if we upgrade the cluster to 8.0.21.

The log is way too big to fit the char limit so I’m attaching the file: log-8.0.20.txt (454.1 KB)

2 Likes