How to disable TLS for the server

Gerwin_van_de_Steeg · August 7, 2025, 6:41am

Description:

I have the need to disable the TLS functions on an instance of MySQL created by the Operator and have gone through the details as documented however the clients are still connecting with TLS. Since the deployment uses HAProxy to load balance things via it’s service using an AWS Network Load Balancer, is this the source of the TLS connections from the clients (the clients run outside the Kubernetes cluster in a different VPC)

The consumer with the TLS problem cannot be configured to disable TLS, nor can it be replaced/updated/downgraded. TLS MUST be disabled at the server side for this client to function.

Steps to Reproduce:

Apply the following CRD

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  annotations:
    meta.helm.sh/release-name: mysql
    meta.helm.sh/release-namespace: default
  finalizers:
  - percona.com/delete-pxc-pods-in-order
  labels:
    app.kubernetes.io/instance: mysql
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: pxc-db
    app.kubernetes.io/version: 1.17.0
    helm.sh/chart: pxc-db-1.17.0
    helm.toolkit.fluxcd.io/name: mysql
    helm.toolkit.fluxcd.io/namespace: default
  name: mysql-pxc-db
  namespace: default
spec:
  backup:
    image: percona/percona-xtradb-cluster-operator:1.17.0-pxc8.0-backup-pxb8.0.35
    pitr:
      enabled: true
      resources:
        limits: {}
        requests: {}
      storageName: binlogs
      timeBetweenUploads: 300
      timeoutSeconds: 60
    schedule:
    - keep: 360
      name: every-six-hours
      schedule: 0 */6 * * *
      storageName: sql
    storages:
      binlogs:
        s3:
          bucket: redacted/sql/voicecore-binlogs/
          credentialsSecret: redacted
          endpointUrl: https://s3.us-west-2.amazonaws.com
          region: us-west-2
        type: s3
      sql:
        s3:
          bucket: redacted/sql/voicecore/
          credentialsSecret: redacted
          endpointUrl: https://s3.us-west-2.amazonaws.com
          region: us-west-2
        type: s3
  crVersion: 1.17.0
  enableCRValidationWebhook: false
  haproxy:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations:
      prometheus.io/port: "8404"
      prometheus.io/scrape: "true"
    enabled: true
    exposePrimary:
      annotations:
        external-dns.alpha.kubernetes.io/hostname: db-vpbx.redacted.local,db-freeswitch.redacted.local,db-cdrs.redacted.local
        external-dns.alpha.kubernetes.io/ttl: "60"
        kubernetes.io/ingress.class: internal-nginx
        service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: creation:Tool=flux,project:Name=redacted,Purpose=SQL
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-name: nv-internal-haproxy-sql
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-type: external
      loadBalancerSourceRanges:
      - 10.0.0.0/8
      type: LoadBalancer
    gracePeriod: 30
    image: percona/haproxy:2.8.14
    labels: {}
    livenessDelaySec: 300
    livenessProbes:
      failureThreshold: 4
      initialDelaySeconds: 60
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5
    nodeSelector: {}
    podDisruptionBudget:
      maxUnavailable: 1
    readinessDelaySec: 15
    readinessProbes:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits: {}
      requests:
        cpu: 600m
        memory: 1G
    sidecarPVCs: []
    sidecarResources:
      limits: {}
      requests: {}
    sidecarVolumes: []
    sidecars: []
    size: 3
    tolerations: []
    volumeSpec:
      emptyDir: {}
  logCollectorSecretName: mysql-pxc-db-log-collector
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.17.0-logcollector-fluentbit4.0.0
    resources:
      limits: {}
      requests:
        cpu: 200m
        memory: 100M
  pmm:
    enabled: false
  proxysql:
    enabled: false
  pxc:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations:
      prometheus.io/port: "9104"
      prometheus.io/scrape: "true"
    autoRecovery: true
    configuration: |2

      [mysqld]
      local_infile = 1

      [client]
      local_infile = 1
    gracePeriod: 600
    image: percona/percona-xtradb-cluster:8.0.41-32.1
    labels: {}
    livenessDelaySec: 300
    livenessProbes:
      failureThreshold: 3
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    nodeSelector:
      topology.kubernetes.io/zone: us-west-2a
    podDisruptionBudget:
      maxUnavailable: 1
    readinessDelaySec: 15
    readinessProbes:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 15
    resources:
      limits:
        cpu: 2000m
        memory: 8G
      requests:
        cpu: 600m
        memory: 4G
    sidecarPVCs: []
    sidecarResources:
      limits: {}
      requests: {}
    sidecarVolumes: []
    sidecars:
    - args:
      - --web.listen-address=0.0.0.0:9104
      - --mysqld.username=monitor
      - --mysqld.address=127.0.0.1:3306
      - --tls.insecure-skip-verify
      env:
      - name: MYSQLD_EXPORTER_PASSWORD
        valueFrom:
          secretKeyRef:
            key: monitor
            name: redacted
      image: prom/mysqld-exporter
      imagePullPolicy: Always
      name: mysqld-exporter
    size: 3
    tolerations: []
    volumeSpec:
      persistentVolumeClaim:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
        storageClassName: csi-gp3-retained
  secretsName: redacted
  tls:
    enabled: false
    issuerConf:
      group: cert-manager.io
      kind: ClusterIssuer
      name: redacted
  unsafeFlags:
    tls: true
  updateStrategy: SmartUpdate
  upgradeOptions:
    apply: disabled
    schedule: 0 4 * * *
    versionServiceEndpoint: https://check.percona.com
  vaultSecretName: mysql-pxc-db-vault
status:
  backup: {}
  haproxy:
    labelSelectorPath: app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=mysql-pxc-db,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster
    ready: 3
    size: 3
    status: ready
  host: nv-internal-haproxy-sql-30d1b04289e93037.elb.us-west-2.amazonaws.com
  logcollector: {}
  observedGeneration: 31
  pmm: {}
  proxysql: {}
  pxc:
    image: percona/percona-xtradb-cluster:8.0.41-32.1
    labelSelectorPath: app.kubernetes.io/component=pxc,app.kubernetes.io/instance=mysql-pxc-db,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster
    ready: 3
    size: 3
    status: ready
    version: 8.0.41-32.1
  ready: 6
  size: 6
  state: ready

Connect to the client

Run the SQL to check if the clients are connecting using TLS or not.

select t.THREAD_ID,
  t.PROCESSLIST_USER,
  t.PROCESSLIST_HOST,
  t.CONNECTION_TYPE,
  sbt.VARIABLE_VALUE AS cipher
FROM performance_schema.threads t
  LEFT JOIN performance_schema.status_by_thread sbt ON (
    t.THREAD_ID = sbt.THREAD_ID
    AND sbt.VARIABLE_NAME = 'Ssl_cipher'
  )
WHERE t.PROCESSLIST_USER IS NOT NULL;

Version:

percona/percona-xtradb-cluster-operator:1.17.0-pxc8.0-backup-pxb8.0.35
percona/percona-xtradb-cluster:8.0.41-32.1

Logs:

[If applicable, include any relevant log files or error messages]

Excerpt from the SQL query results to show the state, the vpbx user is the problem.

mysql> select t.THREAD_ID,     t.PROCESSLIST_USER,     t.PROCESSLIST_HOST,     t.CONNECTION_TYPE,     sbt.VARIABLE_VALUE AS cipher FROM performance_schema.threads t LEFT JOIN performance_schema.status_by_thread sbt     ON (t.THREAD_ID = sbt.THREAD_ID AND sbt.VARIABLE_NAME = 'Ssl_cipher') WHERE t.PROCESSLIST_USER IS NOT NULL;
+-----------+------------------+----------------------------------------------------------------------------+-----------------+-----------------------------+
| THREAD_ID | PROCESSLIST_USER | PROCESSLIST_HOST                                                           | CONNECTION_TYPE | cipher                      |
+-----------+------------------+----------------------------------------------------------------------------+-----------------+-----------------------------+
|         5 | system user      | NULL                                                                       | NULL            | NULL                        |
|         6 | system user      | NULL                                                                       | NULL            | NULL                        |
|      2878 | vpbx             | 10-65-0-42.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local         | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      2879 | freeswitch       | 10-65-0-47.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local         | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      1009 | vpbx             | 10-65-0-42.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local         | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      2238 | vpbx             | 10-65-0-195.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local        | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      2404 | cdr_reader       | 10-65-0-42.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local         | TCP/IP          |                             |
|        59 | event_scheduler  | localhost                                                                  | NULL            | NULL                        |
|        61 | system user      | NULL                                                                       | NULL            | NULL                        |
|      2078 | vpbx             | 10-65-0-42.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local         | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      3205 | vpbx             | mysql-pxc-db-haproxy-2.mysql-pxc-db-haproxy.default.svc.cluster.local      | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      1564 | vpbx             | 10-65-0-47.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local         | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|      3688 | operator         | 10-65-0-219.percona-xtradb-cluster-operator.pxc-operator.svc.cluster.local | SSL/TLS         | TLS_AES_128_GCM_SHA256      |

Expected Result:

The vpbx user can connect to the database without TLS

Actual Result:

The vpbx user connects to the database with TLS even with TLS disabled.

Additional Information:

I’ve also tried to set tls_version= in the [mysqld] and [client]configuration sections to disable TLS but that causes things to break since the various operator/sidecars/sync systems all seem to require TLS to make the cluster functional.

I’ll have to look at the ProxySQL setup to see if that can solve the problem, but I don’t have any experience with it so that’s going to be fun to work with.

Vinodh_Krishnaswamy · August 7, 2025, 8:50am

Hi @Gerwin_van_de_Steeg

Hope you are doing well!

Yes, you can do it. For that, you have to make sure spec.tls.enabled:false and it uses the following config:

[mysqld]
pxc-encrypt-cluster-traffic=OFF

When you have tls.enabled:true, the following parameters are used:

[mysqld]
pxc-encrypt-cluster-traffic=ON
ssl-ca=/etc/mysql/ssl-internal/ca.crt
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt

I can see you have tls disabled in your deployment. Then you can connect to the DB with --ssl-mode=DISABLED option in your MySQL client, which will use TCP/IP connection type.

From my test (see thread id 436):

$ kubectl exec -it pod/minimal-cluster-pxc-0 -n pxc -- mysql -uroot --password="xxxx" -h127.0.0.1 --ssl-mode=DISABLED
Defaulted container "pxc" out of: pxc, pxc-init (init)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 394
Server version: 8.0.39-30.1 Percona XtraDB Cluster (GPL), Release rel30, Revision 46271a0, WSREP version 26.1.4.3

Copyright (c) 2009-2024 Percona LLC and/or its affiliates
Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select t.THREAD_ID,
    ->   t.PROCESSLIST_USER,
    ->   t.PROCESSLIST_HOST,
    ->   t.CONNECTION_TYPE,
    ->   sbt.VARIABLE_VALUE AS cipher
    -> FROM performance_schema.threads t
    ->   LEFT JOIN performance_schema.status_by_thread sbt ON (
    ->     t.THREAD_ID = sbt.THREAD_ID
    ->     AND sbt.VARIABLE_NAME = 'Ssl_cipher'
    ->   )
    -> WHERE t.PROCESSLIST_USER IS NOT NULL;
+-----------+------------------+------------------+-----------------+--------+
| THREAD_ID | PROCESSLIST_USER | PROCESSLIST_HOST | CONNECTION_TYPE | cipher |
+-----------+------------------+------------------+-----------------+--------+
|         5 | root             | NULL             | NULL            | NULL   |
|         6 | root             | NULL             | NULL            | NULL   |
|        49 | event_scheduler  | localhost        | NULL            | NULL   |
|        51 | root             | NULL             | NULL            | NULL   |
|       436 | root             | localhost        | TCP/IP          |        |
+-----------+------------------+------------------+-----------------+--------+
5 rows in set (0.00 sec)

When you have tls.enabled:true, you cannot use --ssl-mode=DISABLED, else you will get below message:

 $ kubectl exec -it pod/minimal-cluster-pxc-0 -n pxc -- mysql -uroot --password="xxxx" -h127.0.0.1 --ssl-mode=DISABLED
Defaulted container "pxc" out of: pxc, pxc-init (init)
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2061 (HY000): Authentication plugin 'caching_sha2_password' reported error: Authentication requires secure connection.
command terminated with exit code 1

hope this helps you!

Regards,

Vinodh Guruji

Gerwin_van_de_Steeg · August 7, 2025, 11:28am

That’s the problem, the client software I have to work with in question cannot be passed any connection parameters or options to enable or disable TLS, the only configurable parameters are the username, password, host, and port.

Even with the change as I’ve described above to disable it according to the details in the chart, the client is still connecting using TLS, which I’ve verified with wireshark and a packet capture. Note that this is connecting to the HAProxy instance and not directly to the database server.

Looking at the process list from that query excerpt above we can see that the TLS connection originates via the HAProxy components. Here’s another excerpt after the software in question was restarted post disabling the TLS support at the server.

|    155371 | vpbx             | 10-65-0-47.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local    | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|       937 | freeswitch       | mysql-pxc-db-haproxy-2.mysql-pxc-db-haproxy.default.svc.cluster.local | TCP/IP          |                             |
|    154004 | vpbx             | mysql-pxc-db-haproxy-0.mysql-pxc-db-haproxy.default.svc.cluster.local | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|    168128 | vpbx             | 10-65-0-42.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local    | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |
|       739 | freeswitch       | mysql-pxc-db-haproxy-2.mysql-pxc-db-haproxy.default.svc.cluster.local | TCP/IP          |                             |
|    168129 | vpbx             | mysql-pxc-db-haproxy-0.mysql-pxc-db-haproxy.default.svc.cluster.local | SSL/TLS         | ECDHE-RSA-AES128-GCM-SHA256 |

So we’re not yet at a working solution here.

Cheers,

Abhinav_Gupta · August 7, 2025, 2:59pm

Can you please share the output below? If you have successfully disabled the TLS (for internal and external communication), then you should not see any secrets available, like cluster1-ssl (kubernetes.io/tls) and cluster1-ssl-internal(kubernetes.io/tls).

kubectl get secrets -n {namespace}

Gerwin_van_de_Steeg · August 7, 2025, 11:46pm

I’ve excluded the non database TLS certificates but here

$ kubectl get secrets | grep tls | grep -v "2y"
sipis-db-cnpg-replication                               kubernetes.io/tls                2      143d
sipis-db-cnpg-server                                    kubernetes.io/tls                2      143d
sipis-db-restore-test-replication                       kubernetes.io/tls                2      17d
sipis-db-restore-test-server                            kubernetes.io/tls                2      17d
voipmonitor-db-pxc-db-ssl                               kubernetes.io/tls                3      301d
voipmonitor-db-pxc-db-ssl-internal                      kubernetes.io/tls                3      301d

So we can see the TLS certs for the postgres servers, and the TLS certs for the 2nd MySQL database (that one can have TLS no problem), and no certificates for the first MySQL database.’

If we use kubectl to get into one of the three database pods that don’t have TLS enabled we see

$ kubectl exec -it mysql-pxc-db-pxc-0 -c pxc -- bash
bash-5.1$ /etc/mysql/cat node.cnf
[mysqld]
pxc-encrypt-cluster-traffic=OFF
wsrep_provider_options="pc.weight=10"

bash-5.1$ cat /etc/mysql/node.cnf | grep tls
bash-5.1$ pwd
/etc/mysql
bash-5.1$ grep  -Rrn ssl-key *
bash-5.1

And on the database with TLS enabled we see

bash-5.1$ /etc/mysql/cat node.cnf
[mysqld]
pxc-encrypt-cluster-traffic=ON
ssl-ca=/etc/mysql/ssl-internal/ca.crt
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt

bash-5.1$ cat node.cnf | grep tls
ssl-key=/etc/mysql/ssl-internal/tls.key
ssl-cert=/etc/mysql/ssl-internal/tls.crt

bash-5.1$ pwd
/etc/mysql
bash-5.1$ grep -Rrn ssl-key *
node.cnf:4:ssl-key=/etc/mysql/ssl-internal/tls.key

Which tells us that there is no TLS in that config file on the node that needs to have TLS disabled.

So when we then connect to the noTLS database from inside one of the pods using the network address

bash-5.1$ mysql -u root -h 127.0.0.1 -P 3306 -p vpbx
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 593386
Server version: 8.0.41-32.1 Percona XtraDB Cluster (GPL), Release rel32, Revision 9cd31bf, WSREP version 26.1.4.3

And our query returns

mysql> select t.THREAD_ID,     t.PROCESSLIST_USER,     t.PROCESSLIST_HOST,     t.CONNECTION_TYPE,     sbt.VARIABLE_VALUE AS cipher FROM performance_schema.threads t LEFT JOIN performance_schema.status_by_thread sbt     ON (t.THREAD_ID = sbt.THREAD_ID AND sbt.VARIABLE_NAME = 'Ssl_cipher') WHERE t.PROCESSLIST_USER IS NOT NULL AND t.PROCESSLIST_USER='root';
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> select t.THREAD_ID,     t.PROCESSLIST_USER,     t.PROCESSLIST_HOST,     t.CONNECTION_TYPE,     sbt.VARIABLE_VALUE AS cipher FROM performance_schema.threads t LEFT JOIN performance_schema.status_by_thread sbt     ON (t.THREAD_ID = sbt.THREAD_ID AND sbt.VARIABLE_NAME = 'Ssl_cipher') WHERE t.PROCESSLIST_USER IS NOT NULL AND t.PROCESSLIST_USER='root';
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    593623
Current database: *** NONE ***

+-----------+------------------+-----------------------------------------------------------------------+-----------------+------------------------+
| THREAD_ID | PROCESSLIST_USER | PROCESSLIST_HOST                                                      | CONNECTION_TYPE | cipher                 |
+-----------+------------------+-----------------------------------------------------------------------+-----------------+------------------------+
|    582584 | root             | mysql-pxc-db-haproxy-0.mysql-pxc-db-haproxy.default.svc.cluster.local | TCP/IP          |                        |
|    589592 | root             | mysql-pxc-db-haproxy-0.mysql-pxc-db-haproxy.default.svc.cluster.local | TCP/IP          |                        |
|    593443 | root             | localhost                                                             | SSL/TLS         | TLS_AES_256_GCM_SHA384 |
|    593680 | root             | 10-65-0-42.mysql-pxc-db-haproxy-replicas.default.svc.cluster.local    | TCP/IP          |                        |
+-----------+------------------+-----------------------------------------------------------------------+-----------------+------------------------+
4 rows in set (0.01 sec)

So, in the database with TLS disabled, the localhost connection from root somehow still uses TLS.

Ideally I’d want to add to the configuration the below to completely disable it, but that seems to break the operator, and the use of caching_sha2_passwordfor authentication.

[mysqld]
tls-version=

Logs

ERROR 2061 (HY000): Authentication plugin 'caching_sha2_password' reported error: Authentication requires secure connection.

WTF..

Gerwin_van_de_Steeg · August 14, 2025, 3:13am

I’ve ended up sticking a ProxySQL deployment in between that explicitly doesn’t allow TLS connections to it to resolve this issue.

Topic		Replies	Views
How to enable TLS on MongoDB operator without requiring client certificates Percona Operator for MongoDB	2	1594	November 1, 2022
Disable TLS Percona PostgreSQL Operator Percona Operator for PostgreSQL percona , postgres	1	558	March 20, 2024
Configure SSL with Percona MySQL Operator Percona Operator for MySQL mysql , percona	1	756	March 4, 2021
Issue connecting to TLS cluster using username and password Percona Operator for MongoDB percona , mongodb , closed-no-reply	0	887	April 7, 2022
Basic cluster with TLS not working, operator k8s/helm setup Percona Server for MongoDB mongodb , psmdb-operator	18	3241	December 18, 2022