# Wsrep sst replication setup fails when using intermediate certificates in a chain

**URL:** <https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595>\
**Category:** Percona Operator for MySQL\
**Tags:** mysql, percona\
**Created:** [June 6, 2023, 1:00pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595 "2023-06-06T13:00:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tilo\_Werner](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tilo_werner/32/10956_2.png) [@Tilo\_Werner](https://forums.percona.com/u/Tilo_Werner)\
**Post date:** [June 6, 2023, 1:00pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/1 "2023-06-06T13:00:36Z")

</div>

We let cert-manager create custom certificates from our own Intermediate CA signed by our Root-CA by an cm issuer which connects to hashi-corp vault where the CAs reside.

The problem comes up when using the intermediate CA, wsrep (sst) complains about the unknown CA-Certificate, which breaks the cluster deployment. Seams that it does not read the whole chain but only the first certificate (did no deeper digging)

On the other hand If the certs get signed by the Root-CA everything runs as expected. So we are able to add CNs on the mysqld service at our will all signed by our Roo-CA. But it would be much better to only use the intermediate CA for that.

I’ve found an old bug report may related to this which got fixed in MySQL version 8.0.30.

[https://bugs.mysql.com/bug.php?id=54158](https://bugs.mysql.com/bug.php?id=54158)

We’re testing with the latest Percona Release 8.0.32-24.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [June 6, 2023, 1:27pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/2 "2023-06-06T13:27:01Z")

</div>

Great! Please let us know your feedback on this issue.

---

<div class="post-metadata">

**Author:** ![Tilo\_Werner](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tilo_werner/32/10956_2.png) [@Tilo\_Werner](https://forums.percona.com/u/Tilo_Werner)\
**Post date:** [June 6, 2023, 1:31pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/3 "2023-06-06T13:31:06Z")

</div>

Uhm, maybe my problem description was misleading, sorry for that.

I wanted to know if you plan to port the fix or if you already did and what we are doing possibly wrong… 🙂

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [June 6, 2023, 1:35pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/4 "2023-06-06T13:35:19Z")

</div>

You said:

> [@Tilo\_Werner](#):
>
> We’re testing with the latest Percona Release 8.0.32-24.

Let us know your feedback with your testing, if 8.0.30+ solves the issue as noted in the bug report.

> [@Tilo\_Werner](#):
>
> I wanted to know if you plan to port the fix

All fixes applied to community/upstream MySQL are automatically part of Percona Server. If something is fixed in MySQL 8.0.30, then it is automatically fixed/included in Percona Server 8.0.30.

---

<div class="post-metadata">

**Author:** ![Tilo\_Werner](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tilo_werner/32/10956_2.png) [@Tilo\_Werner](https://forums.percona.com/u/Tilo_Werner)\
**Post date:** [June 6, 2023, 3:57pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/5 "2023-06-06T15:57:40Z")

</div>

It’s not working with the latest version. I’m using the helm chart version 1.12.0 for operator and db.

On the second pxc-db-node the following error occurs:

```auto
{"log":"2023-06-06T15:45:47.193696Z 0 [ERROR] [MY-000000] [WSREP-SST] ********FATAL ERROR****************************************** \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-06-06T15:45:47.193754Z 0 [ERROR] [MY-000000] [WSREP-SST] * The certifcate and CA (certificate authority) do not match. \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-06-06T15:45:47.193765Z 0 [ERROR] [MY-000000] [WSREP-SST] * It does not appear that the certificate was issued by the CA. \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-06-06T15:45:47.193781Z 0 [ERROR] [MY-000000] [WSREP-SST] * Please check your certificate and CA files. \n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-06-06T15:45:47.193788Z 0 [ERROR] [MY-000000] [WSREP-SST] * Line 424\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2023-06-06T15:45:47.193793Z 0 [ERROR] [MY-000000] [WSREP-SST] *************************************************************** \n","file":"/var/lib/mysql/mysqld-error.log"}

```

The pxc-db charts TLS config looks like:

```auto
tls:
  SANs:
    - pxc.$custom_dns_name
  issuerConf:
    name: $custom_issuer
    kind: ClusterIssuer

```

The $custom\_issuer is bound to the intermediate CA.

This results in a _production-ssl secret_ in the pxc namespace where _ca.crt_ references our Root-CA and tls.crt the chain consisting of the actual cert with CN _production-proxysql_¹ and **below** our intermediate ca cert.

¹which is also a problem, because we have to allow that names explicitly, but it’s another topic.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [June 6, 2023, 4:08pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/6 "2023-06-06T16:08:20Z")

</div>

And you verified the helm chart is downloading/running the Percona Server 8.0.32 image? I did not see this in the log you provided.

---

<div class="post-metadata">

**Author:** ![Tilo\_Werner](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/tilo_werner/32/10956_2.png) [@Tilo\_Werner](https://forums.percona.com/u/Tilo_Werner)\
**Post date:** [June 6, 2023, 4:16pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/7 "2023-06-06T16:16:31Z")

</div>

Yes, the snippet of the ArgoCD computed manifest.

```auto
  pxc:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations: {}
    autoRecovery: true
    configuration: |
 
      [mysqld]
      proxy_protocol_networks = *
      skip-name-resolve
      require_secure_transport=ON
    gracePeriod: 600
    image: 'percona/percona-xtradb-cluster:8.0.32-24.2'

```

```auto
And the events log from first node/pod

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 95s default-scheduler Successfully assigned pxc/production-pxc-0 to ip-10-11-141-218.eu-central-1.compute.internal
  Normal SuccessfulAttachVolume 93s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-628c5050-7e35-4992-83cc-e049a0b93dfc"
  Normal Pulling 89s kubelet Pulling image "percona/percona-xtradb-cluster-operator:1.12.0"
  Normal Pulled 88s kubelet Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0" in 741.22306ms
  Normal Created 88s kubelet Created container pxc-init
  Normal Started 88s kubelet Started container pxc-init
  Normal Pulling 87s kubelet Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector"
  Normal Pulled 87s kubelet Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector" in 736.996273ms
  Normal Created 87s kubelet Created container logs
  Normal Started 87s kubelet Started container logs
  Normal Pulling 87s kubelet Pulling image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector"
  Normal Pulled 86s kubelet Successfully pulled image "percona/percona-xtradb-cluster-operator:1.12.0-logcollector" in 730.269016ms
  Normal Created 86s kubelet Created container logrotate
  Normal Started 86s kubelet Started container logrotate
  Normal Pulling 86s kubelet Pulling image "percona/percona-xtradb-cluster:8.0.32-24.2"
  Normal Pulled 85s kubelet Successfully pulled image "percona/percona-xtradb-cluster:8.0.32-24.2" in 741.267694ms
  Normal Created 85s kubelet Created container pxc
  Normal Started 85s kubelet Started container pxc

```

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [June 6, 2023, 5:17pm UTC](https://forums.percona.com/t/wsrep-sst-replication-setup-fails-when-using-intermediate-certificates-in-a-chain/22595/8 "2023-06-06T17:17:21Z")

</div>

Ok. I’ll see if someone more experienced with our K8S operator has any insight on this aspect.
