PostgresSQL fails to trust kubernetes api certificate

Clay_Risser · July 6, 2022, 9:02pm

When I deploy postgres on a kops cluster, the api calls to the kubernetes api fail because it does not trust the kubernetes api certificate. How can I make percona postgres trust my kuberntes api certificate?

Here is the error I am getting.

2022-07-06 20:31:11,485 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)': /api/v1/namespaces/pg3/pods?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dpostgres
2022-07-06 20:31:11,487 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)': /api/v1/namespaces/pg3/configmaps?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dpostgres
2022-07-06 20:31:11,494 ERROR: Request to server https://100.64.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='100.64.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/pg3/pods?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dpostgres (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))",)
2022-07-06 20:31:11,495 ERROR: Request to server https://100.64.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='100.64.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/pg3/configmaps?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dpostgres (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))",)
2022-07-06 20:31:11,804 ERROR: get_cluster
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 704, in _load_cluster
self._wait_caches(stop_time)
File "/usr/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 696, in _wait_caches
raise RetryFailedError('Exceeded retry deadline')
patroni.utils.RetryFailedError: 'Exceeded retry deadline'
2022-07-06 20:31:11,804 WARNING: Can not get cluster from dcs
2022-07-06 20:31:12,495 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2022-07-06 20:31:12,496 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2022-07-06 20:31:12,502 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)': /api/v1/namespaces/pg3/pods?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dpostgres

Clay_Risser · July 9, 2022, 12:58pm

I believe the issue is because the following certificate authority is not trusted. Why is percona not trusting the kubernetes certificate authority? Seems like a basic requirement to me. The following certificate should be trusted.

/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

As a basic example, if I call the kubernetes api with curl, it will fail with a certificate error, which is expected because the kubernetes certificate authority is not trusted by pods by default.

curl https://kubernetes.default

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

In order to call kubernetes api’s I need to explicity trust the kubernetes certificate authority. I can do this with curl in the following example.

curl https://kubernetes.default --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

The fact that percona calls to the kubernetes api is not trusting the /var/run/secrets/kubernetes.io/serviceaccount/ca.crt certificate is creating big problems for certain cluster setups, specifically in my case kops clusters.

What steps need to be taken to ensure percona api calls to kubernets trust this certificate authority?

Clay_Risser · July 13, 2022, 5:31pm

I cannot copy the certificate because readOnlyRootFilesystem is enabled. Can someone on the percona team please address this issue. This is a critical blocker for several people.

Ege_Gunes · July 15, 2022, 9:02am

Hi @Clay_Risser,

I spent some time to understand the problem you described, but still have a hard time. Usually the tools that work with Kubernetes API server add this service account CA certificate their trust pools. For example:

client-go:

func InClusterConfig() (*Config, error) {
	const (
		tokenFile  = "/var/run/secrets/kubernetes.io/serviceaccount/token"
		rootCAFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
	)
	host, port := os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT")
	if len(host) == 0 || len(port) == 0 {
		return nil, ErrNotInCluster
	}

	token, err := ioutil.ReadFile(tokenFile)
	if err != nil {
		return nil, err
	}

	tlsClientConfig := TLSClientConfig{}

	if _, err := certutil.NewPool(rootCAFile); err != nil {
		klog.Errorf("Expected to load root CA config from %s, but got err: %v", rootCAFile, err)
	} else {
		tlsClientConfig.CAFile = rootCAFile
	}

	return &Config{
		// TODO: switch to using cluster DNS.
		Host:            "https://" + net.JoinHostPort(host, port),
		TLSClientConfig: tlsClientConfig,
		BearerToken:     string(token),
		BearerTokenFile: tokenFile,
	}, nil
}

patroni:

    def load_incluster_config(self, ca_certs=SERVICE_CERT_FILENAME,
                              token_refresh_interval=datetime.timedelta(minutes=1)):
        if SERVICE_HOST_ENV_NAME not in os.environ or SERVICE_PORT_ENV_NAME not in os.environ:
            raise self.ConfigException('Service host/port is not set.')
        if not os.environ[SERVICE_HOST_ENV_NAME] or not os.environ[SERVICE_PORT_ENV_NAME]:
            raise self.ConfigException('Service host/port is set but empty.')

        if not os.path.isfile(ca_certs):
            raise self.ConfigException('Service certificate file does not exists.')
        with open(ca_certs) as f:
            if not f.read():
                raise self.ConfigException('Cert file exists but empty.')
        self.pool_config['ca_certs'] = ca_certs
        self._token_refresh_interval = token_refresh_interval
        token = self._read_token_file()
        self._make_headers(token=token)
        self._server = uri('https', (os.environ[SERVICE_HOST_ENV_NAME], os.environ[SERVICE_PORT_ENV_NAME]))

It feels like something might be wrong in your cluster. Could you share more details about your setup? Why do you think clusters installed by kops is prone to such errors?

Clay_Risser · July 16, 2022, 8:08pm

Hmm, I’m not sure. I’ll try and dig into this more. If you have any suggestions, please let me know. I believe my kops cluster is setup correctly, because the following command runs with no errors.

curl https://kubernetes.default --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Clay_Risser · July 16, 2022, 10:36pm

@Ege_Gunes I just noticed you were looking at the wrong files. Make sure you’re looking at Patroni 2.1.1 because that is the Patroni version percona Postgres operator 1.2.0 is using.

github.com

zalando/patroni/blob/v2.1.1/patroni/dcs/kubernetes.py

import datetime
import functools
import json
import logging
import os
import random
import socket
import six
import sys
import time
import urllib3
import yaml

from urllib3 import Timeout
from urllib3.exceptions import HTTPError
from six.moves.http_client import HTTPException
from threading import Condition, Lock, Thread

from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState, TimelineHistory
from ..exceptions import DCSError

This file has been truncated. show original

Clay_Risser · July 16, 2022, 10:39pm

Here is exactly where it is failing.

github.com

zalando/patroni/blob/v2.1.1/patroni/dcs/kubernetes.py#L696


      
          def member(pod):
              annotations = pod.metadata.annotations or {}
              member = Member.from_node(pod.metadata.resource_version, pod.metadata.name, None, annotations.get('status', ''))
              member.data['pod_labels'] = pod.metadata.labels
              return member
          
          
def _wait_caches(self, stop_time):
              while not (self._pods.is_ready() and self._kinds.is_ready()):
                  timeout = stop_time - time.time()
                  if timeout <= 0:
                      raise RetryFailedError('Exceeded retry deadline')
                  self._condition.wait(timeout)
          
          
def _load_cluster(self):
              stop_time = time.time() + self._retry.deadline
              self._api.refresh_api_servers_cache()
              try:
                  with self._condition:
                      self._wait_caches(stop_time)
          
          
            members = [self.member(pod) for pod in self._pods.copy().values()]

Clay_Risser · July 16, 2022, 10:51pm

Seems like the following issue is closely related.

github.com/zalando/patroni

Patroni wrong cert error

opened 10:04AM - 14 Nov 20 UTC

closed 02:15PM - 17 Aug 21 UTC

andrei-matei

Hello all, I am opening this here because I am not sure if it's a bug or if I… am doing something wrong. I am using Patroni (in Crunchy postgres-operator) and at the time it start, I get a weird error regarding SSL failing when trying to connect to Kubernetes API. Is there a place where I need to setup a CA so that communication works? Here is the error: `2020-11-14 09:59:41,806 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /api/v1/namespaces/pgo/pods?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dhippo1 **2020-11-14 09:59:41,811 ERROR: Request to server https://10.32.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.32.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/pgo/configmaps?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dhippo1 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))",) 2020-11-14 09:59:41,812 ERROR: Request to server https://10.32.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.32.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/pgo/pods?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dhippo1 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))",)** 2020-11-14 09:59:42,734 ERROR: get_cluster **Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 700, in _load_cluster self._wait_caches(stop_time) File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 692, in _wait_caches raise RetryFailedError('Exceeded retry deadline') patroni.utils.RetryFailedError: 'Exceeded retry deadline'** 2020-11-14 09:59:42,734 WARNING: Can not get cluster from dcs 2020-11-14 09:59:42,812 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",) 2020-11-14 09:59:42,813 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",) 2020-11-14 09:59:42,819 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /api/v1/namespaces/pgo/configmaps?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dhippo1 2020-11-14 09:59:42,821 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /api/v1/namespaces/pgo/pods?labelSelector=vendor%3Dcrunchydata%2Ccrunchy-pgha-scope%3Dhippo1` On Kubernetes API Server I can see this error: log.go:172] http: TLS handshake error from 10.10.81.113:37460: remote error: tls: unknown certificate authority I am running in Kubernetes 18.10, installed with Kubeadm, running on local Centos 7 20.03 machines, with 3 masters connected to HAProxy and external etcd. Does anyone have any idea why the error? I am banging my head against the table for 3 days :) Thanks in advance, Andrei

Clay_Risser · July 16, 2022, 11:25pm

The following issue is also related.

github.com/jupyterhub/kubespawner

Expose Kubernetes client configuration options

opened 03:08AM - 04 Mar 21 UTC

closed 09:12AM - 14 Mar 21 UTC

kafonek

enhancement

### Proposed change Expose a way to inject configuration options for the `kubes…pawner.KubeSpawner.api` object. Right now it appears to be instantiated without passing in args or kwargs (`self.api = shared_client('CoreV1Api')`). ### Alternative options Some configuration options can be set via environment variables, but others cannot. For instance, `kubernetes.client.Configuration.get_default_copy().ssl_ca_cert` ### Who would use this feature? Kubernetes clusters that use custom CA certs may be misconfigured such that the ca-certs provided to service accounts (`/var/run/secrets/kubernetes.io/serviceaccounts/ca.crt` inside the Hub pod) do not work with the Kubernetes API server. When Kubespawner attempts to initialize and list pods in the namespace, it raises a `CERTIFICATE_VERIFY_FAILED` error. See https://discourse.jupyter.org/t/enabling-jh-to-use-an-additional-ca-cert-when-calling-private-k8s-api/6662 and discussions in Jupyerhub Gitter between me and @consideRatio on Mar 1 / Mar 2 2021. It is worth noting that there's probably a cluster administration solution to this problem. I do not know what needs to change on the cluster configuration side to provision working `ca.crt` files to the serviceaccount / accompany secrets. Either way, I think offering config options in Kubespawner and zero-to-jupyterhub `config.yaml` is worthwhile. ### (Optional): Suggest a solution I defer to @yuvipanda @consideRatio on how to implement this. From a user perspective, I would like to be able to mount my actual CA certs into the Pod and then tell KubeSpawner to use them either via ENV variables or as a config option. Imagine I have a `ca-certs` Secret in my namespace with a `ca.crt` file that includes the k8s API ca's, and I'm deploying jupyterhub with the zero-to-jupyterhub-k8s helm chart. ``` # config.yaml hub: extraVolumes: - name: ca-crt secret: secretName: ca-certs extraVolumeMounts: - name: ca-crt mountPath: /etc/pki/certs/ca-certificate.crt subPath: ca.crt # NEW SYNTAX (?) kubeSpawner: ssl_ca_cert: /etc/pki/certs/ca-certificate.crt # ALTERNATIVELY extraEnv: KUBESPAWNER_CA_CERT: /etc/pki/certs/ca-certificate.crt ``` ``` # kubespanwer.clients.py import kubernetes.client def shared_client(ClientType, *args, **kwargs): # caching stuff in here, not included for the sake of brevity # kubernetes.config.load_incluster_config() already called prior to invoking this function if 'KUBESPAWNER_CA_CERT' in os.environ: conf = kubernetes.client.Configuration.get_default_copy() conf.ssl_ca_cert = os.environ['KUBESPAWNER_CA_CERT'] kubernetes.client.Configuration.set_default(conf) Client = getattr(kubernetes.client, ClientType) client = Client(*args, **kwargs) return client ```

Clay_Risser · July 16, 2022, 11:32pm

One more related issue.

github.com/rook/rook

rook mgr SSL: CERTIFICATE_VERIFY_FAILED

opened 06:58PM - 06 May 19 UTC

closed 06:32AM - 02 Apr 20 UTC

NickUfer

bug wontfix

**Is this a bug report or feature request?** * Bug Report **Deviation from expected behavior:** The mgr opens with the dashboard but it has just an empty page. After some time the web panel crashes. Something with the "rook" ceph module is not right. It cannot verify some certificate. SSL/TLS is turned on for the mgr and it creates a new certificate on every pod recreation. ``` debug 2019-05-06 18:47:45.695 7f19e2794700 1 mgr[restful] server not running: no certificate configured [06/May/2019:18:47:45] ENGINE Started monitor thread '_TimeoutMonitor'. debug 2019-05-06 18:47:45.759 7f19e1f93700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'rook' while running on mgr.a: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) debug 2019-05-06 18:47:45.759 7f19e1f93700 -1 rook.serve: [06/May/2019:18:47:45] ENGINE Bus STARTING [06/May/2019:18:47:45] ENGINE Serving on :::9283 [06/May/2019:18:47:45] ENGINE Bus STARTED [06/May/2019:18:47:45] ENGINE Started monitor thread '_TimeoutMonitor'. debug 2019-05-06 18:47:45.759 7f19e1f93700 -1 Traceback (most recent call last): File "/usr/share/ceph/mgr/rook/module.py", line 280, in serve self._k8s.list_namespaced_pod(cluster_name) File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 12514, in list_namespaced_pod (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 12617, in list_namespaced_pod_with_http_info collection_formats=collection_formats) File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api _request_timeout=_request_timeout) File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request headers=headers) File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET query_params=query_params) File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request headers=headers) File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 68, in request **urlopen_kw) File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 81, in request_encode_url return self.urlopen(method, url, **urlopen_kw) File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 247, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 574, in urlopen raise SSLError(e) SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) [06/May/2019:18:47:45] ENGINE Serving on :::8443 [06/May/2019:18:47:45] ENGINE Bus STARTED ``` **Expected behavior:** The mgr doesn't crash and the dashboard shows information. **How to reproduce it (minimal and precise):** Upgrade vom 0.9 to 1.0. **Environment**: * OS (e.g. from /etc/os-release): `Ubuntu 18.10 (Cosmic Cuttlefish)` * Kernel (e.g. `uname -a`): `Linux <servername> 4.18.0-17-generic #18-Ubuntu SMP Wed Mar 13 14:34:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux` * Cloud provider or hardware configuration: bare metal * Rook version (use `rook version` inside of a Rook Pod): * Kubernetes version (use `kubectl version`): `rook: v1.0.0-9.g5060a89` * Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): `Kubernetes` * Storage backend status (e.g. for Ceph use `ceph health` in the [Rook Ceph toolbox](https://rook.io/docs/rook/master/ceph-toolbox.html)): `HEALTH_ERR Module 'rook' has failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618); 1 monitors have not enabled msgr2`

Clay_Risser · July 17, 2022, 2:51am

@Ege_Gunes I figured out that the reason it’s not working is because python does not trust intermediate certificates, and the ca that kops generates is an intermediate certificate.

Anyone have any idea how to prevent kops from creating intermediate certificates?

Lenard_Pasztor · July 4, 2023, 11:06am

@Ege_Gunes I’ve just installed the percona pg operator 2.2.0 then I created a cluster with the pg-db helm chart. The instance fails to start, because fails to verify the kubernetes api certificate
The Kubernetes using an intermediate CA so I think the problem is the same as I seen in the thread.
I’m wondering about if I can provide the full CA chain it can be work? I haven’t seen option in the crd to mount a volume to the pg pods. Can you suggest how to solve this issue?

Topic		Replies	Views
Kubernetes api with intermediate CA not supported Percona Operator for PostgreSQL	2	810	July 14, 2023
FATAL: SSL required when connecting to Percona Everest postgres engine Percona Everest postgresql	18	233	July 29, 2024
Disable TLS Percona PostgreSQL Operator Percona Operator for PostgreSQL percona , postgres	1	460	March 20, 2024
Installing Percona in kubernetes using Helm Percona Distribution for PostgreSQL	2	238	April 29, 2024
PostgreSQL operator pods fail to come up Percona Operator for PostgreSQL	36	3049	April 10, 2023

PostgresSQL fails to trust kubernetes api certificate

Related topics