Monitor PerconaServerMongoDBBackup with Prometheus on Kubernetes

MongoDB Percona Operator for MongoDB
,
1

Description:

I want to monitor my PerconaServerMongoDBBackup CRD on Kubernetes to check the status of the backup

Steps to Reproduce:

I can see my PerconaServerMongoDBBackup CRD, but I can’t receive any alerts if the backup failed, I need to manually check the “status” field of the CRD

Version:

PSMDB operator 1.16.3

Logs:

I only have psmdb-operator logs, but I wanna check the PerconaServerMongoDBBackup CRD state and not the logs where there is no “successful” mention

Expected Result:

Scrape the status of the PerconaServerMongoDBBackup CRD with Prometheus Stack and get the last status of several previous status

Get an alert rule on Grafana which check with Prometheus the PerconaServerMongoDBBackup CRD state

Or better, ask with prometheus to the PSMDB operator if the last backups are ready or failed

Actual Result:

No monitoring on backups

Additional Information:

I juste don’t want to create a new exporter as I know Prometheus include a kube state metrics, I don’t know if PerconaServerMongoDBBackup CRD status can be scrapped by Prometheus

2

Hello @Julien_Senzier ,

kube-state-metrics (KSM) supports CRDs. You can add the specific field of a CRD and it will be exposed by it. We actually added it into Percona Monitoring and Management the same way - read here: Monitor Kubernetes - Percona Operator for MongoDB

This is step-by-step process.

  1. Create the ConfigMap resource for KSM that would define the CRDs and fields you need. You can find an example for our operators here: k8s-monitoring/vm-operator-k8s-stack/ksm-configmap.yaml at main · Percona-Lab/k8s-monitoring · GitHub

It will look like this for PerconaServerMongoDBBackup:

apiVersion: v1
kind: ConfigMap
metadata:
  name: customresource-config-ksm
  #namespace: default
data:
  config: |
    kind: CustomResourceStateMetrics
    spec:
      resources:
        - groupVersionKind:
            group: psmdb.percona.com
            version: v1
            kind: PerconaServerMongoDBBackup
          labelsFromPath:
            name:
            - metadata
            - name
            namespace:
            - metadata
            - namespace
            dbcluster:
            - spec
            - clusterName            
          metricNamePrefix: kube_psmdb_backup
          metrics:
            - name: info
              help: "Information of PSMDB Backup"
              each:
                type: Info
                info:
                  labelsFromPath:
                    storage:
                    - spec
                    - storageName
            - name: status_state
              help: "PSMDB Backup state"
              each:
                type: StateSet
                stateSet:
                  nilIsZero: true
                  labelName: state
                  path:
                  - status
                  - state
                  list:
                  - waiting
                  - requested
                  - rejected
                  - running
                  - error
                  - ready

That way KSM will expose the metric kube_psmdb_backup_status_state that you can use to monitor the backups.

  1. Apply this ConfigMap
  2. Make sure that kube-state-metrics uses this ConfigMap to fetch these metrics. The configuration depends on what exactly are you using. See how we do it with Victoria Metrics operator in our doc.

But in a nutshell, KSM allows you to reference the ConfigMap through --custom-resource-state-config-file: kube-state-metrics/docs/metrics/extend/customresourcestate-metrics.md at main · kubernetes/kube-state-metrics · GitHub

One last piece - RBAC. Make sure that KSM has access to get PerconaServerMongoDBBackup resource. This is again how we do it in Victoria Metrics operator helm chart, but you get the idea.

rbac: 
    extraRules:
    - apiGroups:
      - apiextensions.k8s.io
      resources:
      - customresourcedefinitions
      verbs:
      - list
      - watch       
    - apiGroups:
      - psmdb.percona.com
      resources:
      - perconaservermongodbs
      - perconaservermongodbs/status
      - perconaservermongodbbackups
      - perconaservermongodbbackups/status
      - perconaservermongodbrestores
      - perconaservermongodbrestores/status
      verbs:
      - list
      - watch

@Julien_Senzier I’m curious to learn more about your use case. Please feel free to schedule a call with me, we can look into your setup together and chat about tech in general: Zoom Scheduler