Expose metrics with prometheus

Hello all,

I’ve a sidecar with mysqld-exporter image like this :

    sidecars:
    - name: mysqld-exporter
      annotations:
        prometheus.io/port: "9104"
        prometheus.io/scrape: "true" 
      image: prom/mysqld-exporter
      imagePullPolicy: IfNotPresent
      env: 
        - name: ROOT_PASS
          valueFrom:
            secretKeyRef:
              name: pxc-mysqld-exporter
              key: username
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: DATA_SOURCE_NAME
          value: "user:pass@(127.0.0.1:3306)/ --web.listen-address=$(POD_IP):9104"

and I want to catch metrics with promethus on port 9104 but nothing is goes up.

Thanks.

1 Like

When you manually access the prometheus URL, do you get data? If not, that means you have an issue with the sidecar launching properly.

1 Like

Hello @matthewb

Yes, when I use kubectl proxy and acces it with prometheus-url/metrics I have data. Prometheus annotations they work with percona xtradb operator ?

1 Like

Can you curl from the main host (ie: not using kubectl)?

1 Like

yes I have pod metrics with curl host-ip/metric

svc :

kubectl  get svc | grep -E "oper|pxc|proxy" 
lab-haproxy                           ClusterIP      10.10.10.236   <none>                              3306/TCP,3309/TCP,33062/TCP,33060/TCP            10d
lab-haproxy-replicas                  ClusterIP      10.10.10.237   <none>                              3306/TCP                                         10d
lab-pxc                               ClusterIP      None            <none>                              3306/TCP,33062/TCP,33060/TCP                     10d
lab-pxc-unready                       ClusterIP      None            <none>                              3306/TCP,33062/TCP,33060/TCP                     10d
percona-xtradb-cluster-operator       ClusterIP      10.10.10.214   <none>                              443/TCP                                          10d

hosts :

 kubectl get pod -o wide | grep -E "oper|pxc|proxy" 
lab-haproxy-0                                           2/2     Running     0          18h    10.11.10.164   lab-3   <none>           <none>
lab-haproxy-1                                           2/2     Running     0          18h    10.11.10.51    lab-4   <none>           <none>
lab-haproxy-2                                           2/2     Running     0          18h     10.11.10.157   lab-6   <none>           <none>
lab-pxc-0                                               3/3     Running     0          17h     10.11.10.104   lab-8   <none>           <none>
lab-pxc-1                                               3/3     Running     0          17h     10.11.10.159   lab-6   <none>           <none>
lab-pxc-2                                               3/3     Running     0          17h     10.11.10.168   lab-5   <none>           <none>
percona-xtradb-cluster-operator-56974495db-dn5rb        1/1     Running     0          10d     10.11.10.47   lab-4   <none>           <none>

describe lab-pxc-0 :

kubectl describe pod lab-pxc-0
...
Containers:
  pxc:
    Container ID:  docker://821931a2d58c6d685362d0b4c93eaecdcd9d0a59fb9401249baad129bcb6e2e0
    Image:         percona/percona-xtradb-cluster:8.0.23-14.1
    Image ID:      docker-pullable://percona/percona-xtradb-cluster@sha256:8109f7ca4fc465ba862c08021df12e77b65d384395078e31e270d14b77810d79
    Ports:         3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP, 33062/TCP, 33060/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      /var/lib/mysql/pxc-entrypoint.sh
...
  mysqld-exporter:
    Container ID:   docker://55bfd4ca20aae316961c13b685b9ff051f2760a264be027480e4407827833fac
    Image:          prom/mysqld-exporter
    Image ID:       docker-pullable://prom/mysqld-exporter@sha256:eb6fe170738bf9181c51f5bc89f93adb26672ec49ffdcb22f55c24834003b45d
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 23 May 2022 17:09:39 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ROOT_PASS:         <set to the key 'username' in secret 'pxc-mysqld-exporter'>  Optional: false
      POD_IP:             (v1:status.podIP)
      DATA_SOURCE_NAME:  user:pass@(127.0.0.1:3306)/ --web.listen-address=$(POD_IP):9104

1 Like

it’s ok with this config :

...
  pxc:
    annotations:
      prometheus.io/port: "9104"
      prometheus.io/scrape: "true"
...
    sidecars:
    - name: mysqld-exporter
      image: prom/mysqld-exporter
      imagePullPolicy: IfNotPresent
      env:
...
        - name: DATA_SOURCE_NAME
          value: "exporter:pass@(127.0.0.1:3306)/"
      args:
        - "--web.listen-address=0.0.0.0:9104"
1 Like

Hi all I added this in my sidecars

  sidecars:
        - name: mysqld-exporter
          image: prom/mysqld-exporter
          imagePullPolicy: IfNotPresent
          env: 
            - name: DATA_SOURCE_NAME
              value: "exporter:XXXX@(127.0.0.1:3306)/" 
          args:
            - "--web.listen-address=0.0.0.0:9104"                 

I type this in the db

CREATE USER 'exporter' IDENTIFIED BY 'XXXX' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, REPLICATION SLAVE, SELECT ON *.* TO 'exporter';

But the pod is not UP

datakube-pxc-recette-pxc-0                                      2/2     Running            0               176m
datakube-pxc-recette-pxc-1                                      2/2     Running            0               178m
datakube-pxc-recette-pxc-2                                      2/3     CrashLoopBackOff   20 (49s ago)    78m

This are the logs

ku logs -c mysqld-exporter -f datakube-pxc-recette-pxc-2
ts=2024-02-23T08:47:38.723Z caller=mysqld_exporter.go:220 level=info msg="Starting mysqld_exporter" version="(version=0.15.1, branch=HEAD, revision=cc349684494b5038ec5a52233bdca9eb9291e6f2)"
ts=2024-02-23T08:47:38.723Z caller=mysqld_exporter.go:221 level=info msg="Build context" build_context="(go=go1.21.5, platform=linux/amd64, user=root@d89c15b9f5ad, date=20231212-07:55:09, tags=unknown)"
ts=2024-02-23T08:47:38.723Z caller=config.go:150 level=error msg="failed to validate config" section=client err="no user specified in section or parent"
ts=2024-02-23T08:47:38.723Z caller=mysqld_exporter.go:225 level=info msg="Error parsing host config" file=.my.cnf err="no configuration found"

Please I need help