Change Containers per pod in operator in kubernetes

Hi, is there a way to change the amount of containers in each pod , i mean in the documentation shows one container per pod but when i run in kubernetes it creates 3 containers per pod using 3 times more resources then it should be , is there a way to change this behavior?

Im asking this because i have right now a 5 nodes cluster , with 32 vcpu each one and with operator as i see it would take 3 times this numbers.

1 Like

It seems the documentation is outdated, what you see in your output are real numbers.
Not all containers require 1 vcpu, some containers are just lightweight helpers.

1 Like

Oh i see! , thanks for the answer !

1 Like

sorry this is me again but im trying to set the pxc cluster to use 1.5 cpu in a 4 cores server but the message i get is this.

message: ‘0/6 nodes are available: 6 Insufficient cpu.’

As i see is trying to allocate 4.5 cores instead of 1.5 cores, this is why i think this is using 1.5 x 3 , i dont know if im wrong.

1 Like

Let’s check with team
@Sergey_Pronin do you know what are CPU requirements for all containers ?

1 Like

This is my pxc Cr.yaml, and my cluster has 5 nodes with 4vcpu and 16 GB of RAM , im using Azure AKS., this is the config that works but if i change cpu requirements to 1.5 it returns the error i mentioned above.

apiVersion: pxc.percona.com/v1-8-0
kind: PerconaXtraDBCluster
metadata:
  name: cluster2
  finalizers:
    - delete-pxc-pods-in-order
#    - delete-proxysql-pvc
#    - delete-pxc-pvc
#  annotations:
#    percona.com/issue-vault-token: "true"
spec:
  crVersion: 1.8.0
  secretsName: my-cluster-secrets
  vaultSecretName: keyring-secret-vault
  sslSecretName: my-cluster-ssl
  sslInternalSecretName: my-cluster-ssl-internal
  logCollectorSecretName: my-log-collector-secrets
#  enableCRValidationWebhook: true
#  tls:
#    SANs:
#      - pxc-1.example.com
#      - pxc-2.example.com
#      - pxc-3.example.com
#    issuerConf:
#      name: special-selfsigned-issuer
#      kind: ClusterIssuer
#      group: cert-manager.io
  allowUnsafeConfigurations: false
#  pause: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: 5.7-recommended
    schedule: "0 4 * * *"
  pxc:
    size: 3
    image: percona/percona-xtradb-cluster:5.7.33-31.49
    autoRecovery: true
#    schedulerName: mycustom-scheduler
#    readinessDelaySec: 15
#    livenessDelaySec: 600
#    forceUnsafeBootstrap: false
#    configuration: |
#      [mysqld]
#      wsrep_debug=ON
#      wsrep_provider_options="gcache.size=1G; gcache.recover=yes"
#      [sst]
#      xbstream-opts=--decompress
#      [xtrabackup]
#      compress=lz4
#      for PXC 5.7
#      [xtrabackup]
#      compress
#    imagePullSecrets:
#      - name: private-registry-credentials
#    priorityClassName: high-priority
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
#    containerSecurityContext:
#      privileged: false
#    podSecurityContext:
#      runAsUser: 1001
#      runAsGroup: 1001
#      supplementalGroups: [1001]
#    serviceAccountName: percona-xtradb-cluster-operator-workload
#    imagePullPolicy: Always
#    runtimeClassName: image-rc
#    sidecars:
#    - image: busybox
#      command: ["/bin/sh"]
#      args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
#      name: my-sidecar-1
    resources:
      requests:
        memory: 4G
        cpu: 1000m
#        ephemeral-storage: 1G
#      limits:
#        memory: 1G
#        cpu: "1"
#        ephemeral-storage: 1G
#    nodeSelector:
#      disktype: ssd
#    sidecarResources:
#      requests:
#        memory: 1G
#        cpu: 500m
#      limits:
#        memory: 2G
#        cpu: 600m
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    volumeSpec:
#      emptyDir: {}
#      hostPath:
#        path: /data
#        type: Directory
      persistentVolumeClaim:
#        storageClassName: standard
#        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 6G
    gracePeriod: 600
  haproxy:
    enabled: false
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.8.0-haproxy
#    imagePullPolicy: Always
#    schedulerName: mycustom-scheduler
#    configuration: |
#      global
#        maxconn 2048
#        external-check
#        stats socket /var/run/haproxy.sock mode 600 expose-fd listeners level user
#
#      defaults
#        log global
#        mode tcp
#        retries 10
#        timeout client 28800s
#        timeout connect 100500
#        timeout server 28800s
#
#      frontend galera-in
#        bind *:3309 accept-proxy
#        bind *:3306 accept-proxy
#        mode tcp
#        option clitcpka
#        default_backend galera-nodes
#
#      frontend galera-replica-in
#        bind *:3307
#        mode tcp
#        option clitcpka
#        default_backend galera-replica-nodes
#    imagePullSecrets:
#      - name: private-registry-credentials
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
#    serviceType: ClusterIP
#    externalTrafficPolicy: Cluster
#    replicasServiceType: ClusterIP
#    replicasExternalTrafficPolicy: Cluster
#    runtimeClassName: image-rc
#    sidecars:
#    - image: busybox
#      command: ["/bin/sh"]
#      args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
#      name: my-sidecar-1
    resources:
      requests:
        memory: 1G
        cpu: 600m
#      limits:
#        memory: 1G
#        cpu: 700m
#    priorityClassName: high-priority
#    nodeSelector:
#      disktype: ssd
#    sidecarResources:
#      requests:
#        memory: 1G
#        cpu: 500m
#      limits:
#        memory: 2G
#        cpu: 600m
#    serviceAccountName: percona-xtradb-cluster-operator-workload
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    gracePeriod: 30
#   loadBalancerSourceRanges:
#     - 10.0.0.0/8
#   serviceAnnotations:
#     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
  proxysql:
    enabled: true
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.8.0-proxysql
#    imagePullPolicy: Always
#    configuration: |
#      datadir="/var/lib/proxysql"
#
#      admin_variables =
#      {
#        admin_credentials="proxyadmin:admin_password"
#        mysql_ifaces="0.0.0.0:6032"
#        refresh_interval=2000
#
#        cluster_username="proxyadmin"
#        cluster_password="admin_password"
#        cluster_check_interval_ms=200
#        cluster_check_status_frequency=100
#        cluster_mysql_query_rules_save_to_disk=true
#        cluster_mysql_servers_save_to_disk=true
#        cluster_mysql_users_save_to_disk=true
#        cluster_proxysql_servers_save_to_disk=true
#        cluster_mysql_query_rules_diffs_before_sync=1
#        cluster_mysql_servers_diffs_before_sync=1
#        cluster_mysql_users_diffs_before_sync=1
#        cluster_proxysql_servers_diffs_before_sync=1
#      }
#
#      mysql_variables=
#      {
#        monitor_password="monitor"
#        monitor_galera_healthcheck_interval=1000
#        threads=2
#        max_connections=2048
#        default_query_delay=0
#        default_query_timeout=10000
#        poll_timeout=2000
#        interfaces="0.0.0.0:3306"
#        default_schema="information_schema"
#        stacksize=1048576
#        connect_timeout_server=10000
#        monitor_history=60000
#        monitor_connect_interval=20000
#        monitor_ping_interval=10000
#        ping_timeout_server=200
#        commands_stats=true
#        sessions_sort=true
#        have_ssl=true
#        ssl_p2s_ca="/etc/proxysql/ssl-internal/ca.crt"
#        ssl_p2s_cert="/etc/proxysql/ssl-internal/tls.crt"
#        ssl_p2s_key="/etc/proxysql/ssl-internal/tls.key"
#        ssl_p2s_cipher="ECDHE-RSA-AES128-GCM-SHA256"
#      }
#    schedulerName: mycustom-scheduler
#    imagePullSecrets:
#      - name: private-registry-credentials
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
#    serviceType: ClusterIP
#    externalTrafficPolicy: Cluster
#    runtimeClassName: image-rc
#    sidecars:
#    - image: busybox
#      command: ["/bin/sh"]
#      args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
#      name: my-sidecar-1
    resources:
      requests:
        memory: 1G
        cpu: 600m
#      limits:
#        memory: 1G
#        cpu: 700m
#    priorityClassName: high-priority
#    nodeSelector:
#      disktype: ssd
#    sidecarResources:
#      requests:
#        memory: 1G
#        cpu: 500m
#      limits:
#        memory: 2G
#        cpu: 600m
#    serviceAccountName: percona-xtradb-cluster-operator-workload
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
    volumeSpec:
#      emptyDir: {}
#      hostPath:
#        path: /data
#        type: Directory
      persistentVolumeClaim:
#        storageClassName: standard
#        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 2G
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    gracePeriod: 30
#   loadBalancerSourceRanges:
#     - 10.0.0.0/8
#   serviceAnnotations:
#     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.8.0-logcollector
#    configuration: |
#      [OUTPUT]
#           Name  es
#           Match *
#           Host  192.168.2.3
#           Port  9200
#           Index my_index
#           Type  my_type
#    resources:
#      requests:
#        memory: 200M
#        cpu: 500m
  pmm:
    enabled: false
    image: percona/pmm-client:2.12.0
    serverHost: monitoring-service
    serverUser: admin
#    pxcParams: "--disable-tablestats-limit=2000"
#    proxysqlParams: "--custom-labels=CUSTOM-LABELS"
#    resources:
#      requests:
#        memory: 200M
#        cpu: 500m
  backup:
    image: percona/percona-xtradb-cluster-operator:1.8.0-pxc8.0-backup
#    serviceAccountName: percona-xtradb-cluster-operator
#    imagePullSecrets:
#      - name: private-registry-credentials
    pitr:
      enabled: false
      storageName: STORAGE-NAME-HERE
      timeBetweenUploads: 60
    storages:
      s3-us-west:
        type: s3
#        nodeSelector:
#          storage: tape
#          backupWorker: 'True'
#        resources:
#          requests:
#            memory: 1G
#            cpu: 600m
#        affinity:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: backupWorker
#                  operator: In
#                  values:
#                  - 'True'
#        tolerations:
#          - key: "backupWorker"
#            operator: "Equal"
#            value: "True"
#            effect: "NoSchedule"
#        annotations:
#          testName: scheduled-backup
#        labels:
#          backupWorker: 'True'
#        schedulerName: 'default-scheduler'
#        priorityClassName: 'high-priority'
#        containerSecurityContext:
#          privileged: true
#        podSecurityContext:
#          fsGroup: 1001
#          supplementalGroups: [1001, 1002, 1003]
        s3:
          bucket: S3-BACKUP-BUCKET-NAME-HERE
          credentialsSecret: my-cluster-name-backup-s3
          region: us-west-2
      fs-pvc:
        type: filesystem
#        nodeSelector:
#          storage: tape
#          backupWorker: 'True'
#        resources:
#          requests:
#            memory: 1G
#            cpu: 600m
#        affinity:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: backupWorker
#                  operator: In
#                  values:
#                  - 'True'
#        tolerations:
#          - key: "backupWorker"
#            operator: "Equal"
#            value: "True"
#            effect: "NoSchedule"
#        annotations:
#          testName: scheduled-backup
#        labels:
#          backupWorker: 'True'
#        schedulerName: 'default-scheduler'
#        priorityClassName: 'high-priority'
#        containerSecurityContext:
#          privileged: true
#        podSecurityContext:
#          fsGroup: 1001
#          supplementalGroups: [1001, 1002, 1003]
        volume:
          persistentVolumeClaim:
#            storageClassName: standard
            accessModes: [ "ReadWriteOnce" ]
            resources:
              requests:
                storage: 6G
    schedule:
      - name: "sat-night-backup"
        schedule: "0 0 * * 6"
        keep: 3
        storageName: s3-us-west
      - name: "daily-backup"
        schedule: "0 0 * * *"
        keep: 5
        storageName: fs-pvc

1 Like

@vadimtk PXC and other containers do not need more than 100m CPUs to run, but it is without load.
But I don’t understand how CPU requirements for containers and "message: ‘0/6 nodes are available: 6 Insufficient cpu.’ error relate.

Insufficient cpu error indicates that there are not enough resources on the k8s node to accommodate the containers. Calculation is made based on requests.

About 3x requests

@Rub_Av in PXC pod we have 3 containers:

  1. PXC itself. Requests are set in pxc.resources.requests section. I assume there you set cpu: 1500m. Right?
    2 and 3) logs and logrotate containers. They are used to keep the logs of the container locally on the PVC in case the Pod crashes. It is useful for debugging.
    Both containers have equal requests and they are set in logcollector.resources.requests section. You can disable logcollector by setting logcollector.enabled: false in the CR.

We also have init container, but it consumes requests only when the pod starts.

We do not set 3x requests for PXC and you can verify it by describing the Pod.

This is how much PXC container consumes if I have 1500m CPUs for PXC and 500m CPU for logcollector:

  Namespace                   Name                                                      CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                      ------------  ----------  ---------------  -------------  ---
 pxc                         cluster1-pxc-0                                            2500m (63%)   0 (0%)      1400M (10%)      0 (0%)         13m

I also see you have ProxySQL enabled. In there we also have 3 containers, two are for monitoring. Only one container consumes requests - proxysql itself.

This is how much ProxySQL consumes when I set 1500m in proxysql.resources.requests.cpu:

  Namespace                   Name                                                      CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                      ------------  ----------  ---------------  -------------  ---
  pxc                         cluster1-proxysql-0                                       1500m (38%)   0 (0%)      1G (7%)          0 (0%)         15m

As you see it is exactly 1.5 CPUs.

Insufficient CPU

I have 3 x 4 CPU nodes in GKE and tried setting the following requests

  • PXC 1500m
  • logcollector 500m
  • ProxySQL 1500m

And the pods are not scheduling. If I look into kubectl describe of one of the nodes I see the following:

  Namespace                   Name                                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                         ------------  ----------  ---------------  -------------  ---
  kube-system                 event-exporter-gke-564fb97f9-4nrvf                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         14h
  kube-system                 fluentbit-gke-ddcrs                                          100m (2%)     0 (0%)      200Mi (1%)       500Mi (4%)     47h
  kube-system                 gke-metrics-agent-t42gc                                      3m (0%)       0 (0%)      50Mi (0%)        50Mi (0%)      47h
  kube-system                 kube-dns-c598bd956-tmzd7                                     260m (6%)     0 (0%)      110Mi (0%)       210Mi (1%)     47h
  kube-system                 kube-proxy-gke-sergey-26338-default-pool-d72e912e-dmkn       100m (2%)     0 (0%)      0 (0%)           0 (0%)         47h
  kube-system                 metrics-server-v0.3.6-7b5cdbcbb8-frgm5                       48m (1%)      143m (3%)   105Mi (0%)       355Mi (2%)     47h
  kube-system                 pdcsi-node-tff4f                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         47h
  kube-system                 stackdriver-metadata-agent-cluster-level-67859c6554-z8vmn    98m (2%)      48m (1%)    202Mi (1%)       202Mi (1%)     14h
  pxc                         cluster1-proxysql-1                                          1500m (38%)   0 (0%)      1G (7%)          0 (0%)         21m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                   Requests          Limits
  --------                   --------          ------
  cpu                        2109m (53%)       191m (4%)
  memory                     1699400192 (13%)  1317Mi (10%)
  ephemeral-storage          0 (0%)            0 (0%)
  hugepages-2Mi              0 (0%)            0 (0%)
  attachable-volumes-gce-pd  0                 0
Events:                      <none>
  • I have 3920m CPU allocatable
  • existing pods already consume 2109m CPU (proxysql 1500m + other pods that GKE deploy automatically)
  • So I have 3920 - 2109 = 1811m free

But I need 2500m CPU for PXC pod - 1500m PXC + (500m + 500m) for logcollector, that is why I get insufficient CPU for this node for PXC pod.

I believe you have some similar story.

To debug it I recommend you do kubectl describe node <node> and review the consumption of requests on the nodes one by one.

To reiterate

  • we do not set 3x requests for PXC pods and it can be easily checked
  • logcollector has 2 containers and they can be disabled in PXC pod
  • proxysql has 3 containers, but only one consumes requests
1 Like

Thanks ! , let me try this configuration and check if i have the same story about cpus and i will let you know, i want to try this operator i think is a great solution, i have some other questions related to azure ultra disk , but first let me try this .

Thanks !

1 Like