# Change Containers per pod in operator in kubernetes

**URL:** <https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328>\
**Category:** Percona Operator for MySQL\
**Created:** [July 13, 2021, 5:26pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328 "2021-07-13T17:26:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rub\_Av](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rub_av/32/3977_2.png) [@Rub\_Av](https://forums.percona.com/u/Rub_Av)\
**Post date:** [July 13, 2021, 5:26pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/1 "2021-07-13T17:26:32Z")

</div>

Hi, is there a way to change the amount of containers in each pod , i mean in the documentation shows one container per pod but when i run in kubernetes it creates 3 containers per pod using 3 times more resources then it should be , is there a way to change this behavior?

 ![pods](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/0/0d0829c6d623486cdd16a06c15928f4f4ef66ba8.png)

Im asking this because i have right now a 5 nodes cluster , with 32 vcpu each one and with operator as i see it would take 3 times this numbers.

---

<div class="post-metadata">

**Author:** ![vadimtk](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadimtk/32/9528_2.png) [@vadimtk](https://forums.percona.com/u/vadimtk)\
**Post date:** [July 13, 2021, 7:47pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/2 "2021-07-13T19:47:54Z")

</div>

It seems the documentation is outdated, what you see in your output are real numbers.  
Not all containers require 1 vcpu, some containers are just lightweight helpers.

---

<div class="post-metadata">

**Author:** ![Rub\_Av](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rub_av/32/3977_2.png) [@Rub\_Av](https://forums.percona.com/u/Rub_Av)\
**Post date:** [July 13, 2021, 8:28pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/3 "2021-07-13T20:28:06Z")

</div>

Oh i see! , thanks for the answer !

---

<div class="post-metadata">

**Author:** ![Rub\_Av](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rub_av/32/3977_2.png) [@Rub\_Av](https://forums.percona.com/u/Rub_Av)\
**Post date:** [July 13, 2021, 8:47pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/4 "2021-07-13T20:47:29Z")

</div>

sorry this is me again but im trying to set the pxc cluster to use 1.5 cpu in a 4 cores server but the message i get is this.

message: ‘0/6 nodes are available: 6 Insufficient cpu.’

As i see is trying to allocate 4.5 cores instead of 1.5 cores, this is why i think this is using 1.5 x 3 , i dont know if im wrong.

---

<div class="post-metadata">

**Author:** ![vadimtk](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/vadimtk/32/9528_2.png) [@vadimtk](https://forums.percona.com/u/vadimtk)\
**Post date:** [July 13, 2021, 9:04pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/5 "2021-07-13T21:04:28Z")

</div>

Let’s check with team  
@Sergey_Pronin do you know what are CPU requirements for all containers ?

---

<div class="post-metadata">

**Author:** ![Rub\_Av](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rub_av/32/3977_2.png) [@Rub\_Av](https://forums.percona.com/u/Rub_Av)\
**Post date:** [July 13, 2021, 9:14pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/7 "2021-07-13T21:14:42Z")

</div>

This is my pxc Cr.yaml, and my cluster has 5 nodes with 4vcpu and 16 GB of RAM , im using Azure AKS., this is the config that works but if i change cpu requirements to 1.5 it returns the error i mentioned above.

```auto
apiVersion: pxc.percona.com/v1-8-0
kind: PerconaXtraDBCluster
metadata:
  name: cluster2
  finalizers:
    - delete-pxc-pods-in-order
# - delete-proxysql-pvc
# - delete-pxc-pvc
# annotations:
# percona.com/issue-vault-token: "true"
spec:
  crVersion: 1.8.0
  secretsName: my-cluster-secrets
  vaultSecretName: keyring-secret-vault
  sslSecretName: my-cluster-ssl
  sslInternalSecretName: my-cluster-ssl-internal
  logCollectorSecretName: my-log-collector-secrets
# enableCRValidationWebhook: true
# tls:
# SANs:
# - pxc-1.example.com
# - pxc-2.example.com
# - pxc-3.example.com
# issuerConf:
# name: special-selfsigned-issuer
# kind: ClusterIssuer
# group: cert-manager.io
  allowUnsafeConfigurations: false
# pause: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: 5.7-recommended
    schedule: "0 4 * * *"
  pxc:
    size: 3
    image: percona/percona-xtradb-cluster:5.7.33-31.49
    autoRecovery: true
# schedulerName: mycustom-scheduler
# readinessDelaySec: 15
# livenessDelaySec: 600
# forceUnsafeBootstrap: false
# configuration: |
# [mysqld]
# wsrep_debug=ON
# wsrep_provider_options="gcache.size=1G; gcache.recover=yes"
# [sst]
# xbstream-opts=--decompress
# [xtrabackup]
# compress=lz4
# for PXC 5.7
# [xtrabackup]
# compress
# imagePullSecrets:
# - name: private-registry-credentials
# priorityClassName: high-priority
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# containerSecurityContext:
# privileged: false
# podSecurityContext:
# runAsUser: 1001
# runAsGroup: 1001
# supplementalGroups: [1001]
# serviceAccountName: percona-xtradb-cluster-operator-workload
# imagePullPolicy: Always
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
    resources:
      requests:
        memory: 4G
        cpu: 1000m
# ephemeral-storage: 1G
# limits:
# memory: 1G
# cpu: "1"
# ephemeral-storage: 1G
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
# minAvailable: 0
    volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
# type: Directory
      persistentVolumeClaim:
# storageClassName: standard
# accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 6G
    gracePeriod: 600
  haproxy:
    enabled: false
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.8.0-haproxy
# imagePullPolicy: Always
# schedulerName: mycustom-scheduler
# configuration: |
# global
# maxconn 2048
# external-check
# stats socket /var/run/haproxy.sock mode 600 expose-fd listeners level user
#
# defaults
# log global
# mode tcp
# retries 10
# timeout client 28800s
# timeout connect 100500
# timeout server 28800s
#
# frontend galera-in
# bind *:3309 accept-proxy
# bind *:3306 accept-proxy
# mode tcp
# option clitcpka
# default_backend galera-nodes
#
# frontend galera-replica-in
# bind *:3307
# mode tcp
# option clitcpka
# default_backend galera-replica-nodes
# imagePullSecrets:
# - name: private-registry-credentials
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# serviceType: ClusterIP
# externalTrafficPolicy: Cluster
# replicasServiceType: ClusterIP
# replicasExternalTrafficPolicy: Cluster
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
    resources:
      requests:
        memory: 1G
        cpu: 600m
# limits:
# memory: 1G
# cpu: 700m
# priorityClassName: high-priority
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
# serviceAccountName: percona-xtradb-cluster-operator-workload
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
# minAvailable: 0
    gracePeriod: 30
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# serviceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
  proxysql:
    enabled: true
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.8.0-proxysql
# imagePullPolicy: Always
# configuration: |
# datadir="/var/lib/proxysql"
#
# admin_variables =
# {
# admin_credentials="proxyadmin:admin_password"
# mysql_ifaces="0.0.0.0:6032"
# refresh_interval=2000
#
# cluster_username="proxyadmin"
# cluster_password="admin_password"
# cluster_check_interval_ms=200
# cluster_check_status_frequency=100
# cluster_mysql_query_rules_save_to_disk=true
# cluster_mysql_servers_save_to_disk=true
# cluster_mysql_users_save_to_disk=true
# cluster_proxysql_servers_save_to_disk=true
# cluster_mysql_query_rules_diffs_before_sync=1
# cluster_mysql_servers_diffs_before_sync=1
# cluster_mysql_users_diffs_before_sync=1
# cluster_proxysql_servers_diffs_before_sync=1
# }
#
# mysql_variables=
# {
# monitor_password="monitor"
# monitor_galera_healthcheck_interval=1000
# threads=2
# max_connections=2048
# default_query_delay=0
# default_query_timeout=10000
# poll_timeout=2000
# interfaces="0.0.0.0:3306"
# default_schema="information_schema"
# stacksize=1048576
# connect_timeout_server=10000
# monitor_history=60000
# monitor_connect_interval=20000
# monitor_ping_interval=10000
# ping_timeout_server=200
# commands_stats=true
# sessions_sort=true
# have_ssl=true
# ssl_p2s_ca="/etc/proxysql/ssl-internal/ca.crt"
# ssl_p2s_cert="/etc/proxysql/ssl-internal/tls.crt"
# ssl_p2s_key="/etc/proxysql/ssl-internal/tls.key"
# ssl_p2s_cipher="ECDHE-RSA-AES128-GCM-SHA256"
# }
# schedulerName: mycustom-scheduler
# imagePullSecrets:
# - name: private-registry-credentials
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# serviceType: ClusterIP
# externalTrafficPolicy: Cluster
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
    resources:
      requests:
        memory: 1G
        cpu: 600m
# limits:
# memory: 1G
# cpu: 700m
# priorityClassName: high-priority
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
# serviceAccountName: percona-xtradb-cluster-operator-workload
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
    volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
# type: Directory
      persistentVolumeClaim:
# storageClassName: standard
# accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2G
    podDisruptionBudget:
      maxUnavailable: 1
# minAvailable: 0
    gracePeriod: 30
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# serviceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.8.0-logcollector
# configuration: |
# [OUTPUT]
# Name es
# Match *
# Host 192.168.2.3
# Port 9200
# Index my_index
# Type my_type
# resources:
# requests:
# memory: 200M
# cpu: 500m
  pmm:
    enabled: false
    image: percona/pmm-client:2.12.0
    serverHost: monitoring-service
    serverUser: admin
# pxcParams: "--disable-tablestats-limit=2000"
# proxysqlParams: "--custom-labels=CUSTOM-LABELS"
# resources:
# requests:
# memory: 200M
# cpu: 500m
  backup:
    image: percona/percona-xtradb-cluster-operator:1.8.0-pxc8.0-backup
# serviceAccountName: percona-xtradb-cluster-operator
# imagePullSecrets:
# - name: private-registry-credentials
    pitr:
      enabled: false
      storageName: STORAGE-NAME-HERE
      timeBetweenUploads: 60
    storages:
      s3-us-west:
        type: s3
# nodeSelector:
# storage: tape
# backupWorker: 'True'
# resources:
# requests:
# memory: 1G
# cpu: 600m
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: backupWorker
# operator: In
# values:
# - 'True'
# tolerations:
# - key: "backupWorker"
# operator: "Equal"
# value: "True"
# effect: "NoSchedule"
# annotations:
# testName: scheduled-backup
# labels:
# backupWorker: 'True'
# schedulerName: 'default-scheduler'
# priorityClassName: 'high-priority'
# containerSecurityContext:
# privileged: true
# podSecurityContext:
# fsGroup: 1001
# supplementalGroups: [1001, 1002, 1003]
        s3:
          bucket: S3-BACKUP-BUCKET-NAME-HERE
          credentialsSecret: my-cluster-name-backup-s3
          region: us-west-2
      fs-pvc:
        type: filesystem
# nodeSelector:
# storage: tape
# backupWorker: 'True'
# resources:
# requests:
# memory: 1G
# cpu: 600m
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: backupWorker
# operator: In
# values:
# - 'True'
# tolerations:
# - key: "backupWorker"
# operator: "Equal"
# value: "True"
# effect: "NoSchedule"
# annotations:
# testName: scheduled-backup
# labels:
# backupWorker: 'True'
# schedulerName: 'default-scheduler'
# priorityClassName: 'high-priority'
# containerSecurityContext:
# privileged: true
# podSecurityContext:
# fsGroup: 1001
# supplementalGroups: [1001, 1002, 1003]
        volume:
          persistentVolumeClaim:
# storageClassName: standard
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 6G
    schedule:
      - name: "sat-night-backup"
        schedule: "0 0 * * 6"
        keep: 3
        storageName: s3-us-west
      - name: "daily-backup"
        schedule: "0 0 * * *"
        keep: 5
        storageName: fs-pvc

```

---

<div class="post-metadata">

**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\
**Post date:** [July 14, 2021, 8:57am UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/8 "2021-07-14T08:57:13Z")

</div>

@vadimtk PXC and other containers do not need more than 100m CPUs to run, but it is without load.  
But I don’t understand how CPU requirements for containers and "message: ‘0/6 nodes are available: 6 Insufficient cpu.’ error relate.

Insufficient cpu error indicates that there are not enough resources on the k8s node to accommodate the containers. Calculation is made based on requests.

**About 3x requests**

@Rub_Av in PXC pod we have 3 containers:

1. PXC itself. Requests are set in pxc.resources.requests section. I assume there you set cpu: 1500m. Right?  
2 and 3) logs and logrotate containers. They are used to keep the logs of the container locally on the PVC in case the Pod crashes. It is useful for debugging.  
Both containers have equal requests and they are set in logcollector.resources.requests section. You can disable logcollector by setting `logcollector.enabled: false` in the CR.

We also have [init](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) container, but it consumes requests only when the pod starts.

We do not set 3x requests for PXC and you can verify it by describing the Pod.

This is how much PXC container consumes if I have 1500m CPUs for PXC and 500m CPU for logcollector:

```auto
  Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
  --------- ---- ------------ ---------- --------------- ------------- ---
 pxc cluster1-pxc-0 2500m (63%) 0 (0%) 1400M (10%) 0 (0%) 13m

```

I also see you have ProxySQL enabled. In there we also have 3 containers, two are for monitoring. Only one container consumes requests - proxysql itself.

This is how much ProxySQL consumes when I set 1500m in proxysql.resources.requests.cpu:

```auto
  Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
  --------- ---- ------------ ---------- --------------- ------------- ---
  pxc cluster1-proxysql-0 1500m (38%) 0 (0%) 1G (7%) 0 (0%) 15m

```

As you see it is exactly 1.5 CPUs.

**Insufficient CPU**

I have 3 x 4 CPU nodes in GKE and tried setting the following requests

- PXC 1500m
- logcollector 500m
- ProxySQL 1500m

And the pods are not scheduling. If I look into `kubectl describe` of one of the nodes I see the following:

```auto
  Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
  --------- ---- ------------ ---------- --------------- ------------- ---
  kube-system event-exporter-gke-564fb97f9-4nrvf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 14h
  kube-system fluentbit-gke-ddcrs 100m (2%) 0 (0%) 200Mi (1%) 500Mi (4%) 47h
  kube-system gke-metrics-agent-t42gc 3m (0%) 0 (0%) 50Mi (0%) 50Mi (0%) 47h
  kube-system kube-dns-c598bd956-tmzd7 260m (6%) 0 (0%) 110Mi (0%) 210Mi (1%) 47h
  kube-system kube-proxy-gke-sergey-26338-default-pool-d72e912e-dmkn 100m (2%) 0 (0%) 0 (0%) 0 (0%) 47h
  kube-system metrics-server-v0.3.6-7b5cdbcbb8-frgm5 48m (1%) 143m (3%) 105Mi (0%) 355Mi (2%) 47h
  kube-system pdcsi-node-tff4f 0 (0%) 0 (0%) 0 (0%) 0 (0%) 47h
  kube-system stackdriver-metadata-agent-cluster-level-67859c6554-z8vmn 98m (2%) 48m (1%) 202Mi (1%) 202Mi (1%) 14h
  pxc cluster1-proxysql-1 1500m (38%) 0 (0%) 1G (7%) 0 (0%) 21m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource Requests Limits
  -------- -------- ------
  cpu 2109m (53%) 191m (4%)
  memory 1699400192 (13%) 1317Mi (10%)
  ephemeral-storage 0 (0%) 0 (0%)
  hugepages-2Mi 0 (0%) 0 (0%)
  attachable-volumes-gce-pd 0 0
Events: <none>

```

- I have 3920m CPU allocatable
- existing pods already consume 2109m CPU (proxysql 1500m + other pods that GKE deploy automatically)
- So I have 3920 - 2109 = 1811m free

But I need 2500m CPU for PXC pod - 1500m PXC + (500m + 500m) for logcollector, that is why I get insufficient CPU for this node for PXC pod.

I believe you have some similar story.

To debug it I recommend you do `kubectl describe node <node>` and review the consumption of requests on the nodes one by one.

**To reiterate**

- we do not set 3x requests for PXC pods and it can be easily checked
- logcollector has 2 containers and they can be disabled in PXC pod
- proxysql has 3 containers, but only one consumes requests

---

<div class="post-metadata">

**Author:** ![Rub\_Av](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/rub_av/32/3977_2.png) [@Rub\_Av](https://forums.percona.com/u/Rub_Av)\
**Post date:** [July 14, 2021, 4:15pm UTC](https://forums.percona.com/t/change-containers-per-pod-in-operator-in-kubernetes/11328/9 "2021-07-14T16:15:59Z")

</div>

Thanks ! , let me try this configuration and check if i have the same story about cpus and i will let you know, i want to try this operator i think is a great solution, i have some other questions related to azure ultra disk , but first let me try this .

Thanks !