Hi, is there a way to change the amount of containers in each pod , i mean in the documentation shows one container per pod but when i run in kubernetes it creates 3 containers per pod using 3 times more resources then it should be , is there a way to change this behavior?
Im asking this because i have right now a 5 nodes cluster , with 32 vcpu each one and with operator as i see it would take 3 times this numbers.
It seems the documentation is outdated, what you see in your output are real numbers.
Not all containers require 1 vcpu, some containers are just lightweight helpers.
Oh i see! , thanks for the answer !
sorry this is me again but im trying to set the pxc cluster to use 1.5 cpu in a 4 cores server but the message i get is this.
message: ‘0/6 nodes are available: 6 Insufficient cpu.’
As i see is trying to allocate 4.5 cores instead of 1.5 cores, this is why i think this is using 1.5 x 3 , i dont know if im wrong.
Let’s check with team
@Sergey_Pronin do you know what are CPU requirements for all containers ?
This is my pxc Cr.yaml, and my cluster has 5 nodes with 4vcpu and 16 GB of RAM , im using Azure AKS., this is the config that works but if i change cpu requirements to 1.5 it returns the error i mentioned above.
apiVersion: pxc.percona.com/v1-8-0
kind: PerconaXtraDBCluster
metadata:
name: cluster2
finalizers:
- delete-pxc-pods-in-order
# - delete-proxysql-pvc
# - delete-pxc-pvc
# annotations:
# percona.com/issue-vault-token: "true"
spec:
crVersion: 1.8.0
secretsName: my-cluster-secrets
vaultSecretName: keyring-secret-vault
sslSecretName: my-cluster-ssl
sslInternalSecretName: my-cluster-ssl-internal
logCollectorSecretName: my-log-collector-secrets
# enableCRValidationWebhook: true
# tls:
# SANs:
# - pxc-1.example.com
# - pxc-2.example.com
# - pxc-3.example.com
# issuerConf:
# name: special-selfsigned-issuer
# kind: ClusterIssuer
# group: cert-manager.io
allowUnsafeConfigurations: false
# pause: false
updateStrategy: SmartUpdate
upgradeOptions:
versionServiceEndpoint: https://check.percona.com
apply: 5.7-recommended
schedule: "0 4 * * *"
pxc:
size: 3
image: percona/percona-xtradb-cluster:5.7.33-31.49
autoRecovery: true
# schedulerName: mycustom-scheduler
# readinessDelaySec: 15
# livenessDelaySec: 600
# forceUnsafeBootstrap: false
# configuration: |
# [mysqld]
# wsrep_debug=ON
# wsrep_provider_options="gcache.size=1G; gcache.recover=yes"
# [sst]
# xbstream-opts=--decompress
# [xtrabackup]
# compress=lz4
# for PXC 5.7
# [xtrabackup]
# compress
# imagePullSecrets:
# - name: private-registry-credentials
# priorityClassName: high-priority
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# containerSecurityContext:
# privileged: false
# podSecurityContext:
# runAsUser: 1001
# runAsGroup: 1001
# supplementalGroups: [1001]
# serviceAccountName: percona-xtradb-cluster-operator-workload
# imagePullPolicy: Always
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
resources:
requests:
memory: 4G
cpu: 1000m
# ephemeral-storage: 1G
# limits:
# memory: 1G
# cpu: "1"
# ephemeral-storage: 1G
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
podDisruptionBudget:
maxUnavailable: 1
# minAvailable: 0
volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
# type: Directory
persistentVolumeClaim:
# storageClassName: standard
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 6G
gracePeriod: 600
haproxy:
enabled: false
size: 3
image: percona/percona-xtradb-cluster-operator:1.8.0-haproxy
# imagePullPolicy: Always
# schedulerName: mycustom-scheduler
# configuration: |
# global
# maxconn 2048
# external-check
# stats socket /var/run/haproxy.sock mode 600 expose-fd listeners level user
#
# defaults
# log global
# mode tcp
# retries 10
# timeout client 28800s
# timeout connect 100500
# timeout server 28800s
#
# frontend galera-in
# bind *:3309 accept-proxy
# bind *:3306 accept-proxy
# mode tcp
# option clitcpka
# default_backend galera-nodes
#
# frontend galera-replica-in
# bind *:3307
# mode tcp
# option clitcpka
# default_backend galera-replica-nodes
# imagePullSecrets:
# - name: private-registry-credentials
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# serviceType: ClusterIP
# externalTrafficPolicy: Cluster
# replicasServiceType: ClusterIP
# replicasExternalTrafficPolicy: Cluster
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
resources:
requests:
memory: 1G
cpu: 600m
# limits:
# memory: 1G
# cpu: 700m
# priorityClassName: high-priority
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
# serviceAccountName: percona-xtradb-cluster-operator-workload
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
podDisruptionBudget:
maxUnavailable: 1
# minAvailable: 0
gracePeriod: 30
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# serviceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
proxysql:
enabled: true
size: 3
image: percona/percona-xtradb-cluster-operator:1.8.0-proxysql
# imagePullPolicy: Always
# configuration: |
# datadir="/var/lib/proxysql"
#
# admin_variables =
# {
# admin_credentials="proxyadmin:admin_password"
# mysql_ifaces="0.0.0.0:6032"
# refresh_interval=2000
#
# cluster_username="proxyadmin"
# cluster_password="admin_password"
# cluster_check_interval_ms=200
# cluster_check_status_frequency=100
# cluster_mysql_query_rules_save_to_disk=true
# cluster_mysql_servers_save_to_disk=true
# cluster_mysql_users_save_to_disk=true
# cluster_proxysql_servers_save_to_disk=true
# cluster_mysql_query_rules_diffs_before_sync=1
# cluster_mysql_servers_diffs_before_sync=1
# cluster_mysql_users_diffs_before_sync=1
# cluster_proxysql_servers_diffs_before_sync=1
# }
#
# mysql_variables=
# {
# monitor_password="monitor"
# monitor_galera_healthcheck_interval=1000
# threads=2
# max_connections=2048
# default_query_delay=0
# default_query_timeout=10000
# poll_timeout=2000
# interfaces="0.0.0.0:3306"
# default_schema="information_schema"
# stacksize=1048576
# connect_timeout_server=10000
# monitor_history=60000
# monitor_connect_interval=20000
# monitor_ping_interval=10000
# ping_timeout_server=200
# commands_stats=true
# sessions_sort=true
# have_ssl=true
# ssl_p2s_ca="/etc/proxysql/ssl-internal/ca.crt"
# ssl_p2s_cert="/etc/proxysql/ssl-internal/tls.crt"
# ssl_p2s_key="/etc/proxysql/ssl-internal/tls.key"
# ssl_p2s_cipher="ECDHE-RSA-AES128-GCM-SHA256"
# }
# schedulerName: mycustom-scheduler
# imagePullSecrets:
# - name: private-registry-credentials
# annotations:
# iam.amazonaws.com/role: role-arn
# labels:
# rack: rack-22
# serviceType: ClusterIP
# externalTrafficPolicy: Cluster
# runtimeClassName: image-rc
# sidecars:
# - image: busybox
# command: ["/bin/sh"]
# args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
# name: my-sidecar-1
resources:
requests:
memory: 1G
cpu: 600m
# limits:
# memory: 1G
# cpu: 700m
# priorityClassName: high-priority
# nodeSelector:
# disktype: ssd
# sidecarResources:
# requests:
# memory: 1G
# cpu: 500m
# limits:
# memory: 2G
# cpu: 600m
# serviceAccountName: percona-xtradb-cluster-operator-workload
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
# advanced:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/e2e-az-name
# operator: In
# values:
# - e2e-az1
# - e2e-az2
# tolerations:
# - key: "node.alpha.kubernetes.io/unreachable"
# operator: "Exists"
# effect: "NoExecute"
# tolerationSeconds: 6000
volumeSpec:
# emptyDir: {}
# hostPath:
# path: /data
# type: Directory
persistentVolumeClaim:
# storageClassName: standard
# accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 2G
podDisruptionBudget:
maxUnavailable: 1
# minAvailable: 0
gracePeriod: 30
# loadBalancerSourceRanges:
# - 10.0.0.0/8
# serviceAnnotations:
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
logcollector:
enabled: true
image: percona/percona-xtradb-cluster-operator:1.8.0-logcollector
# configuration: |
# [OUTPUT]
# Name es
# Match *
# Host 192.168.2.3
# Port 9200
# Index my_index
# Type my_type
# resources:
# requests:
# memory: 200M
# cpu: 500m
pmm:
enabled: false
image: percona/pmm-client:2.12.0
serverHost: monitoring-service
serverUser: admin
# pxcParams: "--disable-tablestats-limit=2000"
# proxysqlParams: "--custom-labels=CUSTOM-LABELS"
# resources:
# requests:
# memory: 200M
# cpu: 500m
backup:
image: percona/percona-xtradb-cluster-operator:1.8.0-pxc8.0-backup
# serviceAccountName: percona-xtradb-cluster-operator
# imagePullSecrets:
# - name: private-registry-credentials
pitr:
enabled: false
storageName: STORAGE-NAME-HERE
timeBetweenUploads: 60
storages:
s3-us-west:
type: s3
# nodeSelector:
# storage: tape
# backupWorker: 'True'
# resources:
# requests:
# memory: 1G
# cpu: 600m
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: backupWorker
# operator: In
# values:
# - 'True'
# tolerations:
# - key: "backupWorker"
# operator: "Equal"
# value: "True"
# effect: "NoSchedule"
# annotations:
# testName: scheduled-backup
# labels:
# backupWorker: 'True'
# schedulerName: 'default-scheduler'
# priorityClassName: 'high-priority'
# containerSecurityContext:
# privileged: true
# podSecurityContext:
# fsGroup: 1001
# supplementalGroups: [1001, 1002, 1003]
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
credentialsSecret: my-cluster-name-backup-s3
region: us-west-2
fs-pvc:
type: filesystem
# nodeSelector:
# storage: tape
# backupWorker: 'True'
# resources:
# requests:
# memory: 1G
# cpu: 600m
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: backupWorker
# operator: In
# values:
# - 'True'
# tolerations:
# - key: "backupWorker"
# operator: "Equal"
# value: "True"
# effect: "NoSchedule"
# annotations:
# testName: scheduled-backup
# labels:
# backupWorker: 'True'
# schedulerName: 'default-scheduler'
# priorityClassName: 'high-priority'
# containerSecurityContext:
# privileged: true
# podSecurityContext:
# fsGroup: 1001
# supplementalGroups: [1001, 1002, 1003]
volume:
persistentVolumeClaim:
# storageClassName: standard
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 6G
schedule:
- name: "sat-night-backup"
schedule: "0 0 * * 6"
keep: 3
storageName: s3-us-west
- name: "daily-backup"
schedule: "0 0 * * *"
keep: 5
storageName: fs-pvc
@vadimtk PXC and other containers do not need more than 100m CPUs to run, but it is without load.
But I don’t understand how CPU requirements for containers and "message: ‘0/6 nodes are available: 6 Insufficient cpu.’ error relate.
Insufficient cpu error indicates that there are not enough resources on the k8s node to accommodate the containers. Calculation is made based on requests.
About 3x requests
@Rub_Av in PXC pod we have 3 containers:
logcollector.enabled: false in the CR.We also have init container, but it consumes requests only when the pod starts.
We do not set 3x requests for PXC and you can verify it by describing the Pod.
This is how much PXC container consumes if I have 1500m CPUs for PXC and 500m CPU for logcollector:
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
pxc cluster1-pxc-0 2500m (63%) 0 (0%) 1400M (10%) 0 (0%) 13m
I also see you have ProxySQL enabled. In there we also have 3 containers, two are for monitoring. Only one container consumes requests - proxysql itself.
This is how much ProxySQL consumes when I set 1500m in proxysql.resources.requests.cpu:
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
pxc cluster1-proxysql-0 1500m (38%) 0 (0%) 1G (7%) 0 (0%) 15m
As you see it is exactly 1.5 CPUs.
Insufficient CPU
I have 3 x 4 CPU nodes in GKE and tried setting the following requests
And the pods are not scheduling. If I look into kubectl describe of one of the nodes I see the following:
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system event-exporter-gke-564fb97f9-4nrvf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 14h
kube-system fluentbit-gke-ddcrs 100m (2%) 0 (0%) 200Mi (1%) 500Mi (4%) 47h
kube-system gke-metrics-agent-t42gc 3m (0%) 0 (0%) 50Mi (0%) 50Mi (0%) 47h
kube-system kube-dns-c598bd956-tmzd7 260m (6%) 0 (0%) 110Mi (0%) 210Mi (1%) 47h
kube-system kube-proxy-gke-sergey-26338-default-pool-d72e912e-dmkn 100m (2%) 0 (0%) 0 (0%) 0 (0%) 47h
kube-system metrics-server-v0.3.6-7b5cdbcbb8-frgm5 48m (1%) 143m (3%) 105Mi (0%) 355Mi (2%) 47h
kube-system pdcsi-node-tff4f 0 (0%) 0 (0%) 0 (0%) 0 (0%) 47h
kube-system stackdriver-metadata-agent-cluster-level-67859c6554-z8vmn 98m (2%) 48m (1%) 202Mi (1%) 202Mi (1%) 14h
pxc cluster1-proxysql-1 1500m (38%) 0 (0%) 1G (7%) 0 (0%) 21m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 2109m (53%) 191m (4%)
memory 1699400192 (13%) 1317Mi (10%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-gce-pd 0 0
Events: <none>
But I need 2500m CPU for PXC pod - 1500m PXC + (500m + 500m) for logcollector, that is why I get insufficient CPU for this node for PXC pod.
I believe you have some similar story.
To debug it I recommend you do kubectl describe node <node> and review the consumption of requests on the nodes one by one.
To reiterate
Thanks ! , let me try this configuration and check if i have the same story about cpus and i will let you know, i want to try this operator i think is a great solution, i have some other questions related to azure ultra disk , but first let me try this .
Thanks !
Unanswered | Unsolved | Solved
MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright © 2006 - 2024 Percona LLC. All rights reserved.