Pods are getting evicted due to PidPressure

favas · March 12, 2021, 5:09am

I have a 4vcore 8gb nodes running a proprietary stack along with percona xtraDB cluster. however occasionally i get pods evicted message. describing the pod gives me PidPressure. there is no limit requests on the proprietary stack and its hardly doing anything now. is there a way to debug this . i can understand if it doesnt have memory or disk space . but PidPressure is triggered when the node runs out of process ids.

kubectl get events 
LAST SEEN   TYPE      REASON                   OBJECT                       MESSAGE
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node was low on resource: pids.
11m         Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logs
11m         Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container pxc
11m         Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logrotate
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
6m10s       Warning   FailedScheduling         pod/dev-pxc-db-pxc-2         0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/pid-pressure: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
10m         Normal    NotTriggerScaleUp        pod/dev-pxc-db-pxc-2         pod didn't trigger scale-up:
5m59s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m56s       Normal    SuccessfulAttachVolume   pod/dev-pxc-db-pxc-2         AttachVolume.Attach succeeded for volume "pvc-2ba1783f-eccb-4b97-9dbc-8327e15561aa"
5m42s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0"
5m42s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0" in 157.179349ms
5m42s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container pxc-init
5m41s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container pxc-init
5m41s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector"
5m41s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector" in 158.567761ms
5m40s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container logs
5m40s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container logs
5m40s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector"
5m40s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector" in 158.177134ms
5m40s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container logrotate
5m40s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container logrotate
5m40s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster:5.7.32-31.47"
5m40s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster:5.7.32-31.47" in 172.939112ms
5m40s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container pxc
5m39s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container pxc
5m38s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node was low on resource: pids.
5m36s       Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logs
5m36s       Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logrotate
5m36s       Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container pxc
5m36s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m36s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m36s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m36s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m35s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m35s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m34s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m34s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m32s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m32s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m31s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m31s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m30s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m30s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
28s         Warning   FailedScheduling         pod/dev-pxc-db-pxc-2         0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/pid-pressure: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
5m12s       Normal    NotTriggerScaleUp        pod/dev-pxc-db-pxc-2         pod didn't trigger scale-up:
18s         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
15s         Normal    SuccessfulAttachVolume   pod/dev-pxc-db-pxc-2         AttachVolume.Attach succeeded for volume "pvc-2ba1783f-eccb-4b97-9dbc-8327e15561aa"
0s          Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0"
0s          Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0" in 190.749006ms
0s          Normal    Created                  pod/dev-pxc-db-pxc-2         Created container pxc-init
11m         Normal    SuccessfulCreate         statefulset/dev-pxc-db-pxc   create Pod dev-pxc-db-pxc-2 in StatefulSet dev-pxc-db-pxc successful
11m         Normal    SuccessfulDelete         statefulset/dev-pxc-db-pxc   delete Pod dev-pxc-db-pxc-2 in StatefulSet dev-pxc-db-pxc successful
5m37s       Warning   RecreatingFailedPod      statefulset/dev-pxc-db-pxc   StatefulSet default/dev-pxc-db-pxc is recreating failed Pod dev-pxc-db-pxc-2
11m         Warning   FailedCreate             statefulset/dev-pxc-db-pxc   create Pod dev-pxc-db-pxc-2 in StatefulSet dev-pxc-db-pxc failed error: The POST operation against Pod could not be completed at this time, please try again.
28s         Normal    NodeHasSufficientPID     node/k8s-8c8qj         Node k8s-8c8qj status is now: NodeHasSufficientPID
5m38s       Warning   EvictionThresholdMet     node/k8s-8c8qj         Attempting to reclaim pids
5m29s       Normal    NodeHasInsufficientPID   node/k8s-8c8qj         Node k8s-8c8qj status is now: NodeHasInsufficientPID

Sergey_Pronin · March 12, 2021, 10:36am

Hello @favas ,

thank you for submitting this.
There are two PID limits: per node and per pod.

Could you please show with what flags kubelet is running?

If it is a limit for the node, PXC pod can be evicted even if it is not the culprit.
If it is a limit per pod - we need to dig deeper. Would be great if you share your cr.yaml in that case.
I cannot think of anything in PXC containers that can be spawning lots of processes.

favas · March 12, 2021, 12:24pm

Hi , what is the command to know what flags its running ? . the cluster is a 3 node managed kubernetes instance from DigitalOcean.
I am using helm charts with the below values.yaml

# Default values for pxc-cluster.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

finalizers:
  - delete-pxc-pods-in-order
## Set this if you want to delete proxysql persistent volumes on cluster deletion
  - delete-proxysql-pvc
## Set this if you want to delete database persistent volumes on cluster deletion
  - delete-pxc-pvc

nameOverride: ""
fullnameOverride: ""

crVersion: 1.7.0
pause: false
allowUnsafeConfigurations: false
updateStrategy: Never
upgradeOptions:
  versionServiceEndpoint: https://check.percona.com
  apply: recommended
  schedule: "0 4 * * *"

pxc:
  size: 3
  image:
    repository: percona/percona-xtradb-cluster
    tag: 5.7.32-31.47
  autoRecovery: true
  imagePullSecrets: []
  # - name: private-registry-credentials
  annotations: {}
  #  iam.amazonaws.com/role: role-arn
  labels: {}
  #  rack: rack-22
  # priorityClassName:
  readinessDelaySec: 15
  livenessDelaySec: 300
  forceUnsafeBootstrap: false
  ## Uncomment to pass in a mysql config file
  configuration: |
   [mysqld]
   #wsrep_debug=ON
   #wsrep_provider_options="gcache.size=1G; gcache.recover=yes"
   sql_mode = TRADITIONAL
  resources:
    requests:
      memory: 1G
      cpu: 600m
    limits: {}
      # memory: 1G
      # cpu: 600m
  nodeSelector: {}
  #  disktype: ssd
  affinity:
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # advanced:
    #   nodeAffinity:
    #     requiredDuringSchedulingIgnoredDuringExecution:
    #       nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: kubernetes.io/e2e-az-name
    #           operator: In
    #           values:
    #           - e2e-az1
    #           - e2e-az2
  tolerations: []
    # - key: "node.alpha.kubernetes.io/unreachable"
    #   operator: "Exists"
    #   effect: "NoExecute"
    #   tolerationSeconds: 6000
  gracePeriod: 600
  podDisruptionBudget:
    # only one of maxUnavailable or minAvaliable can be set
    maxUnavailable: 1
    # minAvailable: 0
  persistence:
    enabled: true
    # if persistence is enabled, you can specify a hostPath (not recommended)
    # hostPath: /data/mysql
    # otherwise you can specify values for a storage claim (default)
    ## percona data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    # storageClass: "-"
    accessMode: ReadWriteOnce
    size: 8Gi

  # If you set this to true the cluster will be created without TLS
  disableTLS: false

  # disable Helm creating TLS certificates if you want to let the operator
  # request certificates from cert-manager
  certManager: false

  # If this is set will not create secrets from values and will instead try to use
  # a pre-existing secret of the same name.
  # clusterSecretName:
upgradeOptions:
  apply: 5.7.32-31.47

haproxy:
  enabled: false
  size: 3
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-haproxy
  imagePullSecrets: []
  # - name: private-registry-credentials
  annotations: {}
  #  iam.amazonaws.com/role: role-arn
  labels: {}
  #  rack: rack-22
  # priorityClassName:
  readinessDelaySec: 15
  livenessDelaySec: 300
  resources:
    requests:
      memory: 1G
      cpu: 600m
    limits: {}
      # memory: 1G
      # cpu: 600m
  nodeSelector: {}
  #  disktype: ssd
  affinity:
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # advanced:
    #   nodeAffinity:
    #     requiredDuringSchedulingIgnoredDuringExecution:
    #       nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: kubernetes.io/e2e-az-name
    #           operator: In
    #           values:
    #           - e2e-az1
    #           - e2e-az2
  tolerations: []
    # - key: "node.alpha.kubernetes.io/unreachable"
    #   operator: "Exists"
    #   effect: "NoExecute"
    #   tolerationSeconds: 6000
  gracePeriod: 600
  # only one of `maxUnavailable` or `minAvailable` can be set.
  podDisruptionBudget:
    maxUnavailable: 1
    # minAvailable: 0

proxysql:
  enabled: true
  size: 3
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-proxysql
  imagePullSecrets: []
  # - name: private-registry-credentials
  annotations: {}
  #  iam.amazonaws.com/role: role-arn
  labels: {}
  #  rack: rack-22
  # priorityClassName:
  readinessDelaySec: 15
  livenessDelaySec: 300
  resources:
    requests:
      memory: 1G
      cpu: 600m
    limits: {}
      # memory: 1G
      # cpu: 600m
  nodeSelector: {}
  #  disktype: ssd
  affinity:
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # advanced:
    #   nodeAffinity:
    #     requiredDuringSchedulingIgnoredDuringExecution:
    #       nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: kubernetes.io/e2e-az-name
    #           operator: In
    #           values:
    #           - e2e-az1
    #           - e2e-az2
  tolerations: []
    # - key: "node.alpha.kubernetes.io/unreachable"
    #   operator: "Exists"
    #   effect: "NoExecute"
    #   tolerationSeconds: 6000
  gracePeriod: 600
  # only one of `maxUnavailable` or `minAvailable` can be set.
  podDisruptionBudget:
    maxUnavailable: 1
    # minAvailable: 0
  persistence:
    enabled: true
    # if persistence is enabled, you can specify a hostPath (not recommended)
    # hostPath: /data/mysql
    # otherwise you can specify values for a storage claim (default)
    ## percona data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    # storageClass: "-"
    accessMode: ReadWriteOnce
    size: 8Gi

logcollector:
  enabled: true
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-logcollector

pmm:
  enabled: false
  image:
    repository: percona/pmm-client
    tag: 2.12.0
  serverHost: monitoring-service
  serverUser: admin

backup:
  enabled: true
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-pxc5.7-backup
  imagePullSecrets: []
  # - name: private-registry-credentials
  pitr:
    enabled: false
    storageName: s3-us-west-binlogs
    timeBetweenUploads: 60
  storages:
    fs-pvc:
      type: filesystem
      volume:
        persistentVolumeClaim:
        #  storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 6Gi
    # s3-us-west:
    #   type: s3
    #   s3:
    #     bucket: S3-BACKUP-BUCKET-NAME-HERE
    #     credentialsSecret: my-cluster-name-backup-s3
    #     region: us-west-2
    #     endpointUrl: https://sfo2.digitaloceanspaces.com
    # s3-us-west-binlogs:
    #   type: s3
    #   s3:
    #     bucket: S3-BACKUP-BUCKET-NAME-HERE/DIRECTORY
    #     credentialsSecret: my-cluster-name-backup-s3
    #     region: us-west-2
    #     endpointUrl: https://sfo2.digitaloceanspaces.com

  schedule:
    - name: "daily-backup"
      schedule: "0 0 * * *"
      keep: 5
      storageName: fs-pvc
    # - name: "sat-night-backup"
    #   schedule: "0 0 * * 6"
    #   keep: 3
    #   storageName: s3-us-west

secrets:
  passwords:
    root: insecure-root-password
    xtrabackup: insecure-xtrabackup-password
    monitor: insecure-monitor-password
    clustercheck: insecure-clustercheck-password
    proxyadmin: insecure-proxyadmin-password
    pmmserver: insecure-pmmserver-password
    operator: insecure-operator-password
  ## If you are using `cert-manager` you can skip this next section.
  tls: {}
    # This should be the name of a secret that contains certificates.
    # it should have the following keys: `ca.crt`, `tls.crt`, `tls.key`
    # If not set the Helm chart will attempt to create certificates
    # for you [not recommended for prod]:
    # cluster:

    # This should be the name of a secret that contains certificates.
    # it should have the following keys: `ca.crt`, `tls.crt`, `tls.key`
    # If not set the Helm chart will attempt to create certificates
    # for you [not recommended for prod]:
    # internal:

Sergey_Pronin · March 17, 2021, 2:17pm

Hello @favas,

It is hard to understand what is going on without looking into the processes list in the Pod. But there are two guesses:

there is a chance that you are hitting this bug [K8SPXC-596] Liveness for pxc container could cause zombie processes - Percona JIRA. It will be fixed in the next release.
If you are running Kubernetes 1.20 it is also possible that it is related to ExecProbeTimeout: kubelet: ship new ExecProbeTimeout featuregate as false by jackfrancis · Pull Request #97057 · kubernetes/kubernetes · GitHub
But we need to verify this.

favas · March 17, 2021, 4:02pm

tried to get ps but its not available . Also i have upgraded from 4 core to 8 core 16gb ram nodes. and still get the eviction. Kubernetes version is 1.20.2 and the operator that i am using is 1.7.0 .

favas · March 18, 2021, 2:44am

Attaching operator logs

Sergey_Pronin · March 22, 2021, 2:21pm

Hello @favas ,

could you please exec into pxc container? On the screenshot you provided I see “Defaulting container name to logs”.

To get into pxc container just do: kubectl exec -ti mysql-pxc-db-pxc-1 -c pxc bash

And then please execute ps -eLf. It will show all the threads.

favas · March 30, 2021, 5:46am

Hi , i switched from ProxySql to HaProxy . now i dont get pod evictions and it has been stable so far .

Sergey_Pronin · March 31, 2021, 1:17pm

@favas thank you for providing this detail.
It is also possible that you are hitting the ProxySQL bug, where ProxySQL Admin was leaving lots of zombie processes: [PSQLADM-256] Wrong password with % character - Percona JIRA

It is fixed already and we will roll out new ProxySQL version with 1.8.0 as well.

Topic		Replies	Views
PXC cluster fails after single pod failure Percona Operator for MySQL	4	301	March 11, 2024
Change Containers per pod in operator in kubernetes Percona Operator for MySQL	7	975	July 14, 2021
Percona MySQL operator kubernetes on EKS Percona XtraDB Cluster 8.x community , mysql , percona , new-release	5	934	March 13, 2022
Kubernetes VPA can not find pxc pods Percona Operator for MySQL mysql , kubernetes	6	915	February 16, 2022
Memory management and RAM OOM kill for PXC pods Percona Operator for MySQL mysql , percona	16	2878	September 1, 2022

Pods are getting evicted due to PidPressure

Related Topics