Pods are getting evicted due to PidPressure

I have a 4vcore 8gb nodes running a proprietary stack along with percona xtraDB cluster. however occasionally i get pods evicted message. describing the pod gives me PidPressure. there is no limit requests on the proprietary stack and its hardly doing anything now. is there a way to debug this . i can understand if it doesnt have memory or disk space . but PidPressure is triggered when the node runs out of process ids.

kubectl get events 
LAST SEEN   TYPE      REASON                   OBJECT                       MESSAGE
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node was low on resource: pids.
11m         Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logs
11m         Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container pxc
11m         Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logrotate
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
11m         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
11m         Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
6m10s       Warning   FailedScheduling         pod/dev-pxc-db-pxc-2         0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/pid-pressure: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
10m         Normal    NotTriggerScaleUp        pod/dev-pxc-db-pxc-2         pod didn't trigger scale-up:
5m59s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m56s       Normal    SuccessfulAttachVolume   pod/dev-pxc-db-pxc-2         AttachVolume.Attach succeeded for volume "pvc-2ba1783f-eccb-4b97-9dbc-8327e15561aa"
5m42s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0"
5m42s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0" in 157.179349ms
5m42s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container pxc-init
5m41s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container pxc-init
5m41s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector"
5m41s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector" in 158.567761ms
5m40s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container logs
5m40s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container logs
5m40s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector"
5m40s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0-logcollector" in 158.177134ms
5m40s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container logrotate
5m40s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container logrotate
5m40s       Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster:5.7.32-31.47"
5m40s       Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster:5.7.32-31.47" in 172.939112ms
5m40s       Normal    Created                  pod/dev-pxc-db-pxc-2         Created container pxc
5m39s       Normal    Started                  pod/dev-pxc-db-pxc-2         Started container pxc
5m38s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node was low on resource: pids.
5m36s       Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logs
5m36s       Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container logrotate
5m36s       Normal    Killing                  pod/dev-pxc-db-pxc-2         Stopping container pxc
5m36s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m36s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m36s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m36s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m35s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m35s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m34s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m34s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m32s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m32s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m31s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m31s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
5m30s       Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
5m30s       Warning   Evicted                  pod/dev-pxc-db-pxc-2         The node had condition: [PIDPressure].
28s         Warning   FailedScheduling         pod/dev-pxc-db-pxc-2         0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/pid-pressure: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
5m12s       Normal    NotTriggerScaleUp        pod/dev-pxc-db-pxc-2         pod didn't trigger scale-up:
18s         Normal    Scheduled                pod/dev-pxc-db-pxc-2         Successfully assigned default/dev-pxc-db-pxc-2 to k8s-8c8qj
15s         Normal    SuccessfulAttachVolume   pod/dev-pxc-db-pxc-2         AttachVolume.Attach succeeded for volume "pvc-2ba1783f-eccb-4b97-9dbc-8327e15561aa"
0s          Normal    Pulling                  pod/dev-pxc-db-pxc-2         Pulling image "percona/percona-xtradb-cluster-operator:1.7.0"
0s          Normal    Pulled                   pod/dev-pxc-db-pxc-2         Successfully pulled image "percona/percona-xtradb-cluster-operator:1.7.0" in 190.749006ms
0s          Normal    Created                  pod/dev-pxc-db-pxc-2         Created container pxc-init
11m         Normal    SuccessfulCreate         statefulset/dev-pxc-db-pxc   create Pod dev-pxc-db-pxc-2 in StatefulSet dev-pxc-db-pxc successful
11m         Normal    SuccessfulDelete         statefulset/dev-pxc-db-pxc   delete Pod dev-pxc-db-pxc-2 in StatefulSet dev-pxc-db-pxc successful
5m37s       Warning   RecreatingFailedPod      statefulset/dev-pxc-db-pxc   StatefulSet default/dev-pxc-db-pxc is recreating failed Pod dev-pxc-db-pxc-2
11m         Warning   FailedCreate             statefulset/dev-pxc-db-pxc   create Pod dev-pxc-db-pxc-2 in StatefulSet dev-pxc-db-pxc failed error: The POST operation against Pod could not be completed at this time, please try again.
28s         Normal    NodeHasSufficientPID     node/k8s-8c8qj         Node k8s-8c8qj status is now: NodeHasSufficientPID
5m38s       Warning   EvictionThresholdMet     node/k8s-8c8qj         Attempting to reclaim pids
5m29s       Normal    NodeHasInsufficientPID   node/k8s-8c8qj         Node k8s-8c8qj status is now: NodeHasInsufficientPID
1 Like

Hello @favas ,

thank you for submitting this.
There are two PID limits: per node and per pod.

Could you please show with what flags kubelet is running?

If it is a limit for the node, PXC pod can be evicted even if it is not the culprit.
If it is a limit per pod - we need to dig deeper. Would be great if you share your cr.yaml in that case.
I cannot think of anything in PXC containers that can be spawning lots of processes.

1 Like

Hi , what is the command to know what flags its running ? . the cluster is a 3 node managed kubernetes instance from DigitalOcean.
I am using helm charts with the below values.yaml

# Default values for pxc-cluster.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

finalizers:
  - delete-pxc-pods-in-order
## Set this if you want to delete proxysql persistent volumes on cluster deletion
  - delete-proxysql-pvc
## Set this if you want to delete database persistent volumes on cluster deletion
  - delete-pxc-pvc

nameOverride: ""
fullnameOverride: ""

crVersion: 1.7.0
pause: false
allowUnsafeConfigurations: false
updateStrategy: Never
upgradeOptions:
  versionServiceEndpoint: https://check.percona.com
  apply: recommended
  schedule: "0 4 * * *"

pxc:
  size: 3
  image:
    repository: percona/percona-xtradb-cluster
    tag: 5.7.32-31.47
  autoRecovery: true
  imagePullSecrets: []
  # - name: private-registry-credentials
  annotations: {}
  #  iam.amazonaws.com/role: role-arn
  labels: {}
  #  rack: rack-22
  # priorityClassName:
  readinessDelaySec: 15
  livenessDelaySec: 300
  forceUnsafeBootstrap: false
  ## Uncomment to pass in a mysql config file
  configuration: |
   [mysqld]
   #wsrep_debug=ON
   #wsrep_provider_options="gcache.size=1G; gcache.recover=yes"
   sql_mode = TRADITIONAL
  resources:
    requests:
      memory: 1G
      cpu: 600m
    limits: {}
      # memory: 1G
      # cpu: 600m
  nodeSelector: {}
  #  disktype: ssd
  affinity:
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # advanced:
    #   nodeAffinity:
    #     requiredDuringSchedulingIgnoredDuringExecution:
    #       nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: kubernetes.io/e2e-az-name
    #           operator: In
    #           values:
    #           - e2e-az1
    #           - e2e-az2
  tolerations: []
    # - key: "node.alpha.kubernetes.io/unreachable"
    #   operator: "Exists"
    #   effect: "NoExecute"
    #   tolerationSeconds: 6000
  gracePeriod: 600
  podDisruptionBudget:
    # only one of maxUnavailable or minAvaliable can be set
    maxUnavailable: 1
    # minAvailable: 0
  persistence:
    enabled: true
    # if persistence is enabled, you can specify a hostPath (not recommended)
    # hostPath: /data/mysql
    # otherwise you can specify values for a storage claim (default)
    ## percona data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    # storageClass: "-"
    accessMode: ReadWriteOnce
    size: 8Gi

  # If you set this to true the cluster will be created without TLS
  disableTLS: false

  # disable Helm creating TLS certificates if you want to let the operator
  # request certificates from cert-manager
  certManager: false

  # If this is set will not create secrets from values and will instead try to use
  # a pre-existing secret of the same name.
  # clusterSecretName:
upgradeOptions:
  apply: 5.7.32-31.47

haproxy:
  enabled: false
  size: 3
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-haproxy
  imagePullSecrets: []
  # - name: private-registry-credentials
  annotations: {}
  #  iam.amazonaws.com/role: role-arn
  labels: {}
  #  rack: rack-22
  # priorityClassName:
  readinessDelaySec: 15
  livenessDelaySec: 300
  resources:
    requests:
      memory: 1G
      cpu: 600m
    limits: {}
      # memory: 1G
      # cpu: 600m
  nodeSelector: {}
  #  disktype: ssd
  affinity:
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # advanced:
    #   nodeAffinity:
    #     requiredDuringSchedulingIgnoredDuringExecution:
    #       nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: kubernetes.io/e2e-az-name
    #           operator: In
    #           values:
    #           - e2e-az1
    #           - e2e-az2
  tolerations: []
    # - key: "node.alpha.kubernetes.io/unreachable"
    #   operator: "Exists"
    #   effect: "NoExecute"
    #   tolerationSeconds: 6000
  gracePeriod: 600
  # only one of `maxUnavailable` or `minAvailable` can be set.
  podDisruptionBudget:
    maxUnavailable: 1
    # minAvailable: 0

proxysql:
  enabled: true
  size: 3
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-proxysql
  imagePullSecrets: []
  # - name: private-registry-credentials
  annotations: {}
  #  iam.amazonaws.com/role: role-arn
  labels: {}
  #  rack: rack-22
  # priorityClassName:
  readinessDelaySec: 15
  livenessDelaySec: 300
  resources:
    requests:
      memory: 1G
      cpu: 600m
    limits: {}
      # memory: 1G
      # cpu: 600m
  nodeSelector: {}
  #  disktype: ssd
  affinity:
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # advanced:
    #   nodeAffinity:
    #     requiredDuringSchedulingIgnoredDuringExecution:
    #       nodeSelectorTerms:
    #       - matchExpressions:
    #         - key: kubernetes.io/e2e-az-name
    #           operator: In
    #           values:
    #           - e2e-az1
    #           - e2e-az2
  tolerations: []
    # - key: "node.alpha.kubernetes.io/unreachable"
    #   operator: "Exists"
    #   effect: "NoExecute"
    #   tolerationSeconds: 6000
  gracePeriod: 600
  # only one of `maxUnavailable` or `minAvailable` can be set.
  podDisruptionBudget:
    maxUnavailable: 1
    # minAvailable: 0
  persistence:
    enabled: true
    # if persistence is enabled, you can specify a hostPath (not recommended)
    # hostPath: /data/mysql
    # otherwise you can specify values for a storage claim (default)
    ## percona data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    # storageClass: "-"
    accessMode: ReadWriteOnce
    size: 8Gi

logcollector:
  enabled: true
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-logcollector

pmm:
  enabled: false
  image:
    repository: percona/pmm-client
    tag: 2.12.0
  serverHost: monitoring-service
  serverUser: admin

backup:
  enabled: true
  image:
    repository: percona/percona-xtradb-cluster-operator
    tag: 1.7.0-pxc5.7-backup
  imagePullSecrets: []
  # - name: private-registry-credentials
  pitr:
    enabled: false
    storageName: s3-us-west-binlogs
    timeBetweenUploads: 60
  storages:
    fs-pvc:
      type: filesystem
      volume:
        persistentVolumeClaim:
        #  storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 6Gi
    # s3-us-west:
    #   type: s3
    #   s3:
    #     bucket: S3-BACKUP-BUCKET-NAME-HERE
    #     credentialsSecret: my-cluster-name-backup-s3
    #     region: us-west-2
    #     endpointUrl: https://sfo2.digitaloceanspaces.com
    # s3-us-west-binlogs:
    #   type: s3
    #   s3:
    #     bucket: S3-BACKUP-BUCKET-NAME-HERE/DIRECTORY
    #     credentialsSecret: my-cluster-name-backup-s3
    #     region: us-west-2
    #     endpointUrl: https://sfo2.digitaloceanspaces.com

  schedule:
    - name: "daily-backup"
      schedule: "0 0 * * *"
      keep: 5
      storageName: fs-pvc
    # - name: "sat-night-backup"
    #   schedule: "0 0 * * 6"
    #   keep: 3
    #   storageName: s3-us-west

secrets:
  passwords:
    root: insecure-root-password
    xtrabackup: insecure-xtrabackup-password
    monitor: insecure-monitor-password
    clustercheck: insecure-clustercheck-password
    proxyadmin: insecure-proxyadmin-password
    pmmserver: insecure-pmmserver-password
    operator: insecure-operator-password
  ## If you are using `cert-manager` you can skip this next section.
  tls: {}
    # This should be the name of a secret that contains certificates.
    # it should have the following keys: `ca.crt`, `tls.crt`, `tls.key`
    # If not set the Helm chart will attempt to create certificates
    # for you [not recommended for prod]:
    # cluster:

    # This should be the name of a secret that contains certificates.
    # it should have the following keys: `ca.crt`, `tls.crt`, `tls.key`
    # If not set the Helm chart will attempt to create certificates
    # for you [not recommended for prod]:
    # internal:
1 Like

Hello @favas,

It is hard to understand what is going on without looking into the processes list in the Pod. But there are two guesses:

  1. there is a chance that you are hitting this bug https://jira.percona.com/browse/K8SPXC-596. It will be fixed in the next release.
  2. If you are running Kubernetes 1.20 it is also possible that it is related to ExecProbeTimeout: kubelet: ship new ExecProbeTimeout featuregate as false by jackfrancis · Pull Request #97057 · kubernetes/kubernetes · GitHub
    But we need to verify this.
1 Like

tried to get ps but its not available . Also i have upgraded from 4 core to 8 core 16gb ram nodes. and still get the eviction. Kubernetes version is 1.20.2 and the operator that i am using is 1.7.0 .

1 Like

Attaching operator logs
https://controlc.com/67f559b4

1 Like

Hello @favas ,

could you please exec into pxc container? On the screenshot you provided I see “Defaulting container name to logs”.

To get into pxc container just do: kubectl exec -ti mysql-pxc-db-pxc-1 -c pxc bash

And then please execute ps -eLf. It will show all the threads.

1 Like

Hi , i switched from ProxySql to HaProxy . now i dont get pod evictions and it has been stable so far .

1 Like

@favas thank you for providing this detail.
It is also possible that you are hitting the ProxySQL bug, where ProxySQL Admin was leaving lots of zombie processes: https://jira.percona.com/browse/PSQLADM-256

It is fixed already and we will roll out new ProxySQL version with 1.8.0 as well.

1 Like