Invalid memory address or nil pointer dereference

A_Niu · February 14, 2022, 5:12am

Hello there,

I created an unmanaged MongoDB cluster, but the pod keep restarting. And the reason is as below, could you please shed some lights on it?

$ k get events
LAST SEEN   TYPE      REASON      OBJECT                  MESSAGE
90s         Warning   Unhealthy   pod/mongo-soby-2-rs-0   (combined from similar events): Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2022-02-14T03:07:34Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x870327]

goroutine 1 [running]:
main.main()
            /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:123 +0xca7

Thanks
Arthur

Ege_Gunes · February 14, 2022, 6:28am

Hi @A_Niu,

Could you please share your cr.yaml and the operator logs? I’ll try to reproduce the issue.

A_Niu · February 14, 2022, 1:58pm

Thanks, Egegunes
Our system doesn’t allow to upload the helm chart. I pasted it within this post.
The chart was downloaded from GitHub - percona/percona-helm-charts: Collection of Helm charts for Percona Kubernetes Operators.
Here is the values overrided.

  cluster_chart_values_override = {
    "allowUnsafeConfigurations" = true # When replicasets size is less than 3, allowUnsafeConfigurations must be true
    "image.tag"        = "3.6.23-13.0"
    "imagePullPolicy"  = "Always"
    "nameOverride"     = "soby-2"
    "replsets[0].name" = "rs"
    "replsets[0].size" = 1    
    "sharding.enabled" = false
    # "unmanaged"            = false
    "unmanaged"            = true
    "upgradeOptions.apply" = "Disabled"
  }
    unmanaged_cluster_override = {
        "replsets[0].expose.enabled" = true
        "replsets[0].expose.exposeType" = "ClusterIP"
        "updateStrategy" = "Never"
    }

operator.log (2.0 MB)

# Default values for psmdb-cluster.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Platform type: kubernetes, openshift
# platform: kubernetes

# Cluster DNS Suffix
# DNSsuffix: svc.cluster.local

finalizers:
## Set this if you want to delete database persistent volumes on cluster deletion
  # - delete-psmdb-pvc

pause: false
unmanaged: false
allowUnsafeConfigurations: true
updateStrategy: SmartUpdate
# https://www.percona.com/doc/kubernetes-operator-for-psmongodb/update.html
upgradeOptions:
  versionServiceEndpoint: https://check.percona.com
  # apply: 4.4-recommended
  apply: Disabled
  schedule: "0 2 * * *"
  setFCV: false

image:
  repository: percona/percona-server-mongodb
  # tag: 4.4.10-11
  # tag: 3.4.23
  tag: 3.6.23-13.0

imagePullPolicy: Always
# imagePullSecrets: []
# runUid: 1001
secrets: {}
  # If you set users secret here, it will not be constructed from the values at the
  # bottom of this file, but the operator will use existing one or generate random values
  # users: my-cluster-name-secrets

# Percona Monitoring and Management
pmm:
  enabled: false
  image:
    repository: percona/pmm-client
    tag: 2.24.0
  serverHost: monitoring-service

# replsets[0].nonvoting.resources.requests.memory
replsets:
  - name: rs0
    size: 1
    # externalNodes:
    # - host: 34.124.76.90
    # - host: 34.124.76.91
    #   port: 27017
    #   votes: 0
    #   priority: 0
    # - host: 34.124.76.92
    # configuration: |
    #   operationProfiling:
    #     mode: slowOp
    #   systemLog:
    #     verbosity: 1
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # tolerations: []
    # priorityClass: ""
    # annotations: {}
    # labels: {}
    # nodeSelector: {}
    livenessProbe:
      failureThreshold: 4
      initialDelaySeconds: 120
      periodSeconds: 30
      timeoutSeconds: 30
      startupDelaySeconds: 7200
    # readinessProbe:
    #   failureThreshold: 8
    #   initialDelaySeconds: 10
    #   periodSeconds: 3
    #   successThreshold: 1
    #   timeoutSeconds: 2
    # runtimeClassName: image-rc
    # sidecars:
    # - image: busybox
    #   command: ["/bin/sh"]
    #   args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
    #   name: rs-sidecar-1
    #   volumeMounts:
    #     - mountPath: /volume1
    #       name: sidecar-volume-claim
    #     - mountPath: /secret
    #       name: sidecar-secret
    #     - mountPath: /configmap
    #       name: sidecar-config
    # sidecarVolumes:
    # - name: sidecar-secret
    #   secret:
    #     secretName: mysecret
    # - name: sidecar-config
    #   configMap:
    #     name: myconfigmap
    # sidecarPVCs:
    # - apiVersion: v1
    #   kind: PersistentVolumeClaim
    #   metadata:
    #     name: sidecar-volume-claim
    #   spec:
    #     resources:
    #       requests:
    #         storage: 1Gi
    #     volumeMode: Filesystem
    #     accessModes:
    #       - ReadWriteOnce
    podDisruptionBudget:
      maxUnavailable: 0
    expose:
      enabled: false
      exposeType: LoadBalancer
    nonvoting:
      enabled: false
      size: 3
      # configuration: |
      #   operationProfiling:
      #     mode: slowOp
      #   systemLog:
      #     verbosity: 1
      antiAffinityTopologyKey: "kubernetes.io/hostname"
      # tolerations: []
      # priorityClass: ""
      # annotations: {}
      # labels: {}
      # nodeSelector: {}
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "300m"
          memory: "0.5G"
        requests:
          cpu: "300m"
          memory: "0.5G"
      volumeSpec:
        # emptyDir: {}
        # hostPath:
        #   path: /data
        pvc:
          # storageClassName: standard
          # accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 3Gi
    arbiter:
      enabled: false
      size: 1
      antiAffinityTopologyKey: "kubernetes.io/hostname"
      # tolerations: []
      # priorityClass: ""
      # annotations: {}
      # labels: {}
      # nodeSelector: {}
    # schedulerName: ""
    resources:
      limits:
        cpu: "300m"
        memory: "2.0G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    volumeSpec:
      # emptyDir: {}
      # hostPath:
      #   path: /data
      pvc:
        # storageClassName: standard
        # accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 3Gi

sharding:
  enabled: false

  configrs:
    size: 3
    # externalNodes:
    # - host: 34.124.76.90
    # - host: 34.124.76.91
    #   port: 27017
    #   votes: 0
    #   priority: 0
    # - host: 34.124.76.92
    # configuration: |
    #   operationProfiling:
    #     mode: slowOp
    #   systemLog:
    #     verbosity: 1
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # tolerations: []
    # priorityClass: ""
    # annotations: {}
    # labels: {}
    # nodeSelector: {}
    # livenessProbe: {}
    # readinessProbe: {}
    # runtimeClassName: image-rc
    # sidecars:
    # - image: busybox
    #   command: ["/bin/sh"]
    #   args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
    #   name: rs-sidecar-1
    #   volumeMounts:
    #     - mountPath: /volume1
    #       name: sidecar-volume-claim
    # sidecarPVCs: []
    # sidecarVolumes: []
    podDisruptionBudget:
      maxUnavailable: 1
    expose:
      enabled: false
      exposeType: LoadBalancer
    resources:
      limits:
        cpu: "300m"
        memory: "0.5G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    volumeSpec:
      # emptyDir: {}
      # hostPath:
      #   path: /data
      #   type: Directory
      pvc:
        # storageClassName: standard
        # accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 3Gi

  mongos:
    size: 2
    # configuration: |
    #   systemLog:
    #     verbosity: 1
    antiAffinityTopologyKey: "kubernetes.io/hostname"
    # tolerations: []
    # priorityClass: ""
    # annotations: {}
    # labels: {}
    # nodeSelector: {}
    # livenessProbe: {}
    # readinessProbe: {}
    # runtimeClassName: image-rc
    # sidecars:
    # - image: busybox
    #   command: ["/bin/sh"]
    #   args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5;done"]
    #   name: rs-sidecar-1
    #   volumeMounts:
    #     - mountPath: /volume1
    #       name: sidecar-volume-claim
    # sidecarPVCs: []
    # sidecarVolumes: []
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: "300m"
        memory: "0.5G"
      requests:
        cpu: "300m"
        memory: "0.5G"
    expose:
      exposeType: ClusterIP
      # loadBalancerSourceRanges:
      #   - 10.0.0.0/8
      # serviceAnnotations:
      #   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    # auditLog:
    #   destination: file
    #   format: BSON
    #   filter: '{}'

backup:
  enabled: false
  restartOnFailure: true
  image:
    repository: percona/percona-server-mongodb-operator
    tag: 1.11.0-backup
  serviceAccountName: percona-server-mongodb-operator
  # annotations:
  #   iam.amazonaws.com/role: role-arn
  # resources:
  #   limits:
  #     cpu: "300m"
  #     memory: "0.5G"
  #   requests:
  #     cpu: "300m"
  #     memory: "0.5G"
  storages:
    # s3-us-west:
    #   type: s3
    #   s3:
    #     bucket: S3-BACKUP-BUCKET-NAME-HERE
    #     credentialsSecret: my-cluster-name-backup-s3
    #     region: us-west-2
    #     prefix: ""
    # minio:
    #   type: s3
    #   s3:
    #     bucket: MINIO-BACKUP-BUCKET-NAME-HERE
    #     region: us-east-1
    #     credentialsSecret: my-cluster-name-backup-minio
    #     endpointUrl: http://minio.psmdb.svc.cluster.local:9000/minio/
    #     prefix: ""
    #   azure-blob:
    #     type: azure
    #     azure:
    #       container: CONTAINER-NAME
    #       prefix: PREFIX-NAME
    #       credentialsSecret: SECRET-NAME
  pitr:
    enabled: false
    # oplogSpanMin: 10
  tasks:
  # - name: daily-s3-us-west
  #   enabled: true
  #   schedule: "0 0 * * *"
  #   keep: 3
  #   storageName: s3-us-west
  #   compressionType: gzip
  # - name: weekly-s3-us-west
  #   enabled: false
  #   schedule: "0 0 * * 0"
  #   keep: 5
  #   storageName: s3-us-west
  #   compressionType: gzip

users:
  MONGODB_BACKUP_USER: backup
  MONGODB_BACKUP_PASSWORD: backup123456
  MONGODB_CLUSTER_ADMIN_USER: clusterAdmin
  MONGODB_CLUSTER_ADMIN_PASSWORD: clusterAdmin123456
  MONGODB_CLUSTER_MONITOR_USER: clusterMonitor
  MONGODB_CLUSTER_MONITOR_PASSWORD: clusterMonitor123456
  MONGODB_USER_ADMIN_USER: userAdmin
  MONGODB_USER_ADMIN_PASSWORD: userAdmin123456
  PMM_SERVER_USER: admin
  PMM_SERVER_PASSWORD: admin

Ege_Gunes · February 15, 2022, 6:39am

Hi @A_Niu ,

MongoDB 3.6 is not supported in PSMDB Operator and the error is most likely because of it. If you have the chance, could you please try to deploy 4.0, 4.2 or 4.4?

A_Niu · February 15, 2022, 10:48am

with 4.4.10-11, I got same error.

LAST SEEN   TYPE      REASON                   OBJECT                                       MESSAGE
8m20s       Normal    NoPods                   poddisruptionbudget/mongo-soby-2-mongod-rs   No matching pods found
13m         Normal    Killing                  pod/mongo-soby-2-rs-0                        Stopping container mongod
12m         Warning   Unhealthy                pod/mongo-soby-2-rs-0                        Readiness probe failed: dial tcp 192.168.0.107:27017: i/o timeout
8m20s       Warning   FailedScheduling         pod/mongo-soby-2-rs-0                        0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
8m15s       Warning   FailedScheduling         pod/mongo-soby-2-rs-0                        0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
8m          Normal    Scheduled                pod/mongo-soby-2-rs-0                        Successfully assigned mongo-2/mongo-soby-2-rs-0 to 172.30.131.119
7m23s       Normal    SuccessfulAttachVolume   pod/mongo-soby-2-rs-0                        AttachVolume.Attach succeeded for volume "ocid1.volume.oc1.iad.abuwcljruigae6e57zhykrazx3syltcgqzi6qgyqlzdnjlvzbcut4y6mffka"
6m46s       Normal    Pulling                  pod/mongo-soby-2-rs-0                        Pulling image "percona/percona-server-mongodb-operator:1.11.0"
6m44s       Normal    Pulled                   pod/mongo-soby-2-rs-0                        Successfully pulled image "percona/percona-server-mongodb-operator:1.11.0" in 1.714313867s
6m44s       Normal    Created                  pod/mongo-soby-2-rs-0                        Created container mongo-init
6m44s       Normal    Started                  pod/mongo-soby-2-rs-0                        Started container mongo-init
2m31s       Normal    Pulling                  pod/mongo-soby-2-rs-0                        Pulling image "percona/percona-server-mongodb:4.4.10-11"
6m42s       Normal    Pulled                   pod/mongo-soby-2-rs-0                        Successfully pulled image "percona/percona-server-mongodb:4.4.10-11" in 404.691606ms
2m30s       Normal    Created                  pod/mongo-soby-2-rs-0                        Created container mongod
2m30s       Normal    Started                  pod/mongo-soby-2-rs-0                        Started container mongod
4m1s        Warning   Unhealthy                pod/mongo-soby-2-rs-0                        Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2022-02-15T10:40:57Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x870327]

goroutine 1 [running]:
main.main()
            /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:123 +0xca7
3m31s       Warning   Unhealthy                pod/mongo-soby-2-rs-0                        Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2022-02-15T10:41:27Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x870327]

goroutine 1 [running]:
main.main()
            /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:123 +0xca7
3m1s        Warning   Unhealthy                pod/mongo-soby-2-rs-0                        Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2022-02-15T10:41:57Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x870327]

goroutine 1 [running]:
main.main()
            /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:123 +0xca7
2m31s       Warning   Unhealthy                pod/mongo-soby-2-rs-0                        Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2022-02-15T10:42:27Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x870327]

goroutine 1 [running]:
main.main()
            /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:123 +0xca7
2m31s       Normal    Killing                  pod/mongo-soby-2-rs-0                        Container mongod failed liveness probe, will be restarted
2m30s       Normal    Pulled                   pod/mongo-soby-2-rs-0                        Successfully pulled image "percona/percona-server-mongodb:4.4.10-11" in 475.493748ms
1s          Warning   Unhealthy                pod/mongo-soby-2-rs-0                        Liveness probe failed: {"level":"info","msg":"Running Kubernetes liveness check for mongod","time":"2022-02-15T10:44:57Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x870327]

goroutine 1 [running]:
main.main()
            /go/src/github.com/percona/percona-server-mongodb-operator/cmd/mongodb-healthcheck/main.go:123 +0xca7
13m         Normal    Killing                  pod/mongo-soby-2-rs-1                                 Stopping container mongod
13m         Warning   Unhealthy                pod/mongo-soby-2-rs-arbiter-0                         Readiness probe failed: dial tcp 192.168.1.67:27017: connect: connection refused
13m         Normal    Killing                  pod/mongo-soby-2-rs-arbiter-0                         Stopping container mongod-arbiter
13m         Warning   FailedToUpdateEndpoint   endpoints/mongo-soby-2-rs                             Failed to update endpoint mongo-2/mongo-soby-2-rs: Operation cannot be fulfilled on endpoints "mongo-soby-2-rs": the object has been modified; please apply your changes to the latest version and try again
8m20s       Normal    SuccessfulCreate         statefulset/mongo-soby-2-rs                           create Claim mongod-data-mongo-soby-2-rs-0 Pod mongo-soby-2-rs-0 in StatefulSet mongo-soby-2-rs success
8m20s       Normal    SuccessfulCreate         statefulset/mongo-soby-2-rs                           create Pod mongo-soby-2-rs-0 in StatefulSet mongo-soby-2-rs successful
8m20s       Normal    ExternalProvisioning     persistentvolumeclaim/mongod-data-mongo-soby-2-rs-0   waiting for a volume to be created, either by external provisioner "oracle.com/oci" or manually created by system administrator
8m9s        Normal    Provisioning             persistentvolumeclaim/mongod-data-mongo-soby-2-rs-0   External provisioner is provisioning volume for claim "mongo-2/mongod-data-mongo-soby-2-rs-0"
8m7s        Normal    ProvisioningSucceeded    persistentvolumeclaim/mongod-data-mongo-soby-2-rs-0   Successfully provisioned volume ocid1.volume.oc1.iad.abuwcljruigae6e57zhykrazx3syltcgqzi6qgyqlzdnjlvzbcut4y6mffka

Topic		Replies	Views
Failed liveness probe in an unmanaged cluster Percona Operator for MongoDB closed-no-reply	0	1040	October 28, 2022
panic: runtime error: invalid memory address or nil pointer dereference PMM 1.x	5	4018	July 17, 2019
Constant host unreachable errors within pod Percona Server for MongoDB	6	654	March 7, 2024
Mongodb node loops on restart with an OOM Percona Operator for MongoDB	2	181	July 8, 2024
Simplest defaults seem to not work? Percona Operator for MongoDB	11	1966	July 4, 2023

Invalid memory address or nil pointer dereference

Related topics