Psmdb-db-rs0 in crashloop after Upgrade

1

psmdb-db-rs0 in crashloop after Percona Mongodb Upgrade (to v6.0.19-16) and Operator Upgrade (to v1.19.0).

2

deployed on GKE v 1.30, psmdb-db - statefulSet kind, and psmdb-operator as deployment.

No specific Error messages in pod logs other than :
exec numactl --interleave=all mongod --bind_ip_all --auth --dbpath=/data/db --port=27017 --replSet=rs0 --storageEngine=wiredTiger --relaxPermChecks --clusterAuthMode=x509 --enableEncryption --encryptionKeyFile=/etc/mongodb-encryption/encryption-key --wiredTigerCacheSizeGB=4.16 --wiredTigerIndexPrefixCompression=true --quiet --tlsMode preferTLS --sslPEMKeyFile /tmp/tls.pem --tlsAllowInvalidCertificates --tlsClusterFile /tmp/tls-internal.pem --tlsCAFile /etc/mongodb-ssl/ca.crt --tlsClusterCAFile /etc/mongodb-ssl-internal/ca.crt

3

Hi @ranjithnairs84,

For checking this properly, could you please share the steps you took to upgrade the operator and cluster? Were you using helm charts? What was the previous installed version?

Thanks!

4

@ranjithnairs84 I can’t reproduce this issue. :frowning: did you use our office way Upgrade MongoDB and the Operator - Percona Operator for MongoDB to update your cluster?

5 
      we are using command to upgrade operator. Operator seem to work fine. helm upgrade "${HELM_RELEASE}" ./helmcharts/psmdb-operator -i -n ${GKE_NAMESPACE} \
      --debug

for database we areusing helm command .
helm upgrade “${HELM_RELEASE}” ./helmcharts/psmdb-db -i -n ${GKE_NAMESPACE}
–values ./helmcharts/psmdb-db/${LZ_ENVIRONMENT}-values.yaml --debug
we are doing in place upgrade.

6

Additional info on versions : Upgrade is done using helm upgrade. Helm-chart values, templates, crds picked up from percona/ percona-helm-charts(GitHub - percona/percona-helm-charts: Collection of Helm charts for Percona Kubernetes Operators.)

helm upgrade percona psmdb-operator from v1.13.0 to v1.19.0
helm upgrade percona psmdb-db from 5.0.11 to 6.0.19

trying to do in place upgrade.

Other than ERROR without any message , all other messages are informational .

Thanks

7

@Srinivasulu_Kota1 @ranjithnairs84 In our official doc, the first step is

kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.19.0/deploy/crd.yaml
kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.19.0/deploy/rbac.yaml

In order to update CRDs and RBAC. Helm update can’t update crds.

8

yes. applied crd manually as we noticed that helm did not update crd. check the version update as well.
rbac is applied using helm .
mongodb server is not providing any error message but restarting everytime.

9

@Srinivasulu_Kota1 Try to check the operator’s log and all logs from different containers (from the affected pod). Also, please check events kubectl get events --sort-by=.metadata.creationTimestamp

10

Hi @Srinivasulu_Kota1

I spotted a few issues with your upgrade procedure:

helm upgrade percona psmdb-operator from v1.13.0 to v1.19.0
helm upgrade percona psmdb-db from 5.0.11 to 6.0.19

We only support incremental upgrades of the operator, so going from 1.13.0 to 1.19.0 directly most likely will result in error, please check: operator upgrade

Also, we only support helm upgrade command for psmdb-operator, not psmdb-db, so for the db upgrade, you need to do it manually as described in: manual upgrade

Since you seem to also want to do a major upgrade (going from Mongo 5 → 6) please check major upgrade as it can be a more complex operation.

Thanks.

1 Like
11

Thank you for the quick response .
Julio and Salva

given we need to upgrade incrementally , we have decided to do a fresh install and restore the database to version 6.0 .

install happened successfully.

1 Like