Helm Upgrade from 1.11 to 1.12 confusing crds

rriverak · August 15, 2022, 9:55am

Hello Team,
we have adjusted the configuration of our clusters according to the release notes for v1.12.

However, some of the settings that were moved do not work as we expected that.

The spec.mongod section is removed from the Custom Resource configuration. Starting from now, mongod options should be passed to Replica Sets using spec.replsets.[].configuration key…

For example, we moved operationProfiling from spec.mongod to spec.replsets[0].configuration without success. The desired OperationProfiling is not used.

Using spec.mongod as before, everything works as expected.

If we take a look at the v1.12 CRDs in the Helm Chart Repo we see that the spec.mongod section is still there and has not been removed as advertised.

github.com

percona/percona-helm-charts/blob/123ca017d2f5133808f4959be19bfe3b25e1a469/charts/psmdb-operator/crds/crd.yaml#L625


      
            type: string
          imagePullSecrets:
            items:
              properties:
                name:
                  type: string
              type: object
            type: array
          initImage:
            type: string
          mongod:
            properties:
              auditLog:
                properties:
                  destination:
                    type: string
                  filter:
                    type: string
                  format:
                    type: string
                type: object

What is now the right way to configure these things?

Here is a rendered version of one of our clusters:

apiVersion: psmdb.percona.com/v1-12-0
kind: PerconaServerMongoDB
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"psmdb.percona.com/v1-12-0","kind":"PerconaServerMongoDB"}
  name: test-psmdb-db
  finalizers:
    - delete-psmdb-pods-in-order
    - delete-psmdb-pvc
spec:
  pause: false
  unmanaged: false
  image: "percona/percona-server-mongodb:4.4.8-9"
  imagePullPolicy: "Always"
  multiCluster:
    enabled: false
  secrets:
    users: test-psmdb-db-secrets
    encryptionKey: test-psmdb-db-mongodb-encryption-key
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: 4.4-recommended
    schedule: 0 2 * * *
    setFCV: false
  pmm:
    enabled: false
    image: "percona/pmm-client:2.28.0"
    serverHost: monitoring-service
  replsets:
  - name: rs0
    size: 3
    configuration: |
      operationProfiling:
        mode: slowOp
        slowOpThresholdMs: 100
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    nodeSelector:
      dedicated: database
    tolerations:
      - effect: NoSchedule
        key: dedicated
        operator: Equal
        value: database
    livenessProbe:
      failureThreshold: 4
      initialDelaySeconds: 60
      periodSeconds: 30
      startupDelaySeconds: 7200
      successThreshold: 1
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 8
      initialDelaySeconds: 10
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 5
    storage:
      engine: wiredTiger
      inMemory:
        engineConfig:
          inMemorySizeRatio: 0.5
      wiredTiger:
        collectionConfig:
          blockCompressor: snappy
        engineConfig:
          cacheSizeRatio: 0.5
          directoryForIndexes: false
          journalCompressor: snappy
        indexConfig:
          prefixCompression: true
    podDisruptionBudget:
      maxUnavailable: 1
    expose:
      enabled: false
      exposeType: ClusterIP
    nonvoting:
      enabled: false
      size: 3
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 10Gi
  mongod:
    setParameter:
      ttlMonitorSleepSecs: 60
      wiredTigerConcurrentReadTransactions: 128
      wiredTigerConcurrentWriteTransactions: 128
    storage:
      engine: wiredTiger
      inMemory:
        engineConfig:
          inMemorySizeRatio: 0.9
      wiredTiger:
        engineConfig:
          cacheSizeRatio: 0.5
          directoryForIndexes: false
          journalCompressor: snappy
        collectionConfig:
          blockCompressor: snappy
        indexConfig:
          prefixCompression: true
    operationProfiling:
      mode: slowOp
      slowOpThresholdMs: 100
      rateLimit: 100
  backup:
    enabled: true
    image: "percona/percona-server-mongodb-operator:1.11.0-backup"
    serviceAccountName: percona-server-mongodb-operator
    storages:
      s11-dev-backup:
        s3:
          bucket: psmdb-backup
          credentialsSecret: test-psmdb-db-backup-secret
          endpointUrl: https://xxx
          region: us-east-1
        type: s3
    pitr:
      enabled: true
    tasks:
      - compressionType: gzip
        enabled: true
        keep: 7
        name: daily-dev
        schedule: 0 18 * * *
        storageName: s11-dev-backup

we would be very happy to be enlightened

best regards,

Ricardo

rriverak · August 15, 2022, 12:07pm

After checking the operator code, there seems to be a problem with spec.crVersion.
It seems our PerconaServerMongoDB Resources have the wrong version.

As can be seen here, the spec.crVersion field is currently not managed by psmdb-db the HelmChart.

github.com

percona/percona-helm-charts/blob/main/charts/psmdb-db/templates/cluster.yaml

apiVersion: psmdb.percona.com/v{{ .Chart.AppVersion | replace "." "-" }}
kind: PerconaServerMongoDB
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"psmdb.percona.com/v{{ .Chart.AppVersion | replace "." "-" }}","kind":"PerconaServerMongoDB"}
  name: {{ include "psmdb-database.fullname" . }}
  labels:
{{ include "psmdb-database.labels" . | indent 4 }}
  finalizers:
{{ .Values.finalizers | toYaml | indent 4 }}
spec:
  pause: {{ .Values.pause }}
  unmanaged: {{ .Values.unmanaged }}
  {{- if .Values.platform }}
  platform: {{ .Values.platform }}
  {{- end }}
  {{- if .Values.clusterServiceDNSSuffix }}
  clusterServiceDNSSuffix: {{ .Values.clusterServiceDNSSuffix }}
  {{- end }}

This file has been truncated. show original

In our case, the spec.crVersion is set to 1.10.0, which should be responsible for the problems with the Config. How should the field be handled? Does the operator take care of it ? Do we have to patch the field ourselves via Helm?

rriverak · August 15, 2022, 2:31pm

ok, i think there is a misunderstanding here…

As i can read here, there are 2 Update variants.

Semi-automatic upgrade
Manual upgrade

we thought Helm would be a third option that would make the other steps unnecessary.

Full-automatic upgrade

I think the part of the documentation only refers to a deployment with deploy/cr.yaml.

For example, the psmdb-operator Helm Chart takes care of the RBAC update from the documentation but doesn’t take care of the other parts of the update.

github.com

percona/percona-helm-charts/blob/main/charts/psmdb-operator/templates/role.yaml

{{- if .Values.watchNamespace }}
kind: ClusterRole
{{- else }}
kind: Role
{{- end }}
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: {{ include "psmdb-operator.fullname" . }}
  labels:
{{ include "psmdb-operator.labels" . | indent 4 }}
rules:
  - apiGroups:
    - psmdb.percona.com
    resources:
    - perconaservermongodbs
    - perconaservermongodbs/status
    - perconaservermongodbbackups
    - perconaservermongodbbackups/status
    - perconaservermongodbrestores
    - perconaservermongodbrestores/status

This file has been truncated. show original

These manual apply is not needed with the psmdb-operator Helm Chart…

$ kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.12.0/deploy/crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.12.0/deploy/rbac.yaml

Patching the operator yourself is also unnecessary.

$ kubectl patch deployment percona-server-mongodb-operator \
   -p'{"spec":{"template":{"spec":{"containers":[{"name":"percona-server-mongodb-operator","image":"percona/percona-server-mongodb-operator:1.12.0"}]}}}}'

But it seems that the last part of the documentation is absolutely necessary

$ kubectl patch psmdb my-cluster-name --type=merge --patch '{
   "spec": {
      "crVersion":"1.12.0",
      "image": "percona/percona-server-mongodb:4.4.13-13",
      "backup": { "image": "percona/percona-server-mongodb-operator:1.12.0-backup" },
      "pmm": { "image": "percona/pmm-client:2.27.0" }
   }}'

However, the fields image, backup and pmm are also managed via the HelmChart.

github.com

percona/percona-helm-charts/blob/b96e32ba26d7b9cc2f464625e0712d4232114bf2/charts/psmdb-db/values.yaml#L31


      
          multiCluster:
            enabled: false
            # DNSSuffix: svc.clusterset.local
          updateStrategy: SmartUpdate
          upgradeOptions:
            versionServiceEndpoint: https://check.percona.com
            apply: 5.0-recommended
            schedule: "0 2 * * *"
            setFCV: false
          
          
image:
            repository: percona/percona-server-mongodb
            tag: 5.0.7-6
          
          
imagePullPolicy: Always
          # imagePullSecrets: []
          # tls:
          #   # 90 days in hours
          #   certValidityDuration: 2160h
          secrets: {}
            # If you set users secret here, it will not be constructed from the values at the

Only crVersion was omitted in the psmdb-db Chart.

It is a 2 phase update anyway, first the psmdb-operator and then the psmdb-db Chart.
Why doesn’t psmdb-db set the appropriate spec.crVersion ?

I think at least the documentation should be expanded…

a.starman · August 18, 2022, 5:44am

Hi Ricardo! I am also a bit confused about upgrading MongoDB using HELM charts.

You found out that HELM charts did everything necessary except to patch crVersion. Why do we need to patch crVersion at all? In my psmdb when i edit it (kubectl edit perconaservermongodb.psmdb.percona.com/databasemgmt-psmdb-db -n mongodb)
there is no crVersion.
With HELM charts upgrades are probably required to be also done incremental (e.g. 1.11 to 1.13 must be done via 1.12)?

Best regards, Anton

rriverak · August 18, 2022, 7:16am

Hey Anton,
it is probably the case that the operator includes all versions.
The PerconaMongoDB CRD then decides in which “mode” the operator runs for this single database.

If you use a 1.12 operator and your database uses spec.crVersion=1.10.0, the operator will use the 1.10.0 code internally. So you can simply upgrade the operator and upgrade the databases later.

So I think there is no need for incremental upgrades for minor / patch versions of the operator.

There is a note in the code regarding the missing spec.crVersion.
The operator always uses spec.crVersion internally to compare versions.

github.com

percona/percona-server-mongodb-operator/blob/986b4579634babe559a9a2ef8a8524a30645140b/pkg/apis/psmdb/v1/psmdb_types.go#L840


      
          			apiVersion = strings.Replace(strings.TrimPrefix(newCR.APIVersion, "psmdb.percona.com/v"), "-", ".", -1)
          		}
          	}
          
          
	cr.Spec.CRVersion = apiVersion
          
          
	return nil
          }
          
          
func (cr *PerconaServerMongoDB) Version() *v.Version {
          	return v.Must(v.NewVersion(cr.Spec.CRVersion))
          }
          
          
func (cr *PerconaServerMongoDB) CompareVersion(version string) int {
          	if len(cr.Spec.CRVersion) == 0 {
          		cr.setVersion()
          	}
          
          
	// using Must because "version" must be right format
          	return cr.Version().Compare(v.Must(v.NewVersion(version)))
          }

If spec.crVersion is empty like in your case, then a fallback is used which fills the version internally.

github.com

percona/percona-server-mongodb-operator/blob/986b4579634babe559a9a2ef8a8524a30645140b/pkg/apis/psmdb/v1/psmdb_types.go#L845


      
          
          
	return nil
          }
          
          
func (cr *PerconaServerMongoDB) Version() *v.Version {
          	return v.Must(v.NewVersion(cr.Spec.CRVersion))
          }
          
          
func (cr *PerconaServerMongoDB) CompareVersion(version string) int {
          	if len(cr.Spec.CRVersion) == 0 {
          		cr.setVersion()
          	}
          
          
	// using Must because "version" must be right format
          	return cr.Version().Compare(v.Must(v.NewVersion(version)))
          }
          
          
const (
          	internalPrefix = "internal-"
          	userPostfix    = "-users"
          )

As you can see setVersion() use kubectl.kubernetes.io/last-applied-configuration as fallback.

github.com

percona/percona-server-mongodb-operator/blob/986b4579634babe559a9a2ef8a8524a30645140b/pkg/apis/psmdb/v1/psmdb_types.go#L816


      
          		Kind:       gvk.Kind,
          		Name:       cr.GetName(),
          		UID:        cr.GetUID(),
          		Controller: &trueVar,
          	}, nil
          }
          
          
// setVersion sets the API version of a PSMDB resource.
          // The new (semver-matching) version is determined either by the CR's API version or an API version specified via the CR's annotations.
          // If the CR's API version is an empty string, it returns "v1"
          func (cr *PerconaServerMongoDB) setVersion() error {
          	if len(cr.Spec.CRVersion) > 0 {
          		return nil
          	}
          
          
	apiVersion := version.Version
          
          
	if lastCR, ok := cr.Annotations["kubectl.kubernetes.io/last-applied-configuration"]; ok {
          		var newCR PerconaServerMongoDB
          		err := json.Unmarshal([]byte(lastCR), &newCR)
          		if err != nil {

It could be that spec.crVersion is from an earlier version and is no longer used for new databases.

best regards,

Ricardo

a.starman · August 18, 2022, 9:52am

Hi Ricardo!

Thank you for explanation.

I did test using HELMs:

installed operator and 2 databases, all version 1.11.0
upgraded operator to 1.12.0
upgraded one database to 1.12.0, other database stayed 1.11.0

After upgrade the value of kubectl.kubernetes.io/last-applied-configuration was successfully applied, at upgraded database to 1.12.0, at other stayed 1.11.0.
So i guess there is no need to patch spec.crVersion when using HELMs.

Do you maybe know if i need to do incremental upgrade of databases or not? Can i just run HELM to upgrade from e.g. 1.10 to 1.12? What does it do, it replaces binaries and sets operator to work in new mode and using new settings… i guess it can be done?

Best regards, Anton

Topic		Replies	Views
Mongod configuration Percona Operator for MongoDB	2	404	March 22, 2024
Basic cluster with TLS not working, operator k8s/helm setup Percona Server for MongoDB mongodb , psmdb-operator	18	2949	December 18, 2022
Percona MongoDB operator backup failure Percona Operator for MongoDB percona , mongodb	4	1226	October 18, 2022
CRD resources - no matches for kind "PerconaServerMongoDBBackup" MongoDB community , percona , mongodb , new-release	1	931	March 16, 2022
Helm upgrade Operator yields errors Percona Operator for MongoDB	2	694	March 15, 2022

Related topics