ACTION REQUIRED: Percona Everest and Bitnami Container Catalog Changes ⚠️

Percona Everest
1

Attention Percona Everest Users: Bitnami is restructuring its container catalog on September 29, 2025. To ensure your Percona Everest operations remain seamless, you must take a few proactive steps now.

Mandatory Fix to Prevent Future Issues :white_check_mark:

To avoid installation and uninstallation failures, every Percona Everest user must perform the following actions. The everestctl CLI caches old chart versions, which can cause issues even after upgrading Everest. By clearing the cache and, if necessary, upgrading your charts, you will ensure all future operations are safe.

  1. Clear the everestctl cache. This step is mandatory for all users, regardless of your Everest version. The everestctl CLI caches chart information. To ensure it uses the new image references for all future operations, you must clear this cache.

    • On Linux:

      rm -rf "$XDG_CACHE_HOME/everestctl"
# Or if $XDG_CACHE_HOME is not set:
rm -rf "$HOME/.cache/everestctl"

    • On macOS:

      rm -rf "$HOME/Library/Caches/everestctl"

    • On Windows (in PowerShell):

      Remove-Item "$env:LocalAppData/everestctl" -Recurse -Force

    • On Plan 9:

      rm -r "$home/lib/cache/everestctl"

If you are on Percona Everest v1.8.1 or earlier:

You must manually update your Helm charts to use the new image. If you are on v1.9.0 or later, you can skip these steps.

  1. List all installed Everest charts. Find all Helm releases for both the everest and everest-db-namespace charts. Note the release name, namespace, and APP VERSION for each one.

    helm list --all-namespaces | grep everest
a1             a1             1          2025-09-18 16:46:31.130925768 +0100 WEST   deployed        everest-db-namespace-1.8.1      1.8.1
everest        everest        1          2025-09-18 16:45:15.362666552 +0100 WEST   deployed        everest-db-namespace-1.8.1      1.8.1
everest-system everest-system 1          2025-09-18 16:44:45.138480161 +0100 WEST   deployed        everest-1.8.1                   1.8.1

  2. Upgrade the main Everest chart. Upgrade the everest chart (e.g., everest-system) using its specific details.

    helm upgrade everest-system percona/everest \
--namespace everest-system \
--version 1.8.1
Release "everest-system" has been upgraded. Happy Helming!
NAME: everest-system
LAST DEPLOYED: Thu Sep 18 16:52:18 2025
NAMESPACE: everest-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Everest has been successfully upgraded to version 1.8.1!

  3. Upgrade the database namespace charts. Repeat the helm upgrade command for each everest-db-namespace chart you found in step 1.

    helm upgrade a1 percona/everest-db-namespace \
--namespace a1 \
--version 1.8.1
Release "a1" has been upgraded. Happy Helming!
NAME: a1
LAST DEPLOYED: Thu Sep 18 16:52:45 2025
NAMESPACE: a1
STATUS: deployed
REVISION: 2
TEST SUITE: None

    helm upgrade everest percona/everest-db-namespace \
--namespace everest \
--version 1.8.1
Release "everest" has been upgraded. Happy Helming!
NAME: everest
LAST DEPLOYED: Thu Sep 18 16:53:02 2025
NAMESPACE: everest
STATUS: deployed
REVISION: 2
TEST SUITE: None

  4. Restart Everest Operator.

    kubectl -n everest-system rollout restart deploy/everest-operator

You’re all set! By following these steps, you’ve ensured your Percona Everest deployment is resilient to the Bitnami container catalog changes.

What’s Changing with Bitnami? :collision:

Bitnami has announced it is restructuring its container catalog and moving most container images to a legacy repository on September 29, 2025. After this date, any installations, updates, or uninstallations that rely on the old image references will fail.

Percona Everest is affected because it depends on the bitnami/kubectl image for several critical tasks:

  • The everest-operators-installer job during installation and namespace provisioning.
  • Cleanup jobs like pre-delete, csv-cleanup, and psp-cleanup during uninstallation.

This means that after September 29, 2025, your Everest deployments could become stuck if you try to perform these operations.

Troubleshooting (If You’re Already Stuck After Sept 29) :hammer_and_wrench:

If you find yourself with a failed operation after the Bitnami change takes effect, use these steps to resolve the issue.

Scenario 1: Installation or Namespace Provisioning Is Stuck

Your everestctl install, everestctl namespaces add, or everestctl namespaces update command is stalled, showing a status like this:

ℹ️ Installing Everest version 1.8.1

✅ Installing Everest Helm chart
✅ Ensuring Everest API deployment is ready
✅ Ensuring Everest operator deployment is ready
✅ Ensuring OLM components are ready
✅ Ensuring monitoring stack is ready
∙∙● Provisioning database namespace 'everest'

Esc/Ctrl+c quit

Step-by-Step Fix

  1. Open a new terminal and leave the stalled process running.

  2. Verify the stuck Helm release. Confirm the everest-db-namespace chart is in a pending-install state. Note the release name and namespace from the output. In this example, it’s everest in the everest namespace.

    helm list --all-namespaces --pending
NAME       NAMESPACE      REVISION   UPDATED                                STATUS          CHART                           APP VERSION
everest    everest        1          2025-09-18 16:12:36.237739864 +0100 WEST   pending-install everest-db-namespace-1.8.1      1.8.1

  3. Confirm the cause. Check the pod status to ensure the problem is an ImagePullBackOff for the bitnami/kubectl:latest image.

    kubectl get pod -l job-name=everest-operators-installer --all-namespaces -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,IMAGE:.spec.containers[0].image,WAITING_REASON:.status.containerStatuses[0].state.waiting.reason'
NAME                                     NAMESPACE  IMAGE                       WAITING_REASON
everest-operators-installer-rllfg        everest    bitnami/kubectl:latest      ImagePullBackOff

  4. Delete the stuck job.

    kubectl delete job everest-operators-installer -n everest
job.batch "everest-operators-installer" deleted from everest namespace

  5. Check that the install continues. The original everestctl command should now proceed. Note the APP VERSION of your Helm release, as you’ll need it in the next step.

    helm list -n everest
NAME       NAMESPACE      REVISION   UPDATED                                STATUS          CHART                           APP VERSION
everest    everest        1          2025-09-18 16:12:36.237739864 +0100 WEST   deployed        everest-db-namespace-1.8.1      1.8.1

  6. Upgrade the Helm release. Use the release name, namespace, and APP VERSION from the previous steps to upgrade the chart.

    helm upgrade everest percona/everest-db-namespace \
--namespace everest \
--version 1.8.1
Release "everest" has been upgraded. Happy Helming!
NAME: everest
LAST DEPLOYED: Thu Sep 18 16:18:50 2025
NAMESPACE: everest
STATUS: deployed
REVISION: 2
TEST SUITE: None

  7. Restart Everest Operator.

    kubectl -n everest-system rollout restart deploy/everest-operator

  8. Your immediate issue is now resolved. Remember to complete the steps in the Mandatory Fix to Prevent Future Issues section if you haven’t already, to avoid this problem in the future.

Scenario 2: Uninstallation Is Stuck

Your everestctl uninstall command is stalled while deleting a namespace:

✅ Deleting database clusters in namespace 'everest'
✅ Deleting backup storages in namespace 'everest'
✅ Deleting monitoring instances in namespace 'everest'
●∙∙ Deleting database namespace 'everest'

Esc/Ctrl+c quit

Step-by-Step Fix

  1. Open a new terminal and leave the stalled process running.

  2. Verify the stuck Helm release. Confirm the everest-db-namespace chart is in an uninstalling state. Note the release name and namespace.

    helm list --all-namespaces --uninstalling
NAME       NAMESPACE      REVISION   UPDATED                                STATUS       CHART                           APP VERSION
everest    everest        1          2025-09-18 16:37:49.047308243 +0100 WEST   uninstalling everest-db-namespace-1.8.1      1.8.1

  3. Identify the stuck job. List the jobs in the namespace and find the one with a Running status. Note the job’s name.

    kubectl get jobs -n everest
NAME                                  STATUS    COMPLETIONS   DURATION   AGE
everest-helm-pre-delete-hook-288662   Running   0/1           25s        25s

  4. Confirm the cause. Check the pod to ensure it’s an ImagePullBackOff for the bitnami/kubectl:latest image. Note the pod’s name and container name.

    kubectl get pod -l job-name=everest-helm-pre-delete-hook-288662 --all-namespaces -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,CONTAINER_NAME:.spec.containers[0].name,IMAGE:.spec.containers[0].image,WAITING_REASON:.status.containerStatuses[0].state.waiting.reason'
NAME                                           NAMESPACE      CONTAINER_NAME                  IMAGE                       WAITING_REASON
everest-helm-pre-delete-hook-288662-dlddp      everest        everest-helm-pre-delete-hook    bitnami/kubectl:latest      ImagePullBackOff

  5. Patch the pod. Replace the bitnami/kubectl image with the new percona/everest-helmtools:0.0.1 image. Make sure to replace the pod and container names with your own.

    kubectl patch pod everest-helm-pre-delete-hook-288662-dlddp -n everest -p '{"spec":{"containers":[{"name":"everest-helm-pre-delete-hook","image":"percona/everest-helmtools:0.0.1"}]}}'
pod/everest-helm-pre-delete-hook-288662-dlddp patched

  6. Repeat as needed. The uninstallation should now progress, but it might get stuck again on other cleanup jobs like psp-cleanup. Repeat steps 3-5, identifying and patching any new stuck jobs until the uninstallation completes.

  7. Your immediate issue is now resolved. Remember to complete the steps in the Mandatory Fix to Prevent Future Issues section if you haven’t already, to avoid this problem in the future.

2 Likes