Upgrade to Everest 1.8.0 fails with context canceled due to hanging pre-upgrade hook

Percona Everest
1

Sure! Here’s the bug report text translated into clear and professional English:

Title: Upgrade to Everest 1.8.0 fails with context canceled due to hanging pre-upgrade hook

:pushpin: Problem Description

Attempting to upgrade the everest-core release to version 1.8.0 fails with the following issues:

  • Helm upgrade fails with:

    UPGRADE FAILED: context canceled

  • The everest-helm-pre-upgrade-hook-<suffix> Job remains in the active state and never completes.

  • The job logs contain an authentication error when trying to interact with the Kubernetes API:

    ❌ failed to get server groups: the server has asked for the client to provide credentials
could not retrieve Everest version

:repeat_button: Steps to Reproduce

  1. Run the upgrade command:

    helm upgrade everest-core percona/everest --namespace everest-system --version "1.8.0"

  2. Helm attempts to run the pre-upgrade hook job.

  3. The job remains active and never completes, eventually resulting in:

    Release everest-core has been cancelled.
Error: UPGRADE FAILED: context canceled

:magnifying_glass_tilted_left: Details

  • The pre-upgrade hook job is created but does not finish.

  • Job logs:

    Checking requirements for upgrade to version 1.8.0
❌ failed to get server groups: the server has asked for the client to provide credentials
could not retrieve Everest version

:hammer_and_wrench: Environment

  • Chart: percona/everest v1.8.0

  • Kubernetes: v1.30.2 (OIDC + RBAC enabled)

  • Helm: v3.x

  • Namespace: everest-system

:light_bulb: Root Cause Hypothesis

The hook job is likely trying to query the Kubernetes API or the Everest API, but fails due to:

  • Missing or misconfigured ServiceAccount or insufficient RBAC permissions.

  • Cluster requires external authentication (e.g., OIDC), and the in-cluster job does not have access to proper credentials.

:white_check_mark: Expected Behavior

The pre-upgrade hook should either:

  • Complete successfully if permissions are sufficient, or

  • Fail clearly with a descriptive error — not hang indefinitely.

The Helm upgrade should not fail with a timeout and “context canceled” if version discovery fails.

:megaphone: Recommendations

  • Improve error handling in the pre-upgrade hook.

  • Document required permissions and service account roles for upgrade hooks.

  • Add fallback behavior when version check fails due to auth issues.

Let me know if you want this in Markdown format or ready for GitHub/Percona submission.

2

I added permissions for service acount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: everest-helm-pre-upgrade-hook-binding
labels:
app.kubernetes.io/component: upgrade-hook
app.kubernetes.io/name: everest
subjects:

  • kind: ServiceAccount
    name: everest-helm-pre-upgrade-hook
    namespace: everest-system
    roleRef:
    kind: ClusterRole
    name: everest-pre-upgrade-hook-role
    apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: everest-pre-upgrade-hook-role
rules:

  • apiGroups: [“”]
    resources: [“pods”, “services”, “namespaces”]
    verbs: [“get”, “list”]
  • apiGroups: [“apiextensions.k8s.io”]
    resources: [“customresourcedefinitions”]
    verbs: [“get”, “list”]
  • apiGroups: [“apps”]
    resources: [“deployments”]
    verbs: [“get”, “list”]
  • apiGroups: [“”]
    resources: [“nodes”]
    verbs: [“get”, “list”]
  • nonResourceURLs: [“/version”, “/healthz”, “/apis”, “/api”]
    verbs: [“get”]

but stuck with next issue:

Checking requirements for upgrade to version 1.8.0

:cross_mark: an empty namespace may not be set when a resource name is provided

could not retrieve subscriptio

3

The same with everestctl:

$ everestctl upgrade --in-cluster --dry-run --version-metadata-url=https://check.percona.com
:cross_mark: an empty namespace may not be set when a resource name is provided
could not retrieve subscription

4

For anyone still interested in this, @pstekunov got his answer on github: everest 1.8.0 upgrade error · Issue #1568 · percona/everest · GitHub

The workaround is to remove the app.kubernetes.io/managed-by=everest label from the everest-system namespace:

kubectl label namespace everest-system app.kubernetes.io/managed-by-
1 Like