Kubernetes cluster upgrades without drama

Kubernetes ships three minor releases a year, and each is supported for roughly fourteen months upstream. Managed providers stretch that a little, then start charging extended support fees or upgrading clusters on their own schedule. Either way, upgrades are not optional. The teams that find them painful usually do them rarely, which makes each one bigger and riskier than it needs to be.

An upgrade has three parts that fail in different ways: the APIs your manifests use, the control plane, and the nodes. Treat them separately.

Find removed APIs first

The most common upgrade failure is not the upgrade itself. It is a deploy a week later that fails because a Helm chart still asks for an API version the new control plane no longer serves. Kubernetes deprecates API versions for several releases before removing them, so there is always warning, but only if someone looks.

Recent removals that still catch teams:

Two tools cover most of the detection work. Pluto, from Fairwinds, scans files and Helm releases:

# Rendered manifests in the repository
pluto detect-files -d deploy/ --target-versions k8s=v1.33.0
 
# Helm release metadata in the cluster
pluto detect-helm -o wide --target-versions k8s=v1.33.0

kubent (kube-no-trouble) looks at what is running in the cluster, including the last-applied configuration that kubectl apply stores:

kubent --target-version 1.33.0 --exit-error

Neither tool sees everything. Manifests generated at deploy time, operators that create resources from code, and CI jobs that call the API directly can all use old versions without leaving a trace in files. For those, the API server itself is the best source. It exports the apiserver_requested_deprecated_apis metric, labeled by group, version and resource, and adds a k8s.io/deprecated annotation to audit log entries. Managed services surface the same data: EKS has upgrade insights, and GKE shows deprecation insights in the console.

Run Pluto in CI against rendered manifests with the next version as the target. That turns the problem into a failing pull request months before the upgrade, instead of a failing deploy afterwards.

Respect version skew

Components in a cluster do not all have to run the same version, but the allowed gaps are specific:

• The control plane moves one minor version at a time. Going from 1.30 to 1.33 means three upgrades, not one.
• The kubelet may be up to three minor versions older than the API server (since 1.28; it was two before). It may never be newer.
• kube-controller-manager and kube-scheduler may be at most one minor version older than the API server.
• kubectl is supported within one minor version, older or newer, of the API server.

The three-version kubelet window lets you upgrade the control plane twice before touching nodes. That is useful in an emergency, but it is not a plan. Nodes that lag too long block the next control plane upgrade, and a large node jump is harder to roll back than a small one.

Check add-ons too. CNI plugins, CSI drivers, ingress controllers, cert-manager and the cluster autoscaler each publish a compatibility range. The cluster autoscaler, for instance, is released per Kubernetes minor version and should match the control plane.

Nodes: surge and disruption budgets

Node upgrades replace machines. Every pod on an old node gets evicted and rescheduled, and this is where application behavior matters more than the platform.

Upgrades drain nodes through the eviction API, which respects PodDisruptionBudgets. A sensible budget on every replicated workload lets the upgrade proceed without taking a service below its minimum. Two common mistakes stall upgrades instead:

• A budget that allows zero disruptions, such as maxUnavailable: 0 or minAvailable equal to the replica count. The drain waits forever, or until the provider gives up and forces it.
• Single-replica workloads with a budget. There is no way to evict the one pod without violating it.

Managed node pools control how many nodes change at once. On GKE, surge upgrades are tuned with maxSurge and maxUnavailable; GKE also offers blue-green node pool upgrades with a soak period before the old nodes go away. EKS managed node groups take an updateConfig with maxUnavailable or maxUnavailablePercentage. AKS node pools have a max surge setting.

gcloud container node-pools update payments-pool \
  --cluster=prod-eu-west \
  --max-surge-upgrade=2 \
  --max-unavailable-upgrade=0

Surging adds capacity before removing it, which avoids running short during the upgrade. It needs quota for the extra nodes, which is easy to forget for GPU or reserved instance pools.

If you run Karpenter, node upgrades follow from changing the AMI or node class. Karpenter detects drift and replaces nodes, constrained by its disruption budgets on the NodePool. The same PodDisruptionBudget rules apply.

In-place or blue/green clusters

There are two broad strategies.

In-place: upgrade the control plane, then roll the node pools. This is the default on every managed service and the right choice for most clusters. It keeps the cluster's identity: load balancers, DNS, IAM bindings and persistent volumes stay where they are.

Blue/green clusters: build a new cluster at the target version, deploy everything to it, shift traffic, then delete the old one. It makes rollback trivial, since the old cluster is untouched until the end. It also tests that the cluster can be rebuilt from code, which is valuable on its own.

The cost is everything that is attached to a specific cluster. Persistent volumes do not move by themselves. Ingress addresses change unless traffic is shifted at DNS or a global load balancer. Workload identity bindings, secrets, and anything installed by hand must be recreated.

Blue/green suits stateless clusters with fully automated provisioning, or a cluster several versions behind where catching up in place would take months. For the rest, in-place with good preparation is less work and less risk.

Upgrade checklist

1. Read the upstream changelog's "urgent upgrade notes" and your provider's release notes for the target version.
2. Run Pluto and kubent against the target version, and check apiserver_requested_deprecated_apis. Fix everything before starting.
3. Check every add-on's supported Kubernetes range and upgrade the ones that need it.
4. Audit PodDisruptionBudgets for zero-disruption budgets and single replicas.
5. Confirm quota for surge nodes.
6. Upgrade staging, soak, then production control plane, then node pools one at a time.
7. Put the next upgrade on the calendar. Staying within one or two versions of the latest makes every upgrade small.