PodDisruptionBudgets and safe node drains

Every node in a cluster is eventually drained. Version upgrades replace nodes, the autoscaler removes underused ones, and spot capacity disappears with two minutes of notice. Each time, the pods on that node have to go somewhere else.

A PodDisruptionBudget (PDB) is how a workload tells Kubernetes how much of that it can tolerate at once. Without one, a drain can evict every replica of a service in the same minute. With a badly written one, a drain cannot evict anything at all, and the cluster upgrade stalls.

What a PDB does and does not cover

A PDB limits voluntary disruptions that go through the Eviction API. That includes kubectl drain, managed node group and node pool upgrades, Cluster Autoscaler and Karpenter scale-down, and anything else that evicts rather than deletes.

It does not cover:

• Involuntary disruptions: a node failing, a kernel panic, a zone outage. The PDB still counts these against the budget, but it cannot prevent them.
• Direct deletes: kubectl delete pod and a Deployment's own rolling update bypass PDBs. Rolling updates are governed by maxSurge and maxUnavailable on the Deployment.
• Eviction from resource pressure: the kubelet evicting pods because a node runs out of memory does not consult PDBs.

minAvailable versus maxUnavailable

A PDB sets exactly one of two fields, as an absolute number or a percentage:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders
  namespace: checkout
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: orders

They sound equivalent and are not:

For most stateless services, maxUnavailable: 1 or a percentage is the better default. It scales with the Deployment, and it always permits one eviction when the workload is healthy. minAvailable with an absolute number is useful for quorum systems, where the requirement really is a fixed count: a three-member etcd or ZooKeeper ensemble needs minAvailable: 2.

Check what the budget currently allows with kubectl get pdb -n checkout. The ALLOWED DISRUPTIONS column is the number of pods that can be evicted right now. If it reads zero for a healthy workload, the PDB will block every drain.

The single-replica trap

The most common problem is a Deployment with one replica and a PDB like this:

spec:
  minAvailable: 1

Allowed disruptions is permanently zero. Every drain of the node hosting that pod waits, retries, and eventually times out. On a cluster with dozens of these, a routine upgrade never finishes.

The same happens with maxUnavailable: 0, and with minAvailable: 100%. All three tell Kubernetes the pod may never be evicted, which is rarely true and never useful during maintenance.

The options for a single-replica workload are honest ones:

• Run two replicas if the service cannot tolerate a restart. That is what the PDB was trying to express.
• Accept the restart and remove the PDB, or set maxUnavailable: 1.
• Fix the application if it cannot run two replicas because of local state or a singleton job. Leader election solves the singleton case.

A policy engine such as Kyverno or Gatekeeper can reject PDBs that allow zero disruptions, or warn when a PDB selects a workload with fewer than two replicas.

Unhealthy pods and stuck drains

A related trap: a Deployment with three replicas and maxUnavailable: 1, where one pod is in CrashLoopBackOff. That pod already counts as unavailable, so allowed disruptions is zero. The drain cannot evict the broken pod, which is the one pod nobody would miss.

The unhealthyPodEvictionPolicy field addresses this. It is beta and enabled by default since Kubernetes 1.27:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders
spec:
  unhealthyPodEvictionPolicy: AlwaysAllow
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: orders

With AlwaysAllow, pods that are running but not ready can be evicted regardless of the budget. The default, IfHealthyBudget, only allows that when the workload is otherwise within budget.

What a drain actually does

kubectl drain cordons the node, then asks the Eviction API to evict each pod. An eviction that would violate a PDB returns HTTP 429, and drain retries until it succeeds or the timeout expires:

kubectl drain ip-10-0-42-17.eu-west-1.compute.internal \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

Managed services wrap this with their own limits, and they differ:

• EKS managed node groups fail the upgrade with a PodEvictionFailure if pods cannot be evicted within 15 minutes, unless you force the update, which ignores PDBs.
• GKE respects PDBs during node upgrades for up to one hour per node, then evicts anyway.
• AKS fails the upgrade when a PDB blocks eviction.

Either outcome is bad. A failed upgrade leaves a mixed-version node pool. A forced one ignores the budget the team thought was protecting them. The fix in both cases is PDBs that allow progress.

Also check terminationGracePeriodSeconds. Each eviction waits for the pod to shut down. A 300-second grace period on a workload evicted one pod at a time adds minutes per node.

Spread replicas so the budget means something

A PDB limits how many pods are evicted at once. It does nothing about where they run. If all three replicas sit on one node, draining that node takes one pod, waits for its replacement, takes the next, and so on. The budget is respected, but if that node had failed instead of being drained, all three would have gone at once.

Topology spread constraints place replicas across nodes and zones:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: orders
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: orders

ScheduleAnyway is the safer default. DoNotSchedule gives a stronger guarantee but can leave pods pending when a zone lacks capacity.

Where to start

1. List PDBs that allow zero disruptions across the cluster: kubectl get pdb -A and look for 0 in ALLOWED DISRUPTIONS.
2. Fix single-replica PDBs by adding a replica or relaxing the budget.
3. Default stateless services to maxUnavailable: 1 or a percentage, generated by your deployment templates rather than written by hand.
4. Set unhealthyPodEvictionPolicy: AlwaysAllow on stateless workloads.
5. Add topology spread constraints for zone and hostname to every multi-replica Deployment.
6. Drain one node in staging before each cluster upgrade, and time it. If it takes longer than expected, find out which PDB was waiting.