Kubernetes autoscaling from pods to nodes

Autoscaling on Kubernetes is not one feature. It is a chain. Something decides that a workload needs more pods. The scheduler tries to place them. If there is no room, something else decides the cluster needs more nodes, and a cloud provider has to deliver them. Each link has its own configuration, delays and failure modes, and most autoscaling problems come from how the links interact rather than from any one of them.

Pod autoscaling: pick the right signal

The Horizontal Pod Autoscaler (HPA) adjusts the replica count of a Deployment or StatefulSet to keep a metric near a target. The quality of the result depends almost entirely on which metric you choose.

CPU utilization is the default, and it works for services where CPU tracks load: request handlers doing real computation. It works poorly for services that wait on I/O, where CPU stays low while latency climbs, and for anything that consumes a queue.

Utilization is measured against the container's CPU request. Without a request, the HPA cannot compute it at all. With a request that is far from actual usage, the target percentage stops meaning anything.

Request rate per pod or in-flight requests per pod is often a better signal for web services. If one replica handles 50 requests per second comfortably, scale to keep each pod near that. This needs a custom metrics adapter, such as the Prometheus Adapter, or KEDA, described below.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 40
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "50"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60

The behavior block matters as much as the metric. Scale up quickly, since slow scale-ups are what users notice. Scale down slowly, so a brief dip does not remove capacity that is needed again a minute later. The five-minute scale-down stabilization above is the HPA default, made explicit.

Queues and events: KEDA

Workers that consume from a queue should scale on the queue, not on their own CPU. A backlog of ten thousand messages is the clearest possible signal that more workers are needed, and CPU will not show it until every existing worker is busy.

KEDA adds scalers for SQS, Kafka, RabbitMQ, Pub/Sub, Prometheus queries, cron schedules and dozens of other sources. It creates and manages an HPA for you, and it adds one thing the HPA cannot do: scale to zero and back.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: invoice-worker
  namespace: billing
spec:
  scaleTargetRef:
    name: invoice-worker
  minReplicaCount: 0
  maxReplicaCount: 50
  cooldownPeriod: 300
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: invoice-worker-aws
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/123456789012/invoices
        queueLength: "20"
        awsRegion: eu-west-1
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: invoice-worker-aws
  namespace: billing
spec:
  podIdentity:
    provider: aws

Here queueLength is the target number of messages per replica. With 400 visible messages, KEDA aims for 20 workers. Use workload identity, as the TriggerAuthentication above does, rather than static keys.

Two cautions. Do not define your own HPA for a workload that has a ScaledObject, because the two will fight over the replica count. And scale-to-zero means the first message after a quiet period waits for a pod to start, and possibly a node, so it suits batch and background work better than anything latency-sensitive.

Node autoscaling: Karpenter

When the scheduler cannot place new pods, they stay Pending. A node autoscaler watches for those and adds capacity. The Cluster Autoscaler does this by growing predefined node groups. Karpenter, which reached v1 in 2024, instead looks at the pending pods' requirements and launches instances sized to fit them directly, choosing from a wide range of instance types.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
  limits:
    cpu: "400"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m

The limits field is a hard cap on what the pool can launch, which protects against a runaway scale-up turning into a runaway bill. Consolidation removes or replaces underused nodes, which is where much of the cost saving comes from, and also the source of most surprises.

Where the links interact

Most autoscaling incidents happen at the boundaries:

• Requests drive everything. The HPA measures utilization against requests, and Karpenter sizes nodes from requests. Inaccurate requests make both decisions wrong. Fix requests before tuning autoscalers.
• Scale-up latency adds up. Metric scrape interval, HPA sync period, scheduling, node launch, image pull and readiness probes all happen in sequence. A pod that needs a new node can take a couple of minutes to serve traffic. For spiky load, keep headroom with a higher minReplicas or a low-priority placeholder deployment that gets preempted when real pods need the space.
• Consolidation moves pods. Karpenter drains nodes to consolidate. Without a PodDisruptionBudget, it may evict every replica of a service at once. Set PDBs on every workload with more than one replica, and use the karpenter.sh/do-not-disrupt annotation for pods that must not be interrupted, such as long batch jobs.
• HPA and VPA on the same metric conflict. If the Vertical Pod Autoscaler changes CPU requests while the HPA scales on CPU utilization, each undoes the other. Use VPA in recommendation mode, or scale the HPA on a custom metric instead.
• Flapping comes from short windows. If replica counts oscillate, lengthen the scale-down stabilization window before changing targets.
• Spot interruptions look like scale events. Allowing several instance types and families gives Karpenter more options when one type is reclaimed.

Where to start

1. Set CPU and memory requests from observed usage for every workload that will autoscale.
2. Put HPAs on stateless services, using CPU only where CPU tracks load and per-pod request rate elsewhere.
3. Move queue consumers to KEDA scaling on backlog, and remove any HPAs they replace.
4. Run Karpenter with broad instance requirements, a CPU limit per NodePool and consolidation enabled.
5. Add PodDisruptionBudgets everywhere before enabling consolidation in production.
6. Load test the full chain and measure the time from a traffic increase to new pods serving it.