Liveness, readiness and startup probes done right

Kubernetes probes look like a small detail of a Deployment manifest. They are also one of the most common ways a cluster turns a minor problem into a full outage. A database slows down, every pod's health check fails, the kubelet restarts all of them at once, and the service that was degraded is now down.

The fix is not to remove probes. It is to understand that the three probe types answer three different questions, and to make each one answer only its own.

Three probes, three questions

Only liveness and startup failures restart anything. Readiness failures are cheap and reversible: the pod stops getting new requests and starts getting them again when it recovers. That difference should drive everything else.

Liveness: check the process, nothing else

A liveness probe should fail only when restarting the container would fix the problem. That means a deadlock, an exhausted thread pool that will never drain, or an event loop that has stopped. It should not fail because something outside the process is unhealthy.

The common mistake is pointing the liveness probe at a health endpoint that checks the database, the cache and two downstream APIs. When the database has a bad minute:

1. Every replica's liveness check fails at roughly the same time.
2. The kubelet restarts all of them.
3. The new containers start, open fresh connection pools against a database that is already struggling, and add load.
4. Startup takes longer under that load, probes fail again, and pods enter CrashLoopBackOff.

The database recovers. The service does not, because it is busy restarting. Restarting a healthy process never fixes a dependency.

A good liveness endpoint is boring. It returns 200 if the HTTP server can answer the request, and it does no I/O. If the application has a known way to wedge, such as a worker loop that stops processing, the endpoint can check a heartbeat timestamp that the loop updates.

Readiness: can I serve traffic?

Readiness is the right place for dependency awareness, with care. A pod that cannot do useful work should stop receiving requests so the load balancer sends them elsewhere. Good readiness signals:

• The application has finished loading configuration, caches or models.
• The pod is shutting down. Failing readiness during graceful termination drains traffic before the process exits.
• The pod is overloaded, for instance because its request queue is full.

Be careful with shared dependencies here too. If every pod depends on the same database and the readiness probe checks it, a database blip removes every endpoint from the Service. Callers then get connection errors instead of a fast, clear 503 from the application. Often it is better for the application to stay ready and return an explicit error for requests that need the missing dependency, while still serving those that do not.

A reasonable rule: readiness checks things that are local to the pod or specific to it. Shared dependencies are handled in application code with timeouts, retries and circuit breakers.

Startup: protect slow starters

Before startup probes existed, slow-starting applications needed a long initialDelaySeconds on the liveness probe. That delay applied to every start, and it left a window where a genuinely hung container went undetected.

A startup probe runs first, and liveness and readiness probes do not run until it succeeds. It can tolerate a long start without weakening the liveness probe afterwards. The total time allowed is failureThreshold × periodSeconds.

Putting it together

containers:
  - name: orders
    image: registry.internal/checkout/orders:2.3.0
    ports:
      - name: http
        containerPort: 8080
    startupProbe:
      httpGet:
        path: /livez
        port: http
      periodSeconds: 5
      failureThreshold: 36      # up to 3 minutes to start
    livenessProbe:
      httpGet:
        path: /livez
        port: http
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: http
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2

The startup and liveness probes share the cheap /livez endpoint. Readiness uses a separate /readyz that reflects whether this pod can serve.

Choosing timing values

The defaults are periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1 and initialDelaySeconds: 0. Some guidance on adjusting them:

• Raise timeoutSeconds above 1. A one-second timeout fails under garbage collection pauses or CPU throttling, which are exactly the moments a restart would make things worse. Two to five seconds is a common range.
• Make liveness slower to fail than readiness. Readiness should react within seconds. Liveness should only trigger after the process has clearly been stuck for a while, often 30 seconds or more in total.
• Size the startup budget from real data. Look at the slowest observed start, including under load and with cold caches, and allow a comfortable margin.
• Account for CPU limits. A container throttled by a tight CPU limit answers probes slowly. Probe failures that line up with throttling metrics point to the limit, not the probe.

For gRPC services, use the native grpc probe type with the standard gRPC health checking protocol rather than shipping a separate HTTP server for probes. For non-HTTP workloads such as queue consumers, an exec probe that checks a heartbeat file works, but keep the command light, since it runs inside the container on every period.

Graceful shutdown

Probes and termination interact. When a pod is deleted, it is removed from endpoints asynchronously, so traffic can still arrive for a few seconds after SIGTERM. Handle that by:

• Failing readiness as soon as SIGTERM arrives.
• Continuing to serve in-flight and late-arriving requests for a short period, often via a preStop sleep of a few seconds.
• Keeping terminationGracePeriodSeconds longer than the drain time.

Review checklist

1. Liveness endpoints do no I/O and check no dependencies.
2. Every liveness probe has a named failure mode it protects against, or it is removed.
3. Readiness reflects pod-local state and fails during shutdown.
4. Slow starters use a startup probe instead of initialDelaySeconds on liveness.
5. timeoutSeconds is at least 2, and liveness takes longer to fail than readiness.
6. Restart counts and probe failures are on the service dashboard, so a probe storm is visible when it starts.