Runbooks as code

Most runbooks are wiki pages. They start accurate, then drift. A command gets a new flag, a cluster is renamed, a step moves to a different console, and nobody updates the page because nobody reads it until 3 a.m. At that point an engineer copies a command into a terminal, edits the namespace by hand, and hopes they changed the right one.

Runbooks as code treats the operational procedure the same way as the service: written as code, stored in version control, reviewed before it changes, tested, and run by a system that records who ran it and what happened. The goal is not to automate on-call away. It is to make the repeated, mechanical parts of an incident safe and fast, so people can spend their attention on the parts that need judgment.

Start from what is repeated

Do not start by converting every page in the runbook wiki. Start from evidence. Incident timelines, on-call handover notes and chat history show which manual actions are actually performed, and how often. A few weeks of that data usually produces a short list like this:

Pick tasks that are frequent, well understood and reversible. Failover and restore deserve automation too, but they are rare, high-stakes and hard to test, so they should come after the team trusts the system with simpler work.

Diagnostics before remediation

The safest first automation does not change anything. A diagnostic runbook collects what an engineer would look at first: recent deploys, pod restarts, error rates from the metrics system, the last lines of logs, the state of dependencies. It posts the result into the incident channel.

This has three benefits. It shortens the first ten minutes of every incident. It cannot make things worse. And it builds the plumbing (credentials, access to clusters and cloud APIs, a place to post results) that remediation runbooks will reuse.

What a good automated runbook looks like

A runbook as code is a small program with strict habits:

• Explicit, validated parameters. The service name comes from an allowed list, not free text. Replica counts have an upper bound.
• Dry run by default. The first run shows what would change. Applying requires a deliberate flag.
• Preconditions checked before acting. Is the target in the expected state? Is another run already in progress?
• Idempotent steps, so running it twice does not do harm.
• Verification after acting. The runbook checks that the change had the intended effect and says clearly if it did not.
• Output a human can read, posted where the incident is being handled.

import argparse
import subprocess
import sys
 
ALLOWED = {"orders", "payments", "checkout"}
 
def kubectl(*args: str) -> str:
    return subprocess.run(
        ["kubectl", *args], check=True, capture_output=True, text=True
    ).stdout
 
def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--service", required=True, choices=sorted(ALLOWED))
    parser.add_argument("--namespace", default="production")
    parser.add_argument("--apply", action="store_true")
    args = parser.parse_args()
 
    target = f"deployment/{args.service}"
    history = kubectl("rollout", "history", target, "-n", args.namespace)
    print(history)
 
    if not args.apply:
        print(f"Dry run: would roll back {target} to the previous revision.")
        return 0
 
    kubectl("rollout", "undo", target, "-n", args.namespace)
    kubectl("rollout", "status", target, "-n", args.namespace, "--timeout=5m")
    print(f"Rolled back {target}; rollout completed.")
    return 0
 
if __name__ == "__main__":
    sys.exit(main())

The logic is deliberately boring. The value is in the guardrails around it, and in the fact that the same reviewed code runs every time.

Running it with approvals

Where the code runs matters as much as what it does. A runbook executed from someone's laptop has no audit trail and depends on whatever credentials that laptop holds. Run it from a system that provides identity, approvals and logs.

There are several reasonable choices. Rundeck and similar job runners are built for this. AWS Systems Manager Automation runs documents with an aws:approve step and records every execution. For many teams, the CI system they already have is enough: a manually triggered workflow with typed inputs, running against a protected environment that requires approval.

on:
  workflow_dispatch:
    inputs:
      service:
        type: choice
        options: [orders, payments, checkout]
      apply:
        type: boolean
        default: false
 
jobs:
  rollback:
    runs-on: [self-hosted, ops]
    environment: production-ops
    concurrency: runbook-rollback-${{ inputs.service }}
    steps:
      - uses: actions/checkout@v4
      - run: >
          python runbooks/rollback_deployment.py
          --service ${{ inputs.service }}
          ${{ inputs.apply && '--apply' || '' }}

The production-ops environment can require a reviewer before the job starts and holds the credentials the runbook needs, so nobody has those credentials on their workstation. The concurrency group stops two people from rolling back the same service at once. Every run is logged with who started it, who approved it, the inputs and the output.

Review and test like production code

Runbooks change through pull requests, with the owning team as reviewers. That alone fixes the drift problem: the runbook changes in the same pull request that renames the deployment or changes the flag.

Testing is harder, because the failure the runbook fixes is not usually present. A few practices help:

• Run dry runs on a schedule against production, so a broken permission or renamed resource is found on a Tuesday afternoon rather than during an incident.
• Exercise remediations in staging regularly, ideally as part of game days.
• Lint and unit test the parameter validation and the decision logic, which is where most bugs live.

Link alerts to runbooks

An alert should point at the runbook that addresses it. Put the runbook link, or the workflow that runs it, in the alert's annotations. The engineer who is paged then sees what to check and what to run, rather than searching a wiki.

Over time, some runbooks become trusted enough to run automatically on the alert itself. Do that only for actions that are reversible and have a long, clean history of manual runs, and keep a way to turn it off quickly.

Where to start

1. Review recent incidents and list the manual actions that were taken, with how often each occurred.
2. Write one diagnostic runbook for your noisiest service and post its output into the incident channel.
3. Automate the most frequent reversible action, with validated inputs, dry run by default and verification after acting.
4. Run it from a system with identity, approvals and logs, never from a laptop.
5. Link each alert to its runbook, and schedule dry runs so breakage is found early.
6. Add the next most repeated task, and retire the wiki page for each one you replace.