Writing operational scripts that are safe to rerun

Every team has a folder of scripts that someone wrote during an incident or a migration. They create users, rotate certificates, copy buckets, patch configuration. Most of them work exactly once, on the machine of the person who wrote them, in the state the system was in that afternoon.

The problem shows up the second time. A script fails halfway through, someone fixes the cause and runs it again, and now it creates a duplicate resource, appends the same line to a config file twice, or deletes something it already moved. A script that is safe to rerun turns that situation into a non-event.

What idempotent means for a script

An operation is idempotent when running it once or ten times leaves the system in the same state. For a script, that means each step describes a desired end state rather than an action:

The second column is what tools like Ansible and Terraform do for you. In a shell script you have to write it yourself, step by step.

Start with strict mode

Bash keeps going after errors by default. A failed cd followed by rm -rf ./* is the classic outcome. Put this at the top of every script:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

• -e exits when a command fails.
• -u treats unset variables as errors, so a typo in $BACKUP_DIR does not silently become an empty string.
• -o pipefail makes a pipeline fail if any command in it fails, not just the last one.
• Setting IFS to newline and tab stops word splitting on spaces in filenames.

Strict mode has known gaps. Commands inside an if condition, or on the left side of || or &&, do not trigger -e. Functions called in those positions also lose it. Treat strict mode as a safety net, not a replacement for checking the result of commands that matter.

Add a trap so failures say where they happened:

trap 'echo "error: line $LINENO: $BASH_COMMAND" >&2' ERR

Check, then act

The core pattern is to test the current state before changing it. Most command-line tools offer a way to ask:

ensure_user() {
  local name="$1"
  if id "$name" >/dev/null 2>&1; then
    echo "user $name already exists"
  else
    useradd --system --create-home "$name"
    echo "created user $name"
  fi
}
 
ensure_line() {
  local line="$1" file="$2"
  grep -qxF -- "$line" "$file" || echo "$line" >>"$file"
}
 
ensure_bucket() {
  local bucket="$1"
  if aws s3api head-bucket --bucket "$bucket" 2>/dev/null; then
    echo "bucket $bucket already exists"
  else
    aws s3api create-bucket --bucket "$bucket" \
      --create-bucket-configuration LocationConstraint=eu-west-1
  fi
}

Many commands also have idempotent forms built in. Use them where they exist:

• mkdir -p succeeds if the directory exists.
• ln -sfn replaces a symlink instead of failing.
• kubectl apply converges to the manifest, where kubectl create fails on the second run.
• install -d -m 0750 -o deploy creates a directory with the right owner and mode, or fixes them.
• cp into a temporary file followed by mv replaces a file atomically, so a half-written file never exists.

Make long jobs resumable

A script that processes thousands of items should not start from zero after a failure. Two approaches work well:

• Derive progress from the system itself. If the script moves objects from one prefix to another, rerunning it moves only what is left. No bookkeeping needed.
• Record completed items in a state file, and skip them on the next run. Write the marker only after the step has succeeded.

done_file="/var/lib/migrate-orders/done"
touch "$done_file"
 
while read -r order_id; do
  grep -qxF -- "$order_id" "$done_file" && continue
  migrate_order "$order_id"
  echo "$order_id" >>"$done_file"
done <orders.txt

Prevent two copies from running at once

Idempotency assumes one run at a time. Two copies of the same script, started by cron and by a person, can both pass the check and both act. On Linux, flock solves this in one line:

exec 9>/var/lock/rotate-certs.lock
flock -n 9 || { echo "another run is in progress" >&2; exit 1; }

The lock is released automatically when the script exits, including when it crashes. For scripts that run across several machines, a local lock is not enough. Use the locking that the target system offers instead, such as a conditional write to a database or a lease in the system being changed.

Add a dry-run flag

A dry run prints what the script would change without changing it. It is the cheapest review step there is, and it forces the script to separate deciding from doing:

DRY_RUN="${DRY_RUN:-false}"
 
run() {
  if [[ "$DRY_RUN" == "true" ]]; then
    echo "would run: $*"
  else
    "$@"
  fi
}
 
run aws s3 rm "s3://orders-exports/tmp/" --recursive

Route every changing command through run, and leave read-only commands alone so the dry run still sees the real state. Make dry run the default for destructive scripts, and require an explicit --apply to change anything.

Validate inputs early

Most damaging script runs come from wrong arguments rather than bugs: the production account instead of staging, an empty variable, a path with a trailing space. Check inputs before the first change:

• Confirm the target with the system, not just the argument. For AWS, compare aws sts get-caller-identity against the expected account ID.
• Refuse to run with an empty or root path for anything that deletes.
• Print a short summary of what will be affected, and ask for confirmation when running interactively.

When to stop writing Bash

Shell is the right tool for gluing a few commands together. It is the wrong tool once the script needs to track state across many resources, handle partial failure carefully or parse structured data. Signs it is time to move on:

• The script manages long-lived infrastructure. Terraform or Pulumi already implement check-then-act, drift detection and plans. Writing a worse version in Bash is not a good use of time.
• It configures servers. Ansible modules are idempotent by design and report what changed. A script full of grep -q || lines is reimplementing them.
• It parses JSON with more than one jq call per step, or needs retries with backoff. Python or Go with the cloud provider's SDK will be shorter and easier to test.
• More than one person maintains it. At that point it needs tests and review like any other code.

Scripts that stay scripts should still live in a repository, run through CI with shellcheck, and have a short header comment saying what they change and how to run them safely.

Checklist for the next script

1. Start with set -euo pipefail and an ERR trap.
2. Write each step as "ensure this state", with a check before every change.
3. Make long loops resumable from the system's state or a progress file.
4. Take a lock with flock if the script can be started twice.
5. Add a dry-run mode, and make it the default for anything destructive.
6. Validate the target account, cluster or host before the first change.
7. Run shellcheck in CI and run the script twice in a test environment. The second run should change nothing.