Letting AI agents act on production safely

An AI agent that can only read dashboards and suggest commands is useful. One that can restart a deployment, scale a node group or roll back a release is far more useful, and far more dangerous. The model can misread a situation, follow instructions planted in a log line or ticket, or simply repeat an action in a loop. Every one of those failures turns into a production change.

The answer is not to trust the model more. It is to design the system around it so that a wrong decision is limited, visible and reversible. Most of the controls are familiar from giving access to people and to automation. They just need to be applied deliberately, because an agent will use every permission it has.

The agent is a principal like any other

Give each agent its own identity. Not a shared admin role, and not the credentials of the engineer who started it. A dedicated service account or IAM role means its permissions can be scoped, its actions can be attributed in audit logs, and its access can be revoked without affecting anyone else.

Then scope permissions per tool, not per agent. Whether the agent's tools are exposed through the Model Context Protocol or through function calling in your own code, each tool runs with credentials that cover exactly what it does. A tool that reads pod status needs get and list on pods. A tool that restarts a deployment needs patch on deployments in specific namespaces, nothing more.

On Kubernetes, the read side is plain RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ops-agent-read
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "pods/log", "deployments", "replicasets", "jobs", "events"]
    verbs: ["get", "list", "watch"]

Leave secrets out of the read role. Logs and secrets are exactly what an attacker would try to pull out through a prompt injection, and the agent rarely needs secret values to diagnose a problem.

Separate reads from writes

Classify every tool by what it can do:

Many useful agents never leave the first row. An agent that gathers evidence during an incident and proposes a fix for a person to run is a large step forward on its own, and it carries very little risk.

When an agent does get write tools, make them narrow and specific. restart_deployment(namespace, name) is far safer than kubectl(args). A generic shell or API passthrough turns every permission check into a guess about what the model might type.

Dry run before every write

Every write tool should support a dry-run mode that reports what would change, and the agent should be required to call it first. Most infrastructure already offers the primitive:

• kubectl apply --dry-run=server and kubectl diff show the result after admission control.
• terraform plan -out=tfplan produces a plan that can be reviewed and then applied exactly.
• Many AWS EC2 APIs accept a DryRun parameter that checks permissions without acting.

The dry-run output becomes the thing a person approves. That is important: the approver should see the concrete change, not the model's description of it.

Human approval, done well

Approval only helps if the approver can make an informed decision quickly. A useful approval request contains:

• The exact action and target, for instance a scale of checkout/orders-api from 4 to 8 replicas.
• The dry-run output or diff.
• The agent's reasoning and the evidence it used, with links to the dashboards or logs.
• How to undo it.

Approvals belong in a channel people already watch, such as Slack or the incident tool, and they should expire. An approval granted for one action must not cover the next one, even if it looks identical.

The gate must sit in code the model cannot influence. Implement it in the tool layer, not in the system prompt:

WRITE_TOOLS = {"scale_deployment", "restart_deployment", "toggle_flag"}
 
def call_tool(name: str, args: dict, session: AgentSession):
    if kill_switch.is_engaged():
        raise ToolRefused("agent actions are paused")
    if not rate_limiter.allow(session.agent_id, name):
        raise ToolRefused(f"rate limit reached for {name}")
 
    audit.record(session, name, args, phase="requested")
 
    if name in WRITE_TOOLS:
        preview = TOOLS[name].dry_run(**args)
        decision = approvals.request(session, name, args, preview, timeout_s=900)
        audit.record(session, name, args, phase="approval", decision=decision)
        if not decision.approved:
            raise ToolRefused("change was not approved")
 
    result = TOOLS[name].run(**args)
    audit.record(session, name, args, phase="executed", result=result)
    return result

Instructions such as "always ask before making changes" in the prompt are helpful, but they are not a control. A prompt injection or an unexpected chain of reasoning can bypass them. The gate above cannot be talked out of its checks.

Over time, some reversible actions can graduate to policy-based approval: restarting a stateless deployment in staging, say, or scaling within a range already allowed by the autoscaler. Decide that from the audit history, not from a sense that the agent is usually right.

Audit everything

Record every tool call with the agent identity, the session, the person or event that started the session, the arguments, the approval decision and the result. Keep the model's input and output for the session too, since that is how you will understand a strange decision later.

Write the audit log somewhere the agent cannot modify, and ship it to the same place as other security logs. Cloud audit trails such as CloudTrail and Kubernetes audit logs will also show the actions, attributed to the agent's own identity, which is one more reason not to share credentials.

Rate limits and a kill switch

Agents can loop. A model that misreads the result of a restart may decide to restart again, and again. Limit write actions per agent, per tool and per target over a time window, and make the limits low. Three restarts of the same deployment in ten minutes is almost certainly a mistake, whoever is doing it.

A kill switch must stop all agent actions at once, without a deploy. A feature flag checked at the start of every tool call works well. So does revoking the agent's role binding or detaching its IAM policy, which also stops anything that bypasses the tool layer. Write the steps into the incident runbook and test them, the same way you would test any other emergency procedure.

Where to start

1. Give each agent its own identity with read-only access, and deploy it to help during incidents.
2. Build narrow write tools for the two or three actions people repeat most, each with a dry-run mode.
3. Put approval, rate limiting and the kill switch in the tool layer, outside the model's control.
4. Log every call, approval and result to an append-only store.
5. Review the audit history regularly, and only relax approval for actions with a clean record.