Auto-remediation with EventBridge and Lambda, without the surprises

Most operations teams have a list of problems they fix the same way every time. A security group opens SSH to the internet, someone closes it. An EBS volume is created without encryption, someone flags it. A bucket loses its public access block, someone puts it back. Each fix takes minutes. The delay before anyone notices takes days.

Event-driven automation closes that gap. AWS already emits an event for almost every change in an account. Amazon EventBridge can match those events and invoke code within seconds. The hard part is not wiring the event to a function. It is making the function safe to run unattended, many times, against production.

The shape of a remediation

Every auto-remediation has the same four parts:

1. A signal. A CloudTrail API call, an AWS Config compliance change, a Security Hub finding or a CloudWatch alarm state change, delivered to the default event bus.
2. A rule that matches only the events you mean to act on.
3. A handler, usually a Lambda function, or a Step Functions state machine when the fix has several steps or needs a human to approve it.
4. A record of what was done, where the owning team will see it.

Take the security group case. When someone adds an inbound rule, CloudTrail records an AuthorizeSecurityGroupIngress call, and EventBridge receives it as an "AWS API Call via CloudTrail" event. The rule pattern matches on that:

{
  "source": ["aws.ec2"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["ec2.amazonaws.com"],
    "eventName": ["AuthorizeSecurityGroupIngress"]
  }
}

Keep patterns narrow. A rule that matches every EC2 API call and filters inside the function costs more, runs more often and hides what the automation is actually for. CloudTrail management events must be enabled for this signal to exist, which a trail in the organization usually already provides.

Act on current state, not on the event

The event says what happened at some moment in the past. By the time the handler runs, someone may have already fixed the rule, deleted the group or added three more open rules. A handler that parses the request parameters out of the event and reverses exactly those is fragile in all of these cases.

A sturdier handler treats the event as a hint. It reads the current state of the resource, decides whether it is out of policy, and corrects whatever it finds:

import os
import boto3
 
ec2 = boto3.client("ec2")
DRY_RUN = os.environ.get("DRY_RUN", "true") == "true"
 
def handler(event, context):
    group_id = event["detail"]["requestParameters"]["groupId"]
 
    groups = ec2.describe_security_groups(GroupIds=[group_id])["SecurityGroups"]
    tags = {t["Key"]: t["Value"] for t in groups[0].get("Tags", [])}
    if tags.get("remediation") == "exempt":
        return {"action": "skipped", "reason": "exempt tag", "group": group_id}
 
    rules = ec2.describe_security_group_rules(
        Filters=[{"Name": "group-id", "Values": [group_id]}]
    )["SecurityGroupRules"]
    offending = [
        r["SecurityGroupRuleId"] for r in rules
        if not r["IsEgress"] and r.get("CidrIpv4") == "0.0.0.0/0"
        and (r["IpProtocol"] == "-1"
             or r.get("FromPort", 0) <= 22 <= r.get("ToPort", 65535))
    ]
 
    if offending and not DRY_RUN:
        ec2.revoke_security_group_ingress(
            GroupId=group_id, SecurityGroupRuleIds=offending
        )
    return {"action": "revoked" if offending else "none",
            "dry_run": DRY_RUN, "group": group_id, "rules": offending}

This handler is idempotent by construction. Running it twice for the same event, or for two events about the same group, produces the same end state. That matters because EventBridge delivers events at least once, and Lambda retries asynchronous invocations that fail. Duplicates are normal, not an edge case.

When an action cannot be made state-based, such as opening a ticket or sending a notification, record that the event was handled. The idempotency utility in Powertools for AWS Lambda stores a hash of the event in DynamoDB and returns the earlier result for a repeat, which saves writing that table yourself.

Plan for failure in two places

There are two separate retry paths, and each needs its own place for failures to land:

A common mistake is to configure only one of them and assume failures are covered. In Terraform, both sit next to the rule:

resource "aws_cloudwatch_event_target" "open_ssh" {
  rule = aws_cloudwatch_event_rule.open_ssh.name
  arn  = aws_lambda_function.remediate_open_ssh.arn
 
  retry_policy {
    maximum_event_age_in_seconds = 3600
    maximum_retry_attempts       = 10
  }
  dead_letter_config {
    arn = aws_sqs_queue.remediation_dlq.arn
  }
}
 
resource "aws_lambda_function_event_invoke_config" "open_ssh" {
  function_name          = aws_lambda_function.remediate_open_ssh.function_name
  maximum_retry_attempts = 2
 
  destination_config {
    on_failure {
      destination = aws_sqs_queue.remediation_dlq.arn
    }
  }
}

The queue's resource policy must allow both events.amazonaws.com and the function's role to send messages. Put an alarm on ApproximateNumberOfMessagesVisible above zero. A dead-letter queue nobody watches is just a slower way to lose events.

Guardrails before autonomy

Automation that changes production can also break production, and it does so faster than a person would. The guardrails worth building in from the first function:

• Start in dry-run mode. Ship every new remediation with DRY_RUN=true, let it log what it would have done for a week or two, and compare that against what people actually did. Only then switch it on.
• Give each function its own narrow role. The handler above needs ec2:DescribeSecurityGroups, ec2:DescribeSecurityGroupRules and ec2:RevokeSecurityGroupIngress, nothing else.
• Cap concurrency. Reserved concurrency on the function limits how much damage a flood of events can cause. A value of 1 to 5 is plenty for most remediations.
• Support an exemption. A tag such as remediation=exempt, protected by a service control policy so only the platform team can set it, gives legitimate exceptions a path that does not involve disabling the rule.
• Watch for loops. A function that tags a resource in response to a CreateTags event will trigger itself. Match on events your own fix cannot produce, or filter out calls made by the remediation role using userIdentity in the pattern.
• Have one switch to stop everything. Disabling the EventBridge rule halts a remediation instantly. Document the command (aws events disable-rule --name open-ssh-remediation) in the runbook.

Tell the owner

A fix that happens silently teaches nobody anything. The same misconfiguration reappears next week from the same Terraform module. Every remediation should notify the team that owns the resource, using an ownership tag to route the message, and say what was found, what was changed and how to make the change properly if it was intended.

Return a structured result from the handler, as the function above does. Lambda's on-success destination can send it to an SNS topic or another event bus, which keeps notification logic out of the remediation code.

Where to start

1. Pick two or three findings your team fixes by hand every month. Repetition is the best predictor of value.
2. Confirm the signal exists: CloudTrail management events, AWS Config rules or Security Hub findings on the default bus.
3. Write each handler to read current state and converge it, with dry run on by default.
4. Configure both the target dead-letter queue and the Lambda on-failure destination, and alarm on them.
5. Add reserved concurrency, a narrow IAM role and a documented exemption tag.
6. Run in dry run, review the log against reality, then enable one remediation at a time.