Moving secrets out of env files and CI variables

Secrets tend to spread. A database password starts in a .env file on one laptop, gets pasted into the CI system's variable settings so the pipeline can template it into a manifest, then appears in a Helm values file for staging and a Kubernetes Secret applied by hand for production. Each copy has different access rules, none of them is audited, and rotating the password means finding all of them.

The goal is one source of truth per secret, held in a store built for it, with workloads fetching what they need at runtime. Nothing secret lives in the repository, the pipeline or a developer's shell history.

Pick one store

The store matters less than having only one. The usual candidates:

• AWS Secrets Manager: managed, integrated with IAM and CloudTrail, built-in rotation for RDS, Aurora, Redshift and DocumentDB credentials.
• HashiCorp Vault: cloud-neutral, fine-grained policies, dynamic secrets that are generated per consumer and expire on their own. More to operate unless you use HCP Vault.
• Google Secret Manager and Azure Key Vault: the equivalents on those clouds.

AWS Systems Manager Parameter Store also holds secure strings and is cheaper, but it lacks native rotation. For anything that should rotate, prefer Secrets Manager.

Name secrets by environment and owner so that access policies can use prefixes: prod/checkout/orders-db, staging/payments/stripe-api-key. A flat namespace makes narrow policies impossible later.

External Secrets Operator

External Secrets Operator (ESO) is a Kubernetes controller that reads from the external store and writes ordinary Kubernetes Secrets. Applications keep consuming Secrets as environment variables or files. Only the source changes.

Two resources do the work. A SecretStore (namespaced) or ClusterSecretStore (cluster-wide) describes how to reach the store and authenticate. An ExternalSecret says which values to fetch and what Secret to produce.

On EKS, authenticate with IAM Roles for Service Accounts, so the operator never holds a static AWS key:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws
  namespace: checkout
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: checkout-secrets

The checkout-secrets service account carries the eks.amazonaws.com/role-arn annotation for a role that can read only checkout's secrets. Then each workload declares what it needs:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: checkout
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: aws
  target:
    name: orders-db
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/checkout/orders-db
        property: username
    - secretKey: password
      remoteRef:
        key: prod/checkout/orders-db
        property: password

This file contains no secret, so it lives in the repository next to the Deployment and goes through the same review and GitOps flow. Use dataFrom with extract to pull every key of a JSON secret at once, and target.template when the application expects a composed value such as a connection URL.

Vault as the backend

The same ExternalSecret works against Vault. Only the store changes. Vault's Kubernetes auth method lets the operator log in with a service account token:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault
  namespace: checkout
spec:
  provider:
    vault:
      server: https://vault.internal:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: checkout
          serviceAccountRef:
            name: checkout-secrets

The Vault role checkout binds that service account and namespace to a policy granting read on secret/data/prod/checkout/* and nothing else.

Access boundaries

The store's access policy is where most of the security comes from. Two patterns work:

Prefer per-namespace stores for production. The IAM policy behind the checkout role is then short enough to review at a glance:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
    "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/checkout/*"
  }]
}

Two more boundaries are easy to miss:

• Who can create SecretStore resources. If any team can create one pointing at any role, the per-namespace model collapses. Restrict it with RBAC, or have the platform generate stores.
• Who can read Kubernetes Secrets. ESO writes plain Secrets. Enable encryption at rest for Secrets in the cluster (KMS envelope encryption on EKS), and do not grant get or list on Secrets to broad groups.

If even a synced Kubernetes Secret is too much exposure, the Secrets Store CSI Driver mounts values directly into pods as files without creating a Secret object, at the cost of less convenient consumption.

Rotation

Central storage makes rotation possible. It does not make it automatic.

1. Rotate in the store. Secrets Manager rotation runs a Lambda function on a schedule. AWS provides templates for RDS credentials, including the alternating-users strategy that keeps the old credential valid while the new one propagates. Vault database secrets engines go further and issue short-lived credentials per consumer.
2. Sync to the cluster. ESO checks every refreshInterval and updates the Secret when the value changes. Pick an interval shorter than the overlap your rotation strategy allows.
3. Get the application to notice. Environment variables are read once at process start, so a pod will keep the old value until it restarts. Either mount the Secret as a file and have the application reread it, or use a controller such as Stakater Reloader to roll the Deployment when its Secret changes.

Test rotation in staging before turning it on in production. The first rotation is when you find the service that cached a connection string forever.

The CI side

Pipelines are the other place secrets accumulate. Once runtime secrets live in the store, most CI variables become unnecessary: the pipeline deploys an ExternalSecret manifest, not the value. For the cloud credentials the pipeline itself needs, use OIDC federation from the CI system to the cloud provider instead of a stored access key. What remains in CI settings should be a short list you can name.

Where to start

1. Inventory secrets in .env files, CI variables, Helm values and hand-applied Secrets. Note who uses each.
2. Choose one store and a naming scheme with environment and owner prefixes.
3. Install External Secrets Operator and create one SecretStore per namespace with a narrowly scoped role.
4. Migrate one service end to end, then delete the old copies, not just stop using them.
5. Turn on rotation for database credentials, with Reloader or file mounts so pods pick up changes.
6. Add secret scanning to repositories and CI logs, so new copies are caught early.