Container image hygiene for production

Most container images in production were built from a Dockerfile written once, early, and copied ever since. They ship a full operating system, a compiler, a package manager and a shell. They run as root. They are tagged latest or with a tag that gets overwritten on every build. Nobody knows exactly which base image is inside, and the scanner reports hundreds of vulnerabilities, most of them in packages the application never uses.

None of that is hard to fix. The practices below are cheap individually, and together they make images smaller, faster to pull, easier to audit and much less interesting to an attacker.

Multi-stage builds

A multi-stage build compiles in one image and copies only the result into another. Build tools, source code and intermediate files never reach the final image.

# syntax=docker/dockerfile:1
FROM golang:1.22-bookworm AS build
WORKDIR /src
 
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
 
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/orders ./cmd/orders
 
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/orders /orders
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/orders"]

Copying the dependency manifests before the source keeps the download layer cached until dependencies actually change. BuildKit cache mounts keep the module and build caches between builds without putting them in any layer. A .dockerignore that excludes .git, local environment files and build output keeps them out of the build context entirely.

The same pattern works for every language. For Node.js, install and build in a node:20-bookworm-slim stage, prune development dependencies, and copy node_modules and the build output into the runtime stage. For Java, build the JAR with the JDK and run it on a JRE-only image.

Minimal and distroless bases

The runtime image should contain the application and what it needs to run, nothing else. The choices, roughly from largest to smallest:

Google's distroless images and Chainguard's images are the common minimal options. Both publish variants with a nonroot user already defined, and both rebuild frequently with patched packages.

The cost of a shell-less image is debugging. You cannot kubectl exec into a container that has no shell. Use kubectl debug with an ephemeral container instead, which attaches a separate image with tools to the running pod's namespaces. Distroless also publishes :debug tags with BusyBox for local troubleshooting, which should never be deployed.

Run as non-root

A process running as root inside a container is one kernel or runtime vulnerability away from root on the node. Set a non-root user in the image, then enforce it in the pod spec so an image without one cannot slip through:

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

readOnlyRootFilesystem usually needs an emptyDir volume mounted at /tmp or wherever the application writes scratch files. Listening on port 8080 instead of 80 avoids the need for NET_BIND_SERVICE. The Pod Security Standards restricted profile, enforced by namespace label, requires most of these settings.

Pin base images by digest

FROM golang:1.22-bookworm resolves to whatever that tag points at when the build runs. Two builds of the same commit a week apart can contain different base layers. Pinning by digest makes the build reproducible:

FROM gcr.io/distroless/static-debian12:nonroot@sha256:6a3500eba1e6c8d1a0b5b2b2f7c8f6e1b9c2d4e8a3f7b6c5d9e0a1b2c3d4e5f6

The tag stays for humans to read. The digest is what Docker uses. Pinning without a way to update just freezes old vulnerabilities in place, so pair it with Renovate or Dependabot. Renovate's pinDigests option pins digests automatically and opens pull requests when the tag moves to a new image.

Immutable tags, deploy by digest

Your own images need the same discipline. Tag each build with the git commit SHA, never reuse a tag, and turn on tag immutability in the registry so it cannot happen by accident:

aws ecr put-image-tag-mutability \
  --repository-name checkout/orders \
  --image-tag-mutability IMMUTABLE

Google Artifact Registry and Azure Container Registry have equivalent settings. With immutable tags, orders:3f9c2a1 means the same image forever, and rollbacks are a matter of pointing back at a previous tag.

For the strongest guarantee, deploy by digest. CI knows the digest after the push, and writing it into the manifest means the cluster runs exactly the image that was scanned and tested.

Scanning

Scan in CI and fail the build on what you can act on:

trivy image \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --exit-code 1 \
  123456789012.dkr.ecr.eu-west-1.amazonaws.com/checkout/orders:3f9c2a1

--ignore-unfixed keeps the build focused on vulnerabilities that have a patch available. Grype and Docker Scout fill the same role. Keep exceptions in a reviewed .trivyignore file with a reason and an expiry, not in pipeline flags.

Scanning at build time is not enough on its own. New vulnerabilities are published against images that are already running. Registry scanning, such as Amazon ECR enhanced scanning with Inspector, or a scheduled scan of running images, catches those. Generate an SBOM with Syft or Trivy at build time so you can answer "are we affected" without rescanning everything.

Enforce at admission

Hygiene that is optional decays. An admission policy with Kyverno or Gatekeeper can require images from approved registries, reject the latest tag, require a non-root security context and, with images signed by cosign in CI, verify signatures before a pod starts.

Where to start

1. Convert the most deployed service to a multi-stage build on a distroless or slim base, and compare image size and scan results.
2. Set a non-root user in every image, and enforce runAsNonRoot with the Pod Security restricted profile in one namespace.
3. Turn on tag immutability in the registry and tag builds by commit SHA.
4. Pin base images by digest and enable Renovate to keep them current.
5. Add Trivy or Grype to CI, failing on fixable high and critical findings, and turn on registry rescanning.
6. Add admission policies once most workloads comply, starting in audit mode.