Rolling out default-deny NetworkPolicies in Kubernetes

By default, every pod in a Kubernetes cluster can reach every other pod, in every namespace, on every port. A compromised frontend can talk to the database of an unrelated service, or to the metrics endpoint of a controller that was never meant to be exposed. NetworkPolicies close that gap, but only once a namespace has a default-deny policy. Until then, policies only add allow rules to a network that already allows everything.

Getting to default-deny is mostly about order. Apply the deny first and things break in ways that are hard to diagnose. Apply it last, after the allow rules are in place and verified, and the change is uneventful.

Check that your CNI enforces policies

The NetworkPolicy API is part of Kubernetes, but enforcement is the job of the network plugin. If the plugin does not support it, the API server accepts the objects and nothing happens. There is no error.

Verify enforcement before relying on it. Apply a deny policy to a scratch namespace and confirm a connection actually fails.

How the rules combine

A few semantics drive every design decision:

• Policies are additive allow lists. A pod selected by no policy allows everything. A pod selected by any policy for a direction, ingress or egress, allows only what some policy permits in that direction.
• Both ends must agree. For a connection from pod A to pod B, A's egress policies must allow it and B's ingress policies must allow it.
• Replies are allowed automatically. Policies are stateful, so you never write a rule for return traffic.
• Selectors in one from entry are ANDed; separate entries are ORed. This is the most common source of overly broad rules, covered below.

The default-deny policy

One policy per namespace, selecting every pod and allowing nothing:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

An empty podSelector selects all pods in the namespace. Listing both policyTypes with no rules means no traffic is allowed in either direction, including DNS.

Allow DNS egress first

Once egress is denied, the first thing to fail is name resolution, and almost everything depends on it. Every namespace with egress deny needs a DNS rule:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Allow TCP as well as UDP: large responses fall back to TCP, and the failures that follow are intermittent and confusing. Check the label on your DNS pods, since k8s-app: kube-dns is conventional for CoreDNS but not universal.

If you run NodeLocal DNSCache, pods send queries to a link-local address on the node rather than to CoreDNS pods. A pod selector does not match that. You need an ipBlock rule for the local cache address, and how that interacts with node-local traffic varies by CNI, so test it specifically.

Namespace selectors and the AND/OR trap

The kubernetes.io/metadata.name label is set automatically on every namespace, which makes it a reliable way to select a namespace by name. Custom labels work too, but anyone who can edit a namespace can add them, so prefer the automatic label for rules that matter.

These two rules look almost identical:

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: checkout
        podSelector:
          matchLabels:
            app: orders

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: checkout
      - podSelector:
          matchLabels:
            app: orders

The only difference is the dash before podSelector. The second is much broader than it looks. Review every from and to block for it.

Rollout order

Roll out per namespace, in this order:

1. Inventory traffic. Observe what actually talks to what before writing rules. Cilium's Hubble, Calico's flow logs and VPC flow logs all help. Without observation you are guessing.
2. Allow shared infrastructure. Ingress from the ingress controller's namespace to workloads that serve traffic, and from the monitoring namespace to metrics ports. These are the rules everyone forgets.
3. Allow service-to-service ingress for the dependencies you observed.
4. Apply default-deny for ingress only. Ingress is easier to reason about, and the failures are contained to the namespace you changed.
5. Add egress rules, starting with DNS, then dependencies in other namespaces, then external endpoints.
6. Apply default-deny for egress. This is the step most likely to break things, because egress needs are spread across libraries, SDKs and sidecars.

Some egress is easy to miss: the Kubernetes API server for operators and anything using a service account token, cloud metadata endpoints for workload identity, and external services such as payment providers or package registries. The API server is a special case, since it is usually not a pod, and policies may need an ipBlock for its endpoint addresses. Behavior differs by CNI, so test it.

Standard NetworkPolicy has no audit or dry-run mode. Cilium has a policy audit mode that logs what would be dropped without dropping it, and commercial editions of other CNIs offer similar staging. Where none is available, roll out on a staging cluster with real traffic first.

Platform-owned defaults are better delivered as part of namespace creation, through the golden path or a policy engine such as Kyverno that generates a default-deny and DNS policy for every new namespace. Then no new namespace starts open.

Testing

Policies are easy to get subtly wrong, so test them like code:

• Connectivity checks after each change. Run a short-lived pod with the same labels as the client, and try the connection (shown below).
• Negative tests too. Confirm that a pod without the right labels is refused. A policy that allows too much passes every positive test.
• Automated checks in CI. Tools such as Cyclonus generate connectivity test cases from policies, and some CNIs ship their own connectivity test suites, such as cilium connectivity test.
• Watch drops in production. Hubble, Calico flow logs or the CNI's metrics show denied flows. A sudden rise after a deploy usually means a new dependency nobody added a rule for.

kubectl run nettest --rm -it --restart=Never \
  -n checkout --labels app=orders \
  --image=nicolaka/netshoot -- \
  curl -sS -m 3 http://payments-api.payments.svc:8080/healthz

Rollout checklist

1. Confirm the CNI enforces policies with a deliberate deny test.
2. Observe real traffic for each namespace before writing rules.
3. Add allow rules for ingress controllers, monitoring and known callers.
4. Apply ingress default-deny one namespace at a time.
5. Add DNS and dependency egress rules, then egress default-deny.
6. Generate default policies for new namespaces automatically.
7. Add positive and negative connectivity tests, and alert on unexpected drops.