Structuring GitOps with Argo CD: app-of-apps or ApplicationSets

Argo CD is easy to start with. One Application points at one path in one repository and syncs it to one cluster. The trouble begins at the twentieth service and the third cluster, when the repository holds hundreds of nearly identical Application manifests and nobody is sure which ones are still deployed.

The structure you choose early decides how painful that growth is. There are two established patterns, and they solve slightly different problems.

App-of-apps

In the app-of-apps pattern, one parent Application points at a directory that contains other Application manifests. Argo CD syncs the parent, which creates the children, which sync the workloads.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/platform-config.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Everything under apps/ is an ordinary Application, written out in full. That is both the strength and the weakness. Each child can differ in any way: its own sync policy, ignore rules, project or destination. Reviewers see exactly what will exist. But adding a service means copying a file, and adding a cluster means copying every file again.

App-of-apps remains a good fit for a small, deliberately heterogeneous set, such as the cluster add-ons a platform team installs once per cluster: ingress controller, cert-manager, external-dns, the monitoring stack. Each has its own quirks, and there are only a dozen of them.

ApplicationSets

The ApplicationSet controller, bundled with Argo CD since 2.3, generates Applications from a template and a list of parameters. The parameters come from generators. Two of them cover most needs.

The git generator

The git directory generator creates one Application per matching directory in a repository. Adding a service becomes adding a folder.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/acme/platform-config.git
        revision: main
        directories:
          - path: services/*
  template:
    metadata:
      name: "{{.path.basename}}"
    spec:
      project: services
      source:
        repoURL: https://github.com/acme/platform-config.git
        targetRevision: main
        path: "{{.path.path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.path.basename}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

The git files generator works the same way but reads parameters from JSON or YAML files, which is useful when each service needs a few values the directory name cannot express, such as an owning team or a target namespace.

The cluster generator

The cluster generator produces one set of parameters per cluster registered in Argo CD. Clusters are Secrets labeled argocd.argoproj.io/secret-type: cluster, and you can add your own labels to them. A selector then decides which clusters receive what:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters:
        selector:
          matchLabels:
            env: production
  template:
    metadata:
      name: "monitoring-{{.name}}"
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/platform-config.git
        targetRevision: main
        path: "addons/monitoring/overlays/{{index .metadata.labels \"env\"}}"
      destination:
        server: "{{.server}}"
        namespace: monitoring

Registering a new production cluster with the right label is now enough to get the monitoring stack on it. Combining the two with the matrix generator gives every service on every matching cluster, which is the shape most multi-cluster platforms end up with.

Choosing between them

Many teams use both: a small app-of-apps at the root that installs the ApplicationSets, and ApplicationSets that generate the rest. The root stays readable, and the repetition moves into templates.

Ordering with sync waves

Within one Application, Argo CD applies resources in waves. The annotation argocd.argoproj.io/sync-wave takes an integer, lower numbers go first, and the default is zero. Argo CD waits for each wave to be healthy before starting the next.

apiVersion: batch/v1
kind: Job
metadata:
  name: orders-migrate
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.internal/checkout/orders:1.4.2
          args: ["migrate", "up"]

Typical uses are custom resource definitions before the resources that use them, a namespace and its secrets before the workload, and database migrations before the new Deployment rolls out.

Waves order resources within an Application. They do not order Applications generated by an ApplicationSet relative to each other. If cluster add-ons must exist before services, either put the dependency inside one Application, or rely on retries and health checks so that a service that syncs too early simply converges on the next attempt. ApplicationSet progressive syncs exist, but they are still an alpha feature and need to be enabled explicitly.

Separate configuration from source code

Keep deployment configuration in its own repository rather than in each service's source repository. The reasons are practical:

• CI commits do not trigger application builds. When the pipeline bumps an image tag, it writes to the config repository, not to the service repository it just built.
• Access differs. Many people can merge application code. Fewer should be able to change what runs in production.
• History is readable. The config repository's log is the deployment log: who changed what, in which environment, and when.
• Argo CD watches less. Fewer, smaller repositories mean faster refreshes and less load on the repo server.

A layout that works well:

bootstrap/          root app-of-apps
appsets/            ApplicationSets
addons/             cluster add-ons, one folder each
  monitoring/
    base/
    overlays/production/
services/           one folder per service
  orders/
  payments/

Use Argo CD Projects to enforce the boundaries: which repositories each project may read, which clusters and namespaces it may deploy to, and which cluster-scoped resources it may create. A services project that cannot create ClusterRoles is a cheap guardrail.

Where to start

1. Move deployment manifests into a dedicated config repository if they are not there already.
2. Write one root Application that points at an appsets/ directory. Keep it as the only Application you create by hand.
3. Convert services to a git directory generator. Start with preserveResourcesOnDeletion: true while you migrate.
4. Label cluster secrets with environment and region, and drive add-ons from a cluster generator.
5. Add sync waves only where order matters: CRDs, migrations, secrets.
6. Lock down Projects so each ApplicationSet can only deploy where it should.