Too Long, Didn't Apply: Server-Side Apply for Oversized CRDs in Argo CD ApplicationSets

When dealing with big resources in Argo CD - especially with Custom Resource Definitions (CRDs) - you will regularly see an error that these resources cannot be synchronized. The error message looks something like this:

error when patching "/dev/shm/1487475670": CustomResourceDefinition.apiextensions.k8s.io "httproutes.gateway.networking.k8s.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes

The root cause is the default synchronization behaviour of Argo CD. It uses client-side apply, which writes the full serialized manifest into the kubectl.kubernetes.io/last-applied-configuration annotation. The problem with this approach is the fact that annotations may not exceed a specific size in Kubernetes - 262144 bytes to be precise.

The fix for this problem is rather easy: use server-side apply (SSA) for the affected Application. SSA records ownership in metadata.managedFields, so nothing ever lands in annotations and the limit doesn't apply. To change the synchronization behaviour you only have to adjust the syncOptions of your Application like this:

spec:
  syncPolicy:
    syncOptions:
      - Validate=true
      # The magic fix
      - ServerSideApply=true

This settings uses ServerSideApply for every resource that is part of this Application. If you want to adjust the synchronisation behaviour only for the resources that are actually too big you can also an annotation, like this:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true

If you only need to adjust a single resource of Application the required changes are quite easy to implement, but what about an ApplicationSet? You usually don't want to activate SSA for every Application that has been created by an ApplicationSet, which would happen if you add the syncOption to it's specification. In such a situation, you can use an Argo CD feature that is still not very widely known: templatePatch.

TemplatePatch to the rescue

With TemplatePatch you can define more complex logic like loops or conditions that will be added as a patch on your template. In this scenario we can use it to only add the ServerSideApply flag on the desired Application. In the example I am assuming a Git Generator and want to activate ServerSideApply when the last folder equals gateway-api-crds:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  ...
spec:
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  generators:
    ...
  template:
    ...
  templatePatch: |
    {{`{{ if eq .path.basename "gateway-api-crds" }}`}}
    spec:
      syncPolicy:
        syncOptions:
          - Validate=true
          - ServerSideApply=true
    {{`{{ end }}`}}

In general you can patch everything you want with a TemplatePatch, except the spec.project field. This feature also requires that you enable goTemplate for the ApplicationSet. The patch is inert without it.