The Argo CD Ownership Conflict — A kube-green Case Study

Argo CD is, for most people running Kubernetes today, just what GitOps means in practice: point an Application at Git, let it reconcile, and trust that whatever's live matches whatever's committed. The trouble starts the moment something else in the cluster also writes to a resource Argo CD manages, on its own terms, independent of Git. This article works through that problem using kube-green as the concrete example, which is a tool that scales workloads to zero on a schedule that you define. But kube-green is just one instance of the pattern, not the cause of it: the root problem, and most of the fixes below, apply to any controller that writes to a field Argo CD also thinks it owns.

I hit this the first time I ran kube-green next to Argo CD: it dutifully scaled a Deployment to zero at 20:00, and by 20:00:05 Argo CD had scaled it right back up. No error anywhere. It just quietly didn't work.

This post uses kube-green as a concrete case study, but the pattern applies just as directly if you're running sleepcycles, snorlax, or something else entirely.

Why this happens: self-heal

The good thing first: it's not a bug in either project. Any second controller that changes a field Argo CD renders from Git will run into this — and it comes down to one specific piece of Argo CD's behavior. An Application's sync policy has two relevant flags: automated, which syncs without waiting for someone to click "Sync", and automated.selfHeal, which goes further and reverts any drift the moment Argo CD notices it — not just on the next scheduled sync. automated alone would leave a scheduler's change alone until the next sync runs; selfHeal is what turns it into something Argo CD actively fights.

And "the moment Argo CD notices" is fast. Its application controller watches the resources it manages via the Kubernetes API rather than only polling on the default 3-minute --app-resync interval, so it reacts to change events almost immediately. Once a drift is detected with selfHeal on, the correction fires after a short debounce (--self-heal-timeout-seconds, 5 seconds by default) — which is exactly why kube-green's scale-down at 20:00 was gone by 20:00:05.

Self-heal is what makes GitOps declarative in the strict sense: without it, Git is documentation. With it, Git is enforced — for every field, all the time, regardless of who else might have a legitimate reason to touch it.

kube-green, our case study

kube-green is a small Kubernetes operator with one job: turn things off on a schedule. You install it, then create a SleepInfo custom resource per namespace:

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: office-hours
  namespace: shop-staging
spec:
  weekdays: "1-5"
  sleepAt: "20:00"
  wakeUpAt: "07:00"
  timeZone: "Europe/Berlin"
  suspendDeployments: true
  suspendCronJobs: true

At sleepAt, the controller sets spec.replicas to 0 on every Deployment/StatefulSet in the namespace (and spec.suspend: true on CronJobs). At wakeUpAt, it puts them back. To know what "back" means, it stashes the pre-sleep state in a Secret named after the SleepInfo, in the same namespace.

Two implementation details matter a lot for what follows — worth knowing straight from the source, not just the docs:

• Every mutation goes through Server-Side Apply, with kube-green registered as its own field manager (kube-green-controller-manager), and it claims ownership of only the fields it actually touched (a "sparse" apply object, not the whole resource).
• SleepInfo is namespace-scoped, and it only ever lists and patches resources inside its own namespace — there's no cluster-wide mode.

The SSA field-manager identity in particular is what one of the fixes below leans on directly.

Why it doesn't just work

Argo CD and kube-green both write to the cluster, but they answer to two different sources of truth: Argo CD answers to Git, kube-green answers to a clock. Neither knows the other exists, and there's no flag that introduces them. Swap in sleepcycles, snorlax, or something else entirely and the sequence is identical — only the name of the controller doing step 1 changes.

Sequence of events, assuming a fully default setup with automated.selfHeal=true:

1. sleepAt fires. kube-green patches spec.replicas on the Deployment from, say, 3 to 0 via Server-Side Apply.
2. Argo CD's controller sees the live object change almost immediately (it's watching it, not just polling).
3. The live state (replicas: 0) no longer matches the manifest rendered from Git (replicas: 3). The Application flips to OutOfSync.
4. Because selfHeal is on, Argo CD re-applies the Git manifest within a few seconds. replicas goes back to 3.
5. kube-green's SleepInfo still believes it's asleep — its own state says so — but the cluster is awake again.

Everything below works by breaking this loop at a different point: suppressing Argo CD's reaction, narrowing what it reacts to, telling it not to look during the window, or — in the last case — removing the diff before Argo CD ever computes it. None of them make the two controllers actually agree on who owns the cluster; they just scope the disagreement.

The approaches

1. Disable self-heal entirely

The bluntest option:

spec:
  syncPolicy:
    automated:
      selfHeal: false

Pros:

• One line, no new components
• Immediately effective
• Easy to explain to anyone reading the manifest

Cons:

• You lose drift correction for the entire Application, not just replica counts, for as long as the flag is off
• If left off permanently, any unrelated manual change to any resource this Application manages will stick until someone runs a manual sync
• Fine for a throwaway dev app; a real loss anywhere you actually rely on GitOps hygiene

2. ignoreDifferences on spec.replicas

Tell Argo CD to stop comparing that one field, full stop:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Pros:

• Simple
• Scoped to exactly the field in conflict
• Self-heal keeps working for everything else

Cons:

• Argo CD now ignores replicas unconditionally, forever — including the times you do want to change it in Git
• Bump replicas: 3 to replicas: 5 in your manifest and the sync will not touch it
• You've traded "kube-green fights Argo CD" for "Argo CD can no longer manage this field at all"

3. ignoreDifferences scoped to kube-green's own writes

A tighter version of the same idea, using Server-Side Apply field-manager attribution instead of ignoring the field outright:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - kube-green-controller-manager
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

Pros:

• Because kube-green applies via SSA and only claims the replicas field, Argo CD can tell "kube-green changed this" apart from "someone changed this some other way," including a legitimate Git-driven change
• HPA-managed replicas and manual kubectl scale still get corrected normally — only kube-green's own writes are left alone
• Of everything here, this is the one with the fewest standing side effects for the least effort

Cons:

• Requires the Application to sync with ServerSideApply=true so Argo CD's own writes are attributed to an identifiable field manager too
• Implicitly depends on kube-green's SSA field-ownership discipline as an implementation detail — it holds today, but it's not a documented contract between the two projects
• This trick only works if the non-Argo CD Controller applies via Server-Side Apply under a stable, identifiable field manager name

4. Sync windows aligned to the sleep schedule

Configure an AppProject to deny syncs for the duration kube-green is supposed to be asleep:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
spec:
  syncWindows:
    - kind: deny
      schedule: "59 19 * * 1-5"
      duration: 11h2m
      timeZone: Europe/Berlin
      applications:
        - shop-staging
      manualSync: false

Pros:

• No per-Application, per-field carve-outs to maintain
• Conceptually clean — Argo CD's automation is simply not allowed to run during the window, full stop, rather than being taught a field-level exception
• The same window mechanism doubles as a general maintenance-window tool if you need one anyway
• Tool-agnostic: works identically no matter which controller does the changes

Cons:

• SleepInfo's schedule is a pair of fixed clock times (sleepAt/wakeUpAt); a sync window is a start time plus a duration — you end up hand-computing the duration and re-deriving it every time either schedule changes, across a potential midnight or DST boundary
• The Application sits there as OutOfSync for the entire window — cosmetically alarming on a dashboard
• Blocks manual syncs of anything else in that Application during the window unless you reach for selective sync

5. Make the desired state itself schedule-aware

Every approach above works by telling Argo CD to look away from a specific field or a specific automation for a while. The Application really is out of sync with Git during the window; the reaction is just suppressed. The root-cause fix is to make the desired state Argo CD computes from Git already match what your scheduler wants, at every point in time — so there's never a real diff to suppress in the first place.

Argo CD supports Config Management Plugins (CMPs): sidecars on argocd-repo-server that sit in the manifest-rendering pipeline alongside Helm and Kustomize. A CMP can wrap your normal helm template/kustomize build output and post-process it before Argo CD ever computes a diff. Read whatever schedule your scheduler of choice uses — kube-green's weekdays/sleepAt/wakeUpAt/timeZone fields in our case study, straight from the live SleepInfo object so there's exactly one schedule definition anywhere — and, if "now" falls inside the sleep window, rewrite spec.replicas to 0 in the rendered manifest before handing it back:

#!/usr/bin/env bash
# generate.sh — CMP entrypoint
set -euo pipefail

helm template "$ARGOCD_APP_NAME" . -f values.yaml > /tmp/rendered.yaml

if schedule-check --sleepinfo "shop-staging/office-hours" --now; then
  yq eval '(select(.kind == "Deployment") | .spec.replicas) = 0' /tmp/rendered.yaml
else
  cat /tmp/rendered.yaml
fi

(schedule-check is a stand-in for whatever reads your scheduler's cron-equivalent fields and evaluates them against the current time in its timezone — the scheduling logic kube-green, sleepcycles, snorlax, and similar tools already have, just invoked by a different program than the one that normally acts on it.)

Because the "desired state" is now a function of time as well as of the Git commit, replicas: 0 at 20:00 is Git truth, not drift from it.

Pros:

• Self-heal stays fully on, everywhere, with zero standing exceptions
• If anyone manually scales the Deployment up at 22:00, Argo CD correctly puts it back to 0, because that genuinely is the desired state right now
• Tool-agnostic in principle — the CMP only needs to read a schedule, it doesn't care which project owns the CRD or annotation it's reading from

Cons:

• A non-obvious gotcha that would silently break this whole approach: Argo CD caches rendered manifests per Git revision, on the assumption that the same commit always renders to the same output — by default for a full day. Since the commit hasn't changed at 20:00, Argo CD will happily keep serving the cached, pre-sleep manifest unless something forces a re-render
• The fix is to pair the CMP with a trigger at exactly the two moments that matter — hitting Argo CD's hard-refresh API at sleepAt and wakeUpAt, via a plain CronJob using the same two cron expressions the SleepInfo already encodes
• Bigger blast radius for a bug: a broken CMP affects manifest rendering for every Application using it, not one field on one resource kind
• Only reuses the scheduler's CRD/config as a schedule format — the actual scaling now happens in the CMP, so the scheduler's own controller becomes redundant for that workload (for kube-green specifically, you'd set suspendDeployments: false and let it handle CronJob suspension only, if you keep it in the loop at all)
• Requires every relevant Application to go through the CMP source type, a bigger lift for an existing repo of plain Helm/Kustomize Applications than adding an ignoreDifferences block

Which one to actually use

For most people, approach 3 is the pragmatic default — a few lines on the Application, doesn't touch the scheduler at all, and keeps Git-driven replica changes working, provided your tool applies via Server-Side Apply with a stable field manager (kube-green confirmed; verify for the others). Approach 4 earns its keep if you want sync windows for other maintenance reasons anyway, and it's the one fix here that works identically regardless of which scheduler you run. Approach 5 is where you end up once standing exceptions have themselves become the problem: it's the only fix here with zero exceptions, ever, works no matter which scheduler is doing the scaling, and doesn't care whether your Applications are bootstrapped by hand or fully GitOps-managed — you're just trading all of that for the cost of owning a CMP.

None of these make Argo CD and whichever scheduler you've bolted on agree about who owns the cluster. They just make the disagreement politely scoped, instead of a nightly fight that quietly cancels itself out — whether that fight is with kube-green, sleepcycles, snorlax, or something else entirely, and whether the field in question is replicas today or something else entirely tomorrow.

Further reading: Akuity's write-up on Argo CD and kube-green covers approaches 2 and 4 in more depth, and the kube-green GitHub issue on GitOps support is where a lot of this thinking has accumulated over time.