"Your GitOps Hub Will Become the Bottleneck Long Before Cluster Count Tells You"

Your GitOps Hub Will Become the Bottleneck Long Before Cluster Count Tells You GitOps hub bottlenecks are usually predicted more accurately by watched object volume, reconcile queue depth, and controller memory growth than by cluster count alone. Large-scale testing described in the discussion showed Argo CD application controllers hitting out-of-memory failures around 15,000 to 20,000 cached objects per hub, while sharding and tuning delayed the limit without removing the underlying memory cost. The most important lesson was uncomfortable because it challenged the usual instinct to keep tuning the existing platform. At very large scale, architecture mattered more than configuration. Hydrated manifests helped. More replicas helped. Dynamic sharding helped. None of them changed the fact that a centralized reconciliation model still had to hold and process a huge amount of state. The testing was not presented as a universal benchmark or as proof that one tool always beats another. It was a record of one setup, built through dozens of iterations over several months, and the failures were as valuable as the successful runs. That is exactly why the results matter. They show where teams should look before the hub becomes the thing taking the fleet down. Cluster count is the wrong first metric A fleet with 1,000 tiny clusters may place less pressure on a GitOps control plane than a much smaller fleet containing thousands of applications and deeply expanded resource trees. The number of managed clusters is visible and easy to report, but it does not describe the controller’s actual workload. The more useful mental model is objects over clusters. Each application contributes desired state, live state, cached trees, reconciliation work, and queue activity. A cluster that runs a few small addons may be cheap to manage. Another cluster with many applications and large manifest sets may consume far more memory and reconciliation time. That means two fleets with the same cluster count can behave very differently. One commenter described watched objects and application controller queue depth as the indicators that predicted the wall in their environment. That matched the reported failure range of roughly 15,000 to 20,000 cached objects per hub. The controller was not failing because it had counted too many clusters. It was failing because one process was trying to watch and hold too much state. This distinction changes capacity planning. Teams should stop asking only how many clusters a hub can manage and start asking how many objects, applications, watches, and queued reconciliations each controller process can sustain. OOM kills are an architectural warning An out-of-memory kill can look like a resource sizing problem. Sometimes it is. At fleet scale, repeated application controller OOMs can also be a sign that the control plane is carrying a workload its reconciliation model handles inefficiently. The discussion described Argo CD consuming about 21 GB of memory in one addon-style rollout scenario, while Sveltos handled a similar pattern with roughly 2 GB. The author was careful not to frame that as a general benchmark claim. The comparison belonged to a specific setup and use case. Still, the size of the gap was difficult to ignore. The reason was not simply better tuning. The tools approached the problem differently. A centralized controller that caches and reconciles large resource trees will accumulate state as the fleet grows. A model that pushes lightweight agents into managed clusters can distribute classification, drift detection, and reconciliation work closer to the spokes. The memory profile changes because the architecture changes. That is the point teams often resist. They keep increasing limits, adding replicas, and tuning flags because replacing or combining control-plane components feels more disruptive. But when memory grows with the amount of watched state, tuning may buy time without changing the curve. A hub that repeatedly recovers from OOM kills is not healthy because it comes back. It is warning that the design is approaching a boundary. Why sharding helps and still disappoints Sharding the Argo CD application controller was one of the most obvious responses, and the testing did include it. The setup used between three and six controller replicas and tested dynamic cluster distribution. It helped. Load was spread more evenly. No single controller had to hold the entire fleet’s cache. The failure arrived later. But the total memory requirement remained. That led to one of the clearest conclusions in the discussion: sharding spreads memory usage, but it does not reduce it. If the fleet requires a certain amount of cached state, dividing that state across more processes can reduce the pressure on each process, but the aggregate cost still exists. The operational burden may also grow because teams now manage more replicas, more shards, more balancing behavior, and more failure modes. Sharding remains valuable. It can create runway, isolate failures, and improve controller distribution. It is especially useful when the current architecture is still fundamentally sound and the goal is to avoid one overloaded process. The mistake is treating sharding as proof that the scalability problem is solved. It may postpone the wall while leaving the same relationship between fleet size and control-plane memory intact. Capacity planning should therefore measure both per-shard pressure and total control-plane cost. A system that scales only by continually adding memory and controller replicas may still be moving toward an expensive and fragile design. Hydrated manifests and tuning can only go so far The testing also found that hydrated manifests improved behavior. Pre-rendering or otherwise reducing work required during reconciliation can remove repeated computation and reduce some of the load placed on controllers. That kind of optimization matters. Large fleets amplify small inefficiencies, so reducing manifest generation work, avoiding unnecessary reconciliation, tightening selectors, and improving distribution can produce meaningful gains. Yet the discussion repeatedly returned to the same limitation: tuning helped only partially. This is a familiar pattern in infrastructure engineering. Teams optimize the obvious hot path, then discover that the real cost is structural. A controller still has to know about the desired state. It still has to compare it with live state. It still has to hold or retrieve enough information to make reconciliation decisions. The exact memory profile may improve, but the relationship between managed state and controller pressure remains. Tuning should be treated as a way to validate the architecture, not as a substitute for architecture. A sensible sequence is to reduce unnecessary work first, then observe whether the resource curve becomes manageable. If memory still grows too quickly with watched objects, the next question should not be which flag has not been tried. It should be whether the reconciliation model matches the scale and workload pattern. One controller per cluster changes the tradeoff Several people in the discussion described a different design: place Argo CD or Flux CD in each managed cluster, then use central configuration, generators, or automation to deploy the correct application sets to the correct locations. This approach distributes reconciliation by default. Each cluster handles its own local state, so one central application controller does not need to cache the entire fleet. The tradeoff is operational duplication. Running a GitOps controller in every cluster means every location needs enough CPU and memory for that controller. Upgrades, configuration, credentials, and lifecycle management must be handled across the fleet. For well-resourced clusters, this may be acceptable. For small edge locations, even Argo CD Core may be too heavy. That edge constraint was central to the original setup. The fleet involved resource-limited locations, which made a fully decentralized Argo CD deployment unattractive. A centralized management model was chosen partly because it reduced spoke-side overhead. This is why there is no universal winner. A per-cluster model may scale more naturally and provide a stronger security posture because spokes can pull from central configuration without allowing the hub broad inbound access. A centralized model may simplify governance and reduce component duplication. An agent-based model can offer a middle path by placing lightweight components in the spokes while keeping a central view. The right design depends on where the organization can afford complexity: in the hub, in every cluster, or in the coordination layer between them. ApplicationSets solve targeting, not infinite scale ApplicationSets were discussed as an important fleet-management mechanism. Their generators can dynamically create Argo CD Applications based on lists, cluster secrets, Git data, matrix combinations, merges, and other inputs. In the described setup, a cluster generator used label selectors on registered cluster secrets. A label such as cert-manager: enabled could cause the generator to create an application for every matching cluster and deploy the relevant addon to one, ten, or one hundred locations. This is operationally useful because it creates manageable fleet slices. Labels define which clusters receive which applications. Progressive syncs then allow teams to roll changes out in controlled waves rather than updating the entire fleet at once. That solves targeting and rollout control. It does not create unlimited control-plane capacity. The generated applications still become objects the control plane must manage. Their resource trees still contribute to cache size and reconciliation work. ApplicationSets can help organize the fleet, but they do not remove the underlying cost of managing every generated application centrally. Teams should use generators t