The Helm Chart Is a Platform Contract - Not a Template

The Helm Chart Is a Platform Contract - Not a Template

Early in building our cloud infrastructure, we had a problem nobody talks about - because it happens so slowly you almost don't notice it.

We had eight separate Helm charts. One for services that needed KEDA scaling. One for standard HPA. One for backends that exposed HTTP. One for workers that didn't. One for Azure Functions. One for frontends. Eight charts, all living in the same repository, all drifting apart from each other.

The charts started as copies of each other. Over time each one picked up its own fixes, its own conventions, its own slightly-different take on security contexts and ServiceAccount annotations and rolling update strategy. Nobody made a decision to diverge. It just happened.

Every time we fixed something in one chart - say, wiring up Azure Workload Identity to every ServiceAccount - we had to remember to propagate that fix to seven others. Sometimes we did. Sometimes we didn't. We'd find out when something broke in an unexpected way six weeks later.

Helm chart drift is more dangerous than dependency drift. At least with a dependency, you know what version you're on. With eight loosely related charts, you just don't know what you don't know.

This is the story of how we replaced all eight with a single versioned chart, published to an OCI registry, and consumed by 70+ services through ArgoCD multi-source Applications - and what that structure forced us to think clearly about.

The Two-Questions Framework

The first thing we had to do was figure out why we had eight charts in the first place. What was actually different between services that justified a different chart?

We landed on two questions:

• Does it expose HTTP? - This determines whether it needs an ingress, a Service, liveness/readiness probes on an HTTP path.
• What drives its scaling? - Standard CPU/memory HPA, or event-driven scaling via KEDA (Azure Service Bus, Event Hubs)?

That's it. Everything else - security contexts, Workload Identity, pod anti-affinity, rolling update strategy, how secrets are mounted - is the same for every service. There was no reason for it to differ. It only differed because nobody had said it shouldn't.

From those two questions, we get three chart archetypes:

platform-backend is the one that carries most of the complexity. The other two are simpler. Everything below focuses on it.

What the Chart Enforces (Non-Negotiables)

The key design decision was separating things the platform cares about from things service teams care about. Platform concerns belong in the chart. They are not options. Service teams do not get to turn them off.

Azure Workload Identity on every ServiceAccount

Every ServiceAccount the chart creates gets the Workload Identity annotations baked in:

# values.yaml (chart defaults)
serviceAccount:
  create: false
  annotations:
    azure.workload.identity/client-id: "<client_id>"
    azure.workload.identity/tenant-id: "<tenant-id>"
  labels:
    azure.workload.identity/use: "true"

The client-id is per-service (it's in the values file the team manages). The label that enables the OIDC token injection is not - it's always there.

Pod anti-affinity across nodes by default

# values.yaml (chart defaults)
podAntiAffinity:
  enabled: true
  topologyKey: "kubernetes.io/hostname"

Services spread across nodes by default. Teams can disable this for workers that don't need it, but they have to make an active choice.

Rolling update strategy

rollingUpdate:
  maxUnavailable: ""
  maxSurge: ""

The chart enforces a rolling update strategy. Recreate is not on the table.

DB migrations as a PreSync hook

If a service has database migrations, those run before the deployment rolls out - not after, not alongside, not "we'll figure it out manually":

argoHooks:
  dbMigration:
    enabled: false
    command:
      - "/app/migrate.sh"

When enabled: true, the chart creates an ArgoCD PreSync Job that runs the migration command using the same container image as the service. The deployment only proceeds once the Job succeeds.

The ArgoCD Multi-Source Pattern: Decoupling Chart from Config

The thing that makes this scale is how we deploy it. We publish the chart to GitHub Container Registry as an OCI artifact. The chart is versioned. A service pins the version it uses. Config values live in a separate Git repository, always at HEAD.

A single ArgoCD Application looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: service-maintenance-stage
  namespace: argocd
spec:
  project: staging
  sources:
    # Source 1: versioned chart from OCI registry
    - repoURL: ghcr.io/platform-team
      chart: platform-backend
      targetRevision: 1.2.0
      helm:
        releaseName: service-maintenance-stage
        valueFiles:
          - $values/applications/staging/service-maintenance/values-STAGE.yaml
        parameters:
          - name: nameOverride
            value: service-maintenance
          - name: fullnameOverride
            value: service-maintenance
    # Source 2: config repo at HEAD (ref alias used above)
    - repoURL: https://github.com/platform-team/aks-platform-config.git
      targetRevision: HEAD
      ref: values
  destination:
    name: staging
    namespace: service-maintenance
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

The $values alias wires the two sources together. The chart comes from the OCI registry at a pinned version. The values file comes from the config repo at whatever HEAD is right now.

This split is the whole game. It means:

• Config changes (env vars, replica counts, probe paths, secret names) land the moment they're merged to the config repo. No chart release needed.
• Chart upgrades happen when a team is ready, not when we force them.
• If we introduce a breaking change in the chart, teams on older versions are unaffected until they choose to migrate.

One Template, Two Scaling Strategies

This is one of the more interesting pieces of the chart. We have a single template file - hpa-keda.yaml - that handles both KEDA event-driven scaling and native Kubernetes HPA. Which one renders depends on a single flag:

# hpa-keda.yaml (simplified)
{{- if .Values.keda.enabled }}
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: {{ include "app.fullname" . }}
spec:
  podIdentity:
    provider: azure-workload
    # no connection strings - uses Workload Identity
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ include "app.fullname" . }}
  annotations:
    scaledobject.keda.sh/transfer-hpa-ownership: "true"
spec:
  scaleTargetRef:
    kind: Deployment
    name: {{ include "app.fullname" . }}
  minReplicaCount: {{ .Values.keda.minReplicaCount }}
  maxReplicaCount: {{ .Values.keda.maxReplicaCount }}
  triggers:
    {{- if .Values.keda.trigger.servicebus.enabled }}
    - type: azure-servicebus
      metadata:
        {{- if .Values.keda.trigger.servicebus.topicName }}
        topicName: {{ .Values.keda.trigger.servicebus.topicName }}
        subscriptionName: {{ .Values.keda.trigger.servicebus.subscriptionName }}
        {{- else }}
        queueName: {{ .Values.keda.trigger.servicebus.queueName }}
        {{- end }}
        namespace: {{ .Values.keda.trigger.servicebus.namespace }}
        messageCount: {{ .Values.keda.trigger.servicebus.messageCount | quote }}
      authenticationRef:
        name: {{ include "app.fullname" . }}
    {{- end }}
    {{- range .Values.keda.trigger.eventhubs }}
    - type: azure-eventhub
      metadata:
        eventHubNamespace: {{ .eventHubNamespace }}
        eventHubName: {{ .eventHubName }}
        storageAccountName: {{ .storageAccountName }}
        blobContainer: {{ .blobContainer }}
        consumerGroup: {{ .consumerGroup | default "$Default" }}
        unprocessedEventThreshold: {{ .unprocessedEventThreshold | default "64" | quote }}
      authenticationRef:
        name: {{ include "app.fullname" $ }}
    {{- end }}
{{- else if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "app.fullname" . }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "app.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}

A few things worth noting:

• No connection strings in KEDA. The TriggerAuthentication uses provider: azure-workload - the KEDA operator authenticates to Azure Service Bus and Event Hubs using the pod's federated identity token. Zero secrets stored anywhere.
• The scaledobject.keda.sh/transfer-hpa-ownership: "true" annotation is important: when KEDA creates its own HPA internally, this tells it to take ownership of any pre-existing HPA rather than conflicting with it.
• KEDA and HPA are mutually exclusive at the template level. You can't accidentally have both. The if/else if structure enforces it.
• KEDA can optionally add CPU/memory triggers alongside queue triggers. A worker might scale primarily on Service Bus message count but also scale up on CPU if the queue is empty but pods are hot.

DB Migrations as a First-Class Deployment Concern

This is the piece that most teams implement ad-hoc and later regret. The ArgoCD PreSync hook runs a Kubernetes Job before the deployment rollout begins. It uses the same container image as the service (so migrations are always paired with the code that needs them), inherits the same env vars and secret mounts, and runs with the same service account and Workload Identity:

# argo-presync-hook.yaml
{{- if .Values.argoHooks.dbMigration.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ $fullName }}-db-migration
  annotations:
    argocd.argoproj.io/hook: "PreSync"
    argocd.argoproj.io/hook-delete-policy: "HookSucceeded,BeforeHookCreation"
    argocd.argoproj.io/sync-wave: "10"
spec:
  ttlSecondsAfterFinished: 900
  template:
    spec:
      serviceAccountName: {{ include "app.serviceAccountName" . }}
      containers:
        - name: db-migrations
          image: "{{ $image }}:{{ $imageTag }}"
          command:
            {{- range .Values.argoHooks.dbMigration.command }}
            - {{ . | quote }}
            {{- end }}
          env:
            {{- range .Values.deployment.environment }}
            - name: {{ .name }}
              value: {{ .value | quote }}
            {{- end }}
      restartPolicy: Never
      backoffLimit: 0
{{- end }}

The HookSucceeded,BeforeHookCreation delete policy means: clean up the Job after it succeeds, and if a new sync starts before the old Job is cleaned up, replace it rather than block. ttlSecondsAfterFinished: 900 is the backstop - if ArgoCD misses the cleanup, the Job self-destructs after 15 minutes.

If migrations fail, the deployment does not proceed. This is not the default Kubernetes behaviour, and it's not something most teams set up correctly on their own.

To enable it for a service:

# service values file
argoHooks:
  dbMigration:
    enabled: true
    command:
      - "/app/migrate.sh"

The Breaking Change That Wasn't

In v1.0.0 of the chart, we changed the default ingress class from nginx to kong. This was a BREAKING CHANGE - it's called out in the changelog in all caps.

Here's what actually happened when we released it: Nothing, for most services. Because they were still on v0.x.x.

Services migrated to v1.0.0 when their team was ready, tested in staging first, and bumped the targetRevision in their ArgoCD Application. They saw the ingress change, updated their Kong-specific annotations, and moved on. The whole process took maybe 30 minutes per service.

If we had been using a shared chart without versioning - something like a Helm repository that all services track at latest - that breaking change would have been a coordination nightmare. Every team would have needed to change their configuration at the same time. Someone would have missed it. An incident would have followed.

Semantic versioning works. OCI publishing enforces it. The multi-source ArgoCD pattern makes it painless.

What a Service Values File Actually Looks Like

After all of this, what does a team actually write? Here's a real production values file (names changed):

replicaCount: 1
rollme: "v1.5.0"
nameOverride: "service-conditionplatform-worker"
fullnameOverride: "service-conditionplatform-worker"
image:
  tag: "v1.5.0"
containers:
  - name: service-conditionplatform-worker
    image: "registry.azurecr.io/service-conditionplatform-worker"
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        drop:
          - ALL
      runAsNonRoot: true
      runAsUser: 10001
      runAsGroup: 2000
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
    resources:
      limits:
        cpu: 500m
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 150Mi
    volumeMounts:
      - name: tmp
        mountPath: /tmp
volumes:
  - name: tmp
    emptyDir: {}
serviceAccount:
  create: false
  name: "service-conditionplatform"
deployment:
  environment:
    - name: ASPNETCORE_ENVIRONMENT
      value: Prod-Cloud
    - name: APP_LOG_LEVEL
      value: Warning
service:
  enabled: false
ingress:
  enabled: false
autoscaling:
  enabled: false
podAntiAffinity:
  enabled: true
  topologyKey: "kubernetes.io/hostname"

That's it. The team wrote about 40 lines of YAML. They own replica count, resource limits, env vars, probe config, and security context. They don't think about Workload Identity setup, rolling update strategy, or migration hooks - those are already handled.

Onboarding a new service is: copy an existing values file, change the name, image, and serviceAccount fields. Five minutes.

We have 119 production values files structured this way.

The Publishing Pipeline

The chart CI is a reusable GitHub Actions workflow gated by a publish boolean:

# reusable-publish-helm-chart.yaml (simplified)
jobs:
  upload_and_lint:
    steps:
      - uses: azure/setup-helm@v4
      - name: Helm lint
        run: helm lint .
      - name: Helm template
        run: helm template .
      - name: Package
        run: helm package . --destination .
      - name: Upload artifact
        uses: actions/upload-artifact@v6
  publish:
    needs: upload_and_lint
    if: ${{ inputs.publish == true }}
    steps:
      - uses: docker/login-action@v4