Disaster Recovery Without Double Costs with Karmada

Most multi-cluster HA setups run active/active, which means paying for full workloads on both clusters around the clock. For our production workloads, I wanted automatic disaster recovery without the standby cluster burning money every day. So I built an active/passive multi-cluster setup with Karmada 👇

Architecture Overview

Karmada runs on a lightweight K3s host and manages both member clusters. Deployments, Services, Ingresses, ConfigMaps, and Secrets are propagated using a single PropagationPolicy.

Using ClusterAffinities, I defined an ordered failover chain. The secondary cluster remains on standby until the primary actually becomes unavailable.

# Example propagation policy structure
propagation_policy:
  strategy: "single"
  # All resources propagate through this policy

Automated Failover

When the primary cluster becomes NotReady, a ClusterTaintPolicy applies a NoExecute taint. Karmada then moves the workloads to the backup cluster automatically - without manual intervention.

This provides seamless disaster recovery while eliminating the cost of continuously running full production capacity on both clusters during normal operations.

Fail-Back Mechanism

A key design decision was to handle the reverse scenario: what happens after recovery? Karmada does not automatically move workloads back to the primary cluster after recovery. Once workloads have failed over, they can remain on the secondary cluster and continue consuming resources even after the primary has recovered.

To solve this, I built a small CronJob-based fail-back mechanism:

• Waits for the primary cluster to remain healthy for 15 minutes
• Clears the scheduler affinity pin
• Triggers a reschedule
• Moves workloads back to the primary cluster

This allows the secondary cluster to return to its standby role automatically.

# Fail-back script logic (simplified)
while true; do
    if karmada status | grep -q "ready"; then
        karmada reschedule --backup
        break
    fi
    sleep 60
done

Traffic Management

Initially, I used a self-hosted HAProxy setup for traffic management. Later, I migrated this responsibility to Cloudflare Load Balancer. The current configuration includes:

• Primary and backup pools
• HTTPS /healthz health monitoring
• Automatic traffic failover
• Edge-level traffic management

This eliminates the need to maintain another HAProxy layer just for external traffic failover.

Full Setup Documentation

For workloads that don't require active/active capacity, an active/passive architecture with automated failover and fail-back can provide disaster recovery while avoiding the cost of continuously running full production capacity on both clusters. The complete setup is documented step by step (installation, cluster join, failover hardening, the fail-back script, and both load balancer options):

🔗 Karmada Kubeadm Cluster Management

---

Author: Nushad Hasan