Two Autoscaling Policies, Zero Coordination: Why More Signals Made Scaling Worse

I had a fleet of stateful worker instances behind an Auto Scaling Group On-Demand floor for guaranteed baseline capacity, Spot instances above that for burst. Standard setup. Then we added a second scaling signal to fix a real gap in the first one, and the fleet started flipping size every 30-90 seconds. This is the story of why that happened, and why it wasn't a tuning problem. The setup Each instance in the fleet holds a fixed pool of "slots" a bounded number of concurrent stateful sessions it can serve. Autoscaling started with the obvious signal: CPU. A TargetTrackingScaling policy on average CPU across the ASG, scaling out when the fleet got busy. It worked, until it didn't. CPU is a fleet-wide average. An individual instance could be completely full zero free slots, rejecting new sessions while the fleet's average CPU still looked comfortable, because three other instances were idle. CPU told you the fleet was fine. The thing that actually mattered was any single instance out of room was invisible to it. So we added a second signal: a CloudWatch Alarm watching per-instance free-slot count, tied to a Step Scaling policy, firing when any instance hit zero free slots for a sustained window. This is a completely reasonable instinct CPU is a lagging, indirect proxy for capacity; free-slot count is the real thing. What went wrong Now there were two independent scaling authorities, reacting to two different metrics, with no way to reconcile a disagreement. CPU's policy would look at the fleet-wide average, see it drop after a burst, and scale in. The pool-alarm's policy would look at the resulting per-instance squeeze, see zero free slots, and scale right back out. Neither policy knew the other existed. Neither was wrong on its own terms. Together, they fought. Real observed behavior, pulled from describe-scaling-activities : the fleet size flipping between 5 and 6 instances every 30 to 90 seconds, repeatedly, for minutes at a stretch. First attempt: tune the timing. Lengthened the pool alarm's sustained-breach window from 3 minutes to 10, on the theory the two policies just needed more separation. Deployed. The real activity log afterward showed it was still oscillating just on a 5-6 minute cadence instead of 30-90 seconds. The timing change reduced frequency. It didn't touch the actual disagreement. Two authorities still existed, still had different opinions, still had no way to agree. The actual fix: one authority, not two The right move wasn't tuning either policy. It was removing one of them. The metric that actually reflects real capacity is occupancy slots in use divided by slots available, aggregated properly across the fleet, not two separate numbers each policy interprets differently. CloudWatch Metric Math makes this computable: occupied_per_instance = pool_max_size - avg(free_slots) , targeting a real utilization band (in our case, 75%). Two things worth knowing if you're building this yourself, because both cost real debugging time: SEARCH() is not supported inside CloudWatch Metric Alarms. Only inside Dashboards and direct GetMetricData calls. First attempt at a true fleet-wide aggregate (summing free slots across every instance, dynamically, without hardcoding instance IDs) used SEARCH inside a plain Alarm's metric query. PutMetricAlarm rejected it outright not a syntax error, a hard platform limitation. Confirmed against AWS's own documentation, not assumed. Target Tracking policies support Metric Math as a separate, genuinely different feature from Alarms live since December 2022, and not subject to the same SEARCH restriction in the same way, because Target Tracking evaluates the metric expression directly rather than routing through the Alarms evaluation engine. This is the detail that made the fix possible: replace both old policies the CPU target-tracking policy and the pool-based step-scaling alarm with a single TargetTrackingScaling policy computing occupancy via Metric Math. One authority. One metric that actually reflects real capacity. Nothing left to disagree with. Verified before applying terraform plan showed exactly the intended diff, one resource added, three removed, nothing else touched. Applied cleanly on the first real attempt. Confirmed afterward via describe-policies : exactly one scaling policy exists. No oscillation since. The generalizable part This wasn't really an AWS-specific bug. It's a distributed-systems pattern that shows up anywhere two control loops react to different observations of the same underlying system state, without a shared source of truth: - Each loop is locally correct it's doing exactly what its own metric tells it to. - Neither loop is aware the other exists, let alone that it just undid the other's decision. - The oscillation isn't a bug in either policy. It's an emergent property of running two of them. The fix generalizes too: don't tune the disagreement, find the metric that actually represents the thing you care about, and have exactly one thing making the decision based on it. If you're tempted to add a second scaling signal because the first one has a real, valid blind spot that blind spot is real, but the fix is usually a better single metric, not a second policy running in parallel. AWS building blocks used: EC2 Auto Scaling Group (On-Demand + Spot mix), Target Tracking Scaling Policies, CloudWatch Metric Math, CloudWatch Alarms, Step Scaling Policies, SNS (for the alert path that first surfaced the oscillation as noisy paging). Top comments (0)