How the Amazon ECS deployment circuit breaker detects failures

Amazon ECS (Elastic Container Service) uses a deployment circuit breaker to automatically roll back failed deployments. This feature monitors the health of your service during a deployment and triggers a rollback if critical failures occur. Here’s a detailed breakdown of how it detects failures.

Prerequisites for Circuit Breaker

Before the circuit breaker can detect failures, you must configure the following:

Health Checks

• Container Health Checks: Each container in your task definition can define health check commands (e.g., HTTP pings, shell commands). ECS runs these checks periodically. If a container fails its health check continuously for a configured period, it’s marked unhealthy.
• Load Balancer Health Checks (if used): If your service uses an Application Load Balancer (ALB) or Network Load Balancer (NLB), configure target group health checks. The load balancer checks if tasks pass health checks (e.g., HTTP 200 OK on a specific endpoint). Unhealthy targets are deregistered from the load balancer.

Deployment Configuration

When creating/updating a service, you specify a deployment configuration with:

• maximumPercent: Max number of tasks that can be unhealthy during deployment (e.g., 200% = can temporarily double tasks).
• minimumHealthyPercent: Min number of healthy tasks that must remain active (e.g., 50% = at least half must stay healthy).
• Circuit Breaker Enabled: You must explicitly enable the circuit breaker in the deployment configuration. When enabled, ECS monitors failures and rolls back automatically.

How Failure Detection Works

Step 1: Deployment Starts

When you deploy a new version of your service:

• ECS incrementally updates tasks with the new task definition.
• New tasks are launched alongside existing ones (within maximumPercent limits).

Step 2: Health Monitoring

ECS continuously checks the health of all tasks (new and old) using:

• Container health checks (from the task definition).
• Load balancer target group health (if applicable).
• ECS service health (e.g., whether tasks are running/stopped).

Step 3: Failure Thresholds

The circuit breaker triggers a rollback if any of the following conditions are met:

A. Task Health Check Failures

A newly deployed task fails its container health check continuously (e.g., 3 consecutive failures within the timeout window).
Result: The task is marked unhealthy, and ECS stops monitoring it.

B. Load Balancer Health Check Failures

If using a load balancer:

• New tasks fail target group health checks (e.g., HTTP 500 errors, timeouts).
• The load balancer deregisters these tasks, causing them to receive no traffic.
• If the percentage of unhealthy tasks exceeds the service’s capacity to maintain minimumHealthyPercent, the circuit breaker trips.

C. Service Stability Violations

During deployment, if the number of healthy tasks drops below minimumHealthyPercent of the desired count for too long, the circuit breaker activates.

Example: Desired tasks = 4, minimumHealthyPercent = 50% → At least 2 tasks must stay healthy. If deployment causes only 1 healthy task, the circuit breaker triggers.

D. Deployment Stalls

If the deployment cannot proceed (e.g., no new tasks pass health checks for the entire deployment duration), ECS marks it as failed.

Rollback Process

When the circuit breaker detects a failure:

• Deployment is paused.
• ECS terminates all new unhealthy tasks.
• Remaining old (healthy) tasks continue serving traffic.
• The deployment status changes to FAILED in the ECS console/AWS CLI.
• You can manually start a new deployment (e.g., fix the issue and redeploy).

Key Metrics & Monitoring

Use CloudWatch to track:

• ECS/ContainerInsights metrics: HealthyTasks, RunningTasks, FailedTasks.
• Load balancer metrics: UnHealthyHostCount, HTTPCode_Target_5XX_Count.
• CloudWatch Alarms: Set alarms to notify you before the circuit breaker triggers (e.g., alert when unhealthy tasks exceed 30%).

Best Practices to Avoid False Positives

• Tune Health Check Parameters: Set appropriate timeout, interval, and retries in container health checks. For ALB, adjust health check path, timeout, and healthy/unhealthy thresholds.
• Set Realistic Deployment Configuration: Example: maximumPercent=200%, minimumHealthyPercent=50% allows rolling updates with zero downtime. Avoid overly aggressive settings (e.g., minimumHealthyPercent=100% prevents parallel deployments).
• Test in Staging: Validate new task definitions in a non-production environment.
• Use Blue/Green Deployments: For critical services, consider AWS CodeDeploy or ECS rolling updates with canary for gradual traffic shifting.

Summary

The ECS deployment circuit breaker detects failures by monitoring task health (via container checks or load balancers) and ensuring service stability during deployments. If new tasks fail health checks, overload the service, or prevent the minimum healthy task threshold, it automatically rolls back the deployment. Proper configuration of health checks, deployment settings, and monitoring is essential for effective failure detection.