Shipyard: How We Built Slack’s Next-Generation EC2 Platform

Shipyard: How We Built Slack’s Next-Generation EC2 Platform

Over the past few years, we’ve been on a journey to modernise how we run Amazon Elastic Compute Cloud (EC2) instances at Slack.

In our first post, Advancing Our Chef Infrastructure, we shared how we moved from a single Chef stack to a resilient, multi-stack setup with versioned cookbook deployments and safer promotion workflows. This afforded us far more reliability and operational control across tens of thousands of EC2 instances.

Subsequently, in Advancing Our Chef Infrastructure: Safety Without Disruption, we tackled deployment risk without the need for teams to rewrite their cookbooks. By introducing split production environments, signal-based Chef runs, and smarter rollout mechanisms, we dramatically reduced the impact radius of failures while keeping our legacy platform stable.

These changes allowed us to safely operate our EC2 ecosystem at scale while we plan the future at a relaxed pace. But as we kept improving, a bigger truth became clear. Even with safer rollouts, better orchestration, and stronger guardrails, the old model-continuously updating long-lived EC2 instances-was hitting its limits. Service-level deployments were tricky, infrastructure drift was inevitable, and coordinating changes across multiple layers added complexity.

Containers solved this for some classes of workloads, but not everything could migrate easily. We needed a new approach-one that brought modern deployment practices like immutability, progressive rollouts, and automated safety directly to EC2 instances.

Enter Shipyard.

Shipyard is Slack’s next-generation EC2 platform. It treats infrastructure as deployable artifacts rather than endlessly mutable instances. It gives teams service-level deployment primitives, tight integration with our build and orchestration systems, and the confidence to update infrastructure with the same safety and predictability we expect from modern app delivery platforms.

In this post-the third chapter of our journey-we’ll explain why we built Shipyard, the principles behind its design, and how it represents a fundamental shift in how we think about running EC2 instances at Slack.

What Shipyard Provides

Shipyard is designed to bring modern infrastructure principles to EC2 instances while preserving the flexibility teams expect from EC2. Rather than focusing on configuration management as the center of the system, Shipyard shifts responsibility toward build pipelines, deployable artifacts, and automated safety mechanisms.

Some of the key capabilities of the platform include:

Multi-Architecture and Multi-OS Support

The platform is designed from the start to support multiple CPU architectures, including both AMD64 and ARM-based Graviton instances, with support for multiple operating systems such as Ubuntu, RHEL, and Amazon Linux. This flexibility allows teams to optimize for cost, performance, or compatibility without needing separate platform implementations.

Shipyard is particularly valuable for workloads that cannot migrate to containers, such as infrastructure components, Kubernetes worker nodes, and our egress network stacks.

Metrics-Driven Deployments with Safety Controls

Each service integrates with our deployment orchestration system called Gondola to enable progressive rollouts with metric-based automated safety checks. Deployments can automatically halt based on service health signals, or automated rollback to previous known good versions.

Fast and Predictable Provisioning

Shipyard uses a layered image approach, inspired by how containers work. A shared “golden” base image provides common infrastructure components, and service-specific images are built on top of that foundation. This minimizes work at launch time, so instances can come online quickly and predictably across regions.

Simplified Configuration Management

One of the biggest architectural shifts is how we use configuration management. Previously, instances would run scheduled Chef jobs that periodically checked and reapplied configuration, so any manual or unexpected changes would be reverted back to the desired state.

In the new model, configuration is applied during well-defined lifecycle phases like image baking and initial provisioning, rather than being continuously enforced in the background. Configuration tools are mainly used to deploy services, not to constantly modify the entire system. This reduces background load, avoids unintended overwrites, and makes system behavior much easier to reason about, since instances aren’t continuously changing over time.

Real-Time Inventory and Fleet Visibility

Shipyard comes with a new inventory system called Peekaboo, giving us near real-time visibility into the state of our EC2 fleet. Instead of relying on Chef Server as the source of truth, Peekaboo taps directly into cloud events and instance metadata, providing better telemetry across environments. It can even track instances from non-Shipyard deployments, giving us a complete view of the entire fleet in one place.

We built Peekaboo using AWS EventBridge, OpenSearch, and Lambda. It has everything teams need: a UI to explore the fleet, an API for integrations, and a command-line interface (CLI) for quick command-line checks. By centralising this information, it removes the guesswork and gives us a single place to view and manage EC2 instances.

Short-Lived, Continuously Refreshed Instances

To keep our EC2 instances secure and truly immutable, each instance has a limited lifespan and is automatically rotated on a regular schedule. This means our fleets are always fresh; potential vulnerabilities have less time to cause issues, and teams focus on replacing instances rather than making in-place changes.

Golden Base Images: Introducing slack-zero

At the foundation of Shipyard is a shared base image called slack-zero. This is the core machine image built by Slack’s Compute Platform Team and maintained collaboratively with our security and monitoring teams.

The slack-zero image contains:

• Operating system baseline and hardening
• Networking and service discovery configuration
• Monitoring and security agents
• Common tooling and foundational system configuration

You can think of slack-zero similarly to how teams use a base Docker image or how we start from a vendor-supplied Ubuntu image and layer additional components on top. It provides a standardised, trusted foundation that every service inherits, while still allowing teams to customize their own runtime environment on top of it.

Base images are treated as immutable but ephemeral. When foundational components need to change-such as a security patch, monitoring update, or networking improvement-a new slack-zero image is produced. Downstream service images can then rebuild on top of the updated base to inherit the latest fixes and improvements.

To build slack-zero, we use AWS Image Builder rather than Packer. Image Builder provides several built-in advantages over our previous approach with Packer, including:

• Lifecycle Management: Old AMIs are automatically cleaned up using lifecycle policies, helping reduce storage costs.
• AWS System Manager (SSM) Parameter Publishing: Each new slack-zero image updates an SSM parameter that indicates the latest available AMI for an account. Service pipelines read this parameter to ensure they always build on the most up-to-date base.
• Event-Driven Automation: When a slack-zero image finishes baking successfully, EventBridge and Lambda automatically trigger downstream pipelines in service owner accounts so dependent images can be rebuilt.
• Built-in Testing: Before an AMI is published, Image Builder launches temporary instances and runs validation tests. This ensures every image is verified before distribution, reducing risk and increasing confidence in production rollouts.

Together, these capabilities allow us to continuously evolve the platform foundation while keeping adoption friction low for service teams.

Service Images

Each service team builds its own AMIs using slack-zero as the foundation. This allows teams to control their runtime environment while inheriting standardized platform components maintained by the platform organization.

Service image pipelines define:

• What software is installed
• How the service is configured
• What happens during instance initialisation for this service

Because most configuration is baked directly into the image, instances launch quickly and consistently, minimizing configuration drift and ensuring predictable behavior across the fleet.

By combining the immutable slack-zero base with service-specific layers, Shipyard provides both platform stability and team-level flexibility, enabling services to innovate safely without sacrificing operational consistency.

Baking and Provisioning

Shipyard separates instance preparation into two distinct phases: baking and provisioning.

During the bake phase, we install packages and include configuration that is consistent across environments. This ensures every instance starts from a fully prepared, known-good state with the majority of work already completed before launch.

Environment-specific settings-such as secrets, regional configuration, or deployment metadata-are applied during the provisioning phase when the instance boots. This step is intentionally lightweight and typically involves only dropping configuration, retrieving secrets, and starting services.

By moving heavy operations like package installation into the bake phase, instances can become operational in seconds rather than minutes. This fast startup time is critical for scaling events, rolling deployments, and automated instance replacement.

This provisioning model provides a strong balance of consistency, speed, and flexibility: images deliver a stable baseline, while minimal provisioning adapts instances to their runtime environment without introducing drift.

Deployments and Fleet Updates

When teams need to roll out changes, they build a new Amazon Machine Instance (AMI) and then run their deployment pipeline to roll it out. Instead of patching existing instances, fleets are updated through controlled replacements, keeping everything consistent and predictable.

For Auto Scaling Groups (ASGs), we use AWS Instance Refresh, and Kubernetes worker fleets use Karpenter for lifecycle-driven updates. Services with special deployment needs can use alternative rollout executors. Our global deployment orchestrator, Gondola, supports these patterns, giving teams flexibility while keeping a consistent deployment experience.

Emergency Fixes and Rapid Deployment Pathways

For urgent situations, the platform allows targeted configuration changes on running instances, but these are meant for emergencies only. Affected instances are expected to be replaced afterward via regular deployment pipelines.

Our emergency workflows use AWS Systems Manager with a predefined document to run selected Chef recipes, letting teams quickly apply critical fixes. Once stable, instances are cycled to return to the intended immutable state.

What Do Customer Pipelines Look Like?

Customer pipelines in Shipyard can have multiple stages, giving teams flexibility to design them around their service and operational needs. Each stage in Gondola represents a deployable unit, such as an ASG, a Kubernetes cluster, or a group of EC2 instances.

For example, the Egress Team runs separate canary and production ASGs in each availability zone, with deployment stages ordered so updates flow sequentially. Gondola updates each stage, monitors key metrics, and rolls back automatically if problems are detected, preventing issues from spreading.

This staged approach, combined with Shipyard’s fast provisioning and immutable AMIs, lets teams safely deploy complex updates at scale while maintaining observability and control.

What Happens in the Gondola Stage?

When Gondola builds an artifact, it produces a deployable package for a service with two main parts:

• The AMI to roll out across the fleet
• The Chef artifact containing versioned recipes associated with a Git commit

Gondola treats these together as a single deployable unit. Each stage uses a service-defined executor to perform the rollout:

• ASG-based deployments: The executor updates the launch template with the new AMI and configuration. Chef code is packaged to Amazon Simple Storage Service (S3), and new instances use a baked-in bootstrapper to fetch the correct artifact and run the relevant recipes. Configuration metadata ensures only the right configuration is applied.
• Kubernetes worker fleets: The executor tells Karpenter which AMI to use and provides the same configuration metadata. Nodes bootstrap in the same way, ensuring consistent provisioning.

For services with special deployment needs, Gondola makes it easy to add new executors. This lets Shipyard support different deployment models while keeping artifact handling and provisioning consistent and predictable.

By combining AMI updates, versioned configuration artifacts, and metadata-driven bootstrapping, Gondola ensures every instance gets the correct software and configuration for its role, no matter how it’s deployed.

Simple Pipeline Shared Responsibility

Shipyard’s layered image model is built on a clear shared responsibility between platform teams and service teams.

The Compute, Security, and Monitoring teams manage the base layer, making sure it includes all global infrastructure components, security patches, and essential configurations. Service teams then build their own AMIs on top of this base, adding the software and settings specific to their service.

Whenever the Compute team rolls out a fix or update-whether it’s a security patch, monitoring agent upgrade, or networking improvement-a new slack-zero image is produced. Downstream service pipelines are automatically triggered to rebuild on top of the updated base, ensuring every team inherits the latest fixes without manual intervention.