[Databricks on AWS #6] How We Structure the Terraform: Terragrunt, YAML-Driven Modules, and Atlantis GitOps

The Repo, From the Top

Two ideas do all the work: reusable modules and per-environment trees.

infra/
├── _modules/          # reusable Terraform modules (the "what")
├── <project>/
│   └── databricks/
│       ├── dev/       # per-env tree (the "where")
│       └── prd/
├── common_vars.yaml   # shared config: hosts, account IDs, CIDRs
└── provider.tmpl      # provider template

_modules/ is a flat library of single-purpose Terraform modules - one per resource concept:

• workspace, unity-catalog, catalog, schema, grants, group, group-ar
• instance-pool, cluster-policy, compute, sql-warehouse, entitlements
• external-location, workspace-assignment, acls, iamrole, iampolicy
• s3, kms, vpc, service-principal, audit-log-delivery, workspace-conf

None of these know about dev or prd. They take inputs and create resources. The environment tree decides which modules run, with what values, in what order. A new workspace is a new folder under dev/, not a new module.

Humans Write YAML, Not HCL

Here's the rule we enforce: a person editing this repo edits YAML. They don't write Terraform, and mostly don't write Terragrunt either. The flow is three hops:

human-authored *.yaml → terragrunt.hcl: yamldecode() + env-prefix → Terraform module inputs

Each leaf directory has a small *.yaml (the desired state, in human terms) and a terragrunt.hcl that reads it. The terragrunt.hcl does yamldecode() on the YAML, then rewrites the keys with an environment prefix before handing them to the module.

Concretely, in the dev tree an instance-pool key like ip_cpu_small becomes dev_ip_cpu_small on the way in. Why the prefix? Because the same YAML shape lives in dev/ and prd/, but the resource names, pool IDs, and group names must be globally distinct. The env prefix is injected once, in one place, by Terragrunt - so no human ever hand-types dev_ or prd_ and nobody fat-fingers a cross-environment collision.

You change a number in YAML; the naming discipline is automatic. This is the single highest-leverage decision in the whole layout. The reviewer of a merge request reads a YAML diff - "pool max went 8 → 16" - not a wall of HCL.

One Provider Override Per Workspace

Databricks has a wrinkle that AWS doesn't: every workspace is its own API host. A databricks_cluster in workspace A and one in workspace B are created against different host URLs, even though it's the same account, same provider, same code.

We solve this with generated provider overrides. Each workspace layer knows its workspace_key (from a small workspace.hcl), and common_vars.yaml maps that key to the workspace host. Terragrunt looks up the host and generates a provider_override.tf on the fly:

locals {
  ws_host = local.common_vars.databricks
    .projects.<project>
    .workspace_hosts[local.env][local.ws_vars.locals.workspace_key]
}

generate "workspace_provider_override" {
  path      = "provider_override.tf"
  contents  = <<-EOF
provider "databricks" {
  host = "${local.ws_host}"
}
EOF
}

The module code is workspace-agnostic. The host is injected at plan time based on which folder you're in. Add a workspace, add its host to common_vars.yaml, drop a workspace.hcl with the key - every layer underneath now targets the right host with zero code changes.

The Two Databricks Providers, and Why

You'll notice the code uses the Databricks provider two different ways, and it's not an accident:

Account-level resources (the workspace itself, account groups, wiring a metastore) don't belong to any one workspace - they're configured against the account API, which is why they use the mws alias. Everything inside a workspace uses the workspace-scoped provider with the host we just injected.

Mixing these up is a classic early mistake: try to create a cluster with the mws provider and you'll get authentication errors that make no sense until you realize you're pointed at the account, not the workspace.

Dependency + mock_outputs (and the "init" Gotcha)

Layers depend on each other's outputs. cluster-policy needs the pool IDs that instance-pool produced; compute needs the policy IDs. Terragrunt expresses this with a dependency block:

dependency "instance_pool" {
  config_path = "../instance-pool"
  mock_outputs = {
    instance_pool_ids = {
      "dev_ip_cpu_small" = "mock-pool-id-1"
    }
  }
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}

The mock_outputs let you plan a layer before its dependency has ever been applied - Terragrunt feeds fake IDs so the plan can render. Handy.

But look closely at that allow-list: ["validate", "plan"]. No "init". That omission is the gotcha, and it ties straight back to the timeout saga from Parts 3 and 4. On a brand-new tree, the very first plan runs an init first - and because init isn't in the mock allow-list, Terragrunt refuses to mock the dependency and instead tries to read real outputs from a state that doesn't exist yet. Result:

Error: detected no outputs from dependency ".../instance-pool"

Your clean-looking layer fails to plan not because it's wrong, but because the thing upstream of it was never applied. Some layers in the repo (workspace-assignment, unity-catalog) deliberately include "init" in the allow-list to dodge this; the compute layers don't. The practical consequence is the same either way, and it's the whole reason for the next section.

Apply Order Is Not a Suggestion

Because of the dependency chain above, you apply in order. Not "usually." Always, the first time through:

Workspace line: workspace → instance-pool → cluster-policy → compute → workspace-assignment → acls

Unity Catalog: unity-catalog → external-location → catalogs → schemas → groups → grants

The rule underneath both lines is the same: apply a layer only after everything it depends on is already applied. Mocks let you plan out of order, but the first real apply has to walk the chain, because a mock pool ID doesn't create a cluster and a mock catalog doesn't hold a grant.

Two failure modes we hit repeatedly, both just ordering:

• cluster-policy / compute plan fails with "detected no outputs" → the pool/policy above it wasn't applied. Apply upstream first.
• workspace-assignment plans to assignments = {} → the groups layer wasn't applied, so there's nothing to assign. Apply groups first.

Neither is a bug. Both are the dependency graph telling you that you skipped a step.

Atlantis Ties the Bow

We don't run terragrunt apply from laptops. Every change is a merge request; Atlantis runs atlantis plan on the MR and atlantis apply once it's approved. The plan output lands in the MR as a comment, so the diff and its consequences are reviewable in one place.

Account credentials live server-side on the Atlantis host, not in anyone's shell - which also means local plan is optional (and honestly, usually skipped) since you get the real plan from the MR anyway.

Put together, the daily loop is small: edit a YAML value, open an MR, read the Atlantis plan, apply. The modules, the env-prefixing, the provider injection, the dependency graph - all of that is machinery you set up once and then mostly forget.

The Series, in One Breath

That's the whole platform:

• Building the platform - workspaces, account, Unity Catalog metastore.
• RBAC - function-role groups and workspace assignment.
• Compute governance - instance pools → cluster policies → clusters, and the guardrails around them.
• The BOOTSTRAP_TIMEOUT mystery - a healthy EC2 node that couldn't phone home, traced across three accounts to a firewall.
• PrivateLink - taking the control-plane traffic off the internet without touching the VPC.
• This - the Terraform structure holding all of it together.

If there's one throughline, it's that the boring parts - naming discipline, apply order, who-writes-what - are exactly the parts that decide whether a platform stays sane at the tenth workspace. Get the structure right and the interesting problems (like a packet dying in a firewall) are the only ones left to solve.

Thanks for reading all six. If you're just arriving, Part 1 is where the estate gets built from nothing.