Terraform on AWS

📹Watch the Lesson Video28 min

Terraform on AWS: Execution Model, Credentials, and the Apply Loop

You've written two hundred lines of HCL, you run terraform apply, and it stops with Error: reading S3 Bucket (checkout-config): ... 403 Forbidden. Nothing was created. The failure happened before Terraform attempted a single write. Or you get the opposite: a plan that wants to replace a database you never touched, which is terrifying and also not real. Both situations come from the same place — the gap between what you think Terraform is (a config language with a deploy button) and what it is (a CLI that logs into AWS with whatever identity it can find, reads the current truth of every resource it manages, and then diffs).

This section builds the operational model you'll use in every later section: how the provider is configured, how credentials are resolved, what terraform plan actually does against AWS, and how to read plan output the way an operator reads it. State storage, locking, and ongoing drift remediation get their own lessons later — here we only need to know that they exist and where the boundary is.

The Apply Loop in One Picture

Every Terraform operation against AWS is some slice of this loop:

terraform plan
  ↓
Load state (local file or remote backend)
  ↓
For each managed resource → AWS read API (Describe/Get/List)   ← the refresh
  ↓
Build dependency graph from attribute references + depends_on
  ↓
Diff desired config vs refreshed state
  ↓
Print plan (optionally saved with -out=tfplan)
  ↓
terraform apply → write APIs in dependency order, update state

Two details in that diagram matter more than the rest. First, the refresh phase makes read API calls, which means it needs credentials and read permissions — a plan is not a dry, offline computation. Second, terraform fmt and terraform validate are offline: they parse HCL and check syntax without touching AWS. That distinction explains a whole class of confusion, like a plan failing for permissions on a config that validate happily passed.

Provider Configuration on AWS

The provider is the plugin that translates resource blocks into AWS API calls. Its configuration decides three things you'll fight with constantly: which region, which tags, and which account.

Region resolution order

When you write provider "aws" {} with no region, Terraform doesn't guess — it walks a fixed order and uses the first hit:

💡 Pro Tip: Set the region explicitly in the provider block for anything you deploy repeatably. It turns a runtime surprise ("why did this land in us-east-1?") into a code review comment.

default_tags: one block, every resource

Rather than repeating the same three tags on eighty resources, declare them once:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      # a version constraint belongs here too — pinning is covered in
      # "Production Reality: Pitfalls, Guardrails, and Automated Pipelines"
    }
  }
}

provider "aws" {
  region = "us-east-1"

  default_tags {
    tags = {
      Project   = "checkout"
      Owner     = "payments-team"
      ManagedBy = "terraform"
    }
  }
}

Three behaviors are worth internalizing:

🧠 Merge, not replace. The provider merges default_tags into each resource's own tags at plan time. The computed tags_all attribute on each resource shows the merged result — that's the attribute to inspect when you're not sure what actually got sent.

🎯 Resource-level tags win on key conflict. If a resource sets Project = "legacy-checkout", that value survives; the default doesn't overwrite it. To change a default tag you must change it in the provider block or override it per resource.

🔒 Resources without a tags schema are silently skipped. A resource that has no tags concept in the provider schema simply doesn't receive them — no error, no warning.

⚠️ Common Mistake: Adding a tag via default_tags and then trying to remove it from a single resource by omitting it. Omitting does nothing — the default reapplies. Override the key with an explicit resource tag, or drop it from the provider.

Provider aliases: two regions, two accounts, one root module

An alias is a second, fully independent provider configuration. You reference it explicitly on each resource that should use it:

# Default provider: the account you run Terraform as
provider "aws" {
  region = "us-east-1"
}

# Aliased provider: a different region AND a different account,
# reached by assuming a role
provider "aws" {
  alias  = "west"
  region = "us-west-2"

  assume_role {
    role_arn     = "arn:aws:iam::111122223333:role/TerraformDeploy"
    session_name = "terraform-replication"
  }
}

resource "aws_s3_bucket" "primary" {
  bucket = "checkout-artifacts-east"
}

resource "aws_s3_bucket" "replica" {
  provider = aws.west
  bucket   = "checkout-artifacts-west"
}

Two rules that trip people up. First, aliased providers are never applied implicitly — every resource that should use aws.west must say so. Second, when a module contains its own provider blocks or expects an aliased provider, the caller passes it in via the providers meta-argument on the module block; how that interacts with module composition gets attention in "Composing AWS at Scale with Modules."

Credential Resolution in Practice

The AWS provider is a thin wrapper over the AWS SDK's default credential chain. It walks sources in order and uses the first one that resolves:

1. Static credentials in the provider block (access_key, secret_key, token) — if present, nothing else is consulted.
2. Environment variables — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN, with AWS_PROFILE, AWS_REGION, AWS_WEB_IDENTITY_TOKEN_FILE, and AWS_ROLE_ARN acting as selectors.
3. Web identity — AWS_WEB_IDENTITY_TOKEN_FILE plus AWS_ROLE_ARN, which is exactly the shape an IRSA-annotated pod provides.
4. Shared config and credentials files — the profile selected by AWS_PROFILE (or default), read from ~/.aws/credentials and ~/.aws/config (overridable with AWS_SHARED_CREDENTIALS_FILE / AWS_CONFIG_FILE). This is where SSO profiles and role_arn-based profile chains live.
5. Instance metadata — the EC2 instance profile via IMDS, or the container credential endpoint.

The finer interleaving of steps 2–4 follows the SDK rather than Terraform itself, so if you have several sources set at once, don't reason about it — confirm it:

aws sts get-caller-identity
# {"UserId":"...","Account":"111122223333","Arn":"arn:aws:iam::111122223333:role/TerraformDeploy"}

That single command answers 90% of "why is Terraform touching the wrong account?" questions before you touch a plan.

The three patterns you'll actually use

SSO for humans. aws sso login --profile platform-sso mints short-lived credentials into the SSO cache; then provider "aws" { profile = "platform-sso" } or AWS_PROFILE=platform-sso terraform plan. Your session expires on its own, which is the point.

Assume-role chains for cross-account. Either in the provider block:

provider "aws" {
  region = "us-east-1"

  assume_role {
    role_arn     = "arn:aws:iam::444455556666:role/TerraformApply"
    session_name = "terraform-ci"
    external_id  = var.external_id   # required by many third-party accounts
  }
}

...or implicitly through a profile that carries role_arn and source_profile, in which case the provider just reads the finished credentials.

Workload identity for machines. On EKS, an annotated service account (IRSA) projects a token into the pod and the pod runs your Terraform with AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE set — the provider exchanges it via sts:AssumeRoleWithWebIdentity and you never store a key. On EC2, the instance profile does the same job through IMDS. Both are covered in depth where compute is discussed; here the important part is that they land in the same credential chain Terraform already walks.

Why long-lived keys are a poor default for CI

An access key pair pasted into a CI secret is a permanent credential: it doesn't expire, it must be rotated manually, and it grants whatever policy it was attached to until someone notices. Anyone with read access to the pipeline config, the logs, or the secret store has standing access to your account. The modern default for AWS-aware CI is OIDC federation — the pipeline exchanges a short-lived identity token for a role session — and the pipeline mechanics for that live in "Production Reality: Pitfalls, Guardrails, and Automated Pipelines." The takeaway for this section: if your credential strategy requires an access key to sit somewhere for weeks, the strategy is the problem, not the rotation calendar.

💡 Mental Model: Credentials are just a selector on who Terraform is. Everything downstream — which account, which permissions, which resources it can even see during refresh — follows from that identity. Debugging a weird plan often means re-checking who you are, not what you wrote.

What terraform plan Really Does Against AWS

A plan is three phases, and most plan pain lives in the first and third.

Phase 1 — Refresh. For every resource in state, Terraform calls the provider, which calls a read API: DescribeInstances, HeadBucket, GetRole, ListTagsForResource. This is why a plan needs credentials, why it takes minutes on a large stack, and why read-only roles produce plans that fail. Terraform issues these calls concurrently — the graph walker's default -parallelism is 10 — so a stack managing a few hundred resources can fire hundreds of read calls in a burst.

Phase 2 — Graph construction. Terraform builds a dependency graph from attribute references. When you write subnet_id = aws_subnet.private.id, the provider doesn't need to be told that the subnet comes first; the reference is the edge. That's also why Terraform can decide at plan time that it doesn't know the value yet: the subnet ID doesn't exist until apply.

Phase 3 — Diff. Desired config is compared to the refreshed state, and Terraform prints the deltas plus an execution order. Data sources are read during this phase too, which is why a data source that resolves to something unexpected changes what the plan proposes.

Throttling and eventual consistency

Two AWS realities make plans slow or wrong-looking:

🔧 Throttling. Hundreds of concurrent reads will hit service limits — you'll see ThrottlingException, Rate exceeded, or RequestLimitExceeded in provider output. The fix is blunt and effective: terraform plan -parallelism=1 (or a low number) trades wall-clock time for a plan that completes. Adding -refresh=false skips the read phase entirely and is legitimate when you just changed a tag and know nothing else moved — but you're then diffing against a possibly stale picture of the world.

🔧 Eventual consistency. Some AWS read APIs don't immediately reflect a write. IAM is the classic offender: a role created seconds ago can fail to read back, producing an error on the very next plan. Other services normalize what you send — tag casing, list ordering, default values filled in server-side — so the refreshed state doesn't textually match your config even though reality is fine. The symptom is a phantom diff: plan says a change is needed, and running plan again says nothing changed. That's a read-side artifact, not drift you caused. Genuine drift — where AWS or a human actually changed something — is diagnosed and remediated in the drift lesson; the operator's rule here is simply: re-run the plan before believing a surprising diff. A second plan is a read-only operation; the worst case is that you waited another minute.

💡 Real-World Example: A plan reports ~ tags = { ... } # forces replacement-free but noisy changes on a bucket right after creation. Nothing is wrong — the read API returned a normalized tag set. The next plan is clean, and you've learned to not touch anything based on a single plan against freshly created resources.

Reading a Plan Like an Operator

Plan output is a report, and reports have a reading order. Learn the symbols first:

Then scan in this order: (1) every # forces replacement line, (2) the summary line's destroy count, (3) every (known after apply). Replacement destroys a live object and can take its data with it; destroy counts are the sanity check on scope; unknowns are where plan-time confidence ends.

Force-new attributes

Some attributes are marked ForceNew in the provider schema because AWS cannot change them in place. Changing one turns a harmless-looking edit into a destroy-and-recreate:

Terraform will perform the following actions:

  # aws_db_instance.primary must be replaced
-/+ resource "aws_db_instance" "primary" {
      ~ engine_version    = "16.3" -> "16.4"
      ~ identifier        = "checkout-prod" -> "checkout-prod-v2"
      ~ instance_class    = "db.r6g.large" -> "db.r6g.xlarge"
      + storage_encrypted = true # forces replacement
        # (4 unchanged attributes hidden)
    }

  # aws_launch_template.app will be updated in place
  ~ resource "aws_launch_template" "app" {
      ~ latest_version = 3 -> (known after apply)
        # (11 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Read it the way an operator does. The engine_version bump and instance_class resize are in place — no replacement, though AWS applies both with downtime. The identifier rename would be in place too. But storage_encrypted carries # forces replacement, which means this plan destroys the database and builds a new one: new endpoint, new data directory, unless you've planned for restoration. The launch template update is in place, but latest_version is (known after apply) because AWS assigns that number on write — so you cannot see the final value in this plan, and anything downstream that depends on it is also unknown until apply.

(known after apply) propagation

Unknown values are contagious. If aws_launch_template.app.latest_version is unknown, then an aws_autoscaling_group that references it has an unknown launch template version, and a resource referencing that is unknown too. Two practical consequences:

🧠 Plan-time validation only covers what's known. A malformed value that arrives from an unknown attribute surfaces as an apply failure, not a plan failure.

🧠 "No changes" plans and "all unknown" plans feel similar in the summary line but mean opposite things. Always read the body.

🎯 Key Principle: A plan is a prediction with known gaps. The summary line tells you scope; the # forces replacement annotations and (known after apply) markers tell you where the prediction is dangerous.

Diagnosing credential and permission errors from provider output

Provider errors are structured. They name the resource, the API action, and the failure class — which is enough to fix the problem without re-running apply:

Error: reading S3 Bucket (checkout-config): operation error S3: HeadBucket,
https response error StatusCode: 403, api error Forbidden

The action name (HeadBucket) points you at the permission you're missing — usually the IAM action of the same name, though not always: HeadBucket itself is authorized by s3:ListBucket — and the phase word tells you which side: reading means a refresh-phase read permission, creating/updating means a write permission. A different error shape — no valid credential sources for Terraform AWS Provider found — is a credential-resolution failure, not a policy failure, and the fix is in the chain, not in IAM.

⚠️ Common Mistake 1: Re-running terraform apply to "get past" a plan error. If the error came from a read during refresh, apply fails identically — you've only wasted a minute and possibly created confusion about whether something half-applied. ⚠️

⚠️ Common Mistake 2: Hard-coding access_key and secret_key in the provider block. Those values end up in any saved plan file, and they override the entire credential chain you carefully set up. ⚠️

⚠️ Common Mistake 3: Leaving AWS_ACCESS_KEY_ID exported in a shell from months ago and wondering why your AWS_PROFILE is ignored. Environment variables outrank the profile. aws sts get-caller-identity reveals it instantly. ⚠️

When you need the underlying API detail, TF_LOG=debug terraform plan (or TF_LOG_PROVIDER=debug to limit noise to the provider) prints the calls and responses. Treat those logs as sensitive: Terraform redacts values marked sensitive in the schema, but debug output is not a safe artifact to paste into a ticket.

For reviewing a saved plan mechanically, dump the JSON and filter rather than squinting at text:

terraform plan -out=tfplan

# list every resource this plan will destroy or replace
terraform show -json tfplan | jq -r '
  .resource_changes[]
  | select(.change.actions | index("delete"))
  | .address'

The same saved plan is what an apply should consume so that CI applies exactly what a human reviewed; the pipeline that wires that up is the subject of the production section. And terraform plan -detailed-exitcode gives you a machine-readable signal — exit code 0 means no changes, 2 means changes are pending, 1 means error — which is how a pipeline decides whether to request approval.

Where This Hands Off

Two questions this section deliberately leaves open: where does state live and who can write to it at the same time, and how do I detect and remediate drift continuously rather than noticing it in a plan. Both are their own lessons — the state/backend material and the drift material — and both build directly on the loop you now have: resolve identity, refresh through read APIs, build the graph, diff, and only then write.

💡 Remember: every Terraform operation on AWS is a conversation with an API using a specific identity — when something looks wrong, check who you are and what a second plan says before you change any code.

Authoring AWS Resources in HCL: Worked Patterns

The AWS provider is a schema, not a helper library. Every resource block you write is validated against that schema and translated into one or more AWS API calls, and the shape of the schema — which arguments you may set, which attributes you can only read, and which sub-resources AWS refuses to accept out of order — decides whether your apply succeeds on the first run or fails halfway through with a 400 you have to reverse-engineer. The patterns below are the ones you write every week, together with the ordering rules the provider and the underlying AWS APIs actually enforce.

Resource Block Anatomy in AWS Terms

Three things live inside a resource block, and confusing them is the root of most first-apply failures.

• Arguments are the inputs you set: bucket = "...", instance_type = "...". Terraform sends these to AWS.
• Computed attributes are values AWS assigns after creation and that you can only read: .id, .arn, .endpoint, .latest_version. You can never write to them; if you try, terraform validate rejects the config because they are not part of the resource's argument schema.
• Nested blocks are the provider's way of expressing structure that AWS represents as a sub-object: versioning_configuration, launch_template, instance_refresh.

Everything else follows from one fact: when you write aws_s3_bucket.config.arn inside another resource's argument, you are not "passing a value" — you are adding a directed edge to the dependency graph. Terraform will not evaluate that expression until the bucket exists, so creation order is a consequence of your references, not of the order the blocks appear in the file.

aws_s3_bucket.config
        │
        │  .id / .arn referenced by every sub-resource below
        ▼
aws_s3_bucket_versioning.config
aws_s3_bucket_server_side_encryption_configuration.config
aws_s3_bucket_public_access_block.config
aws_s3_bucket_ownership_controls.config
aws_s3_bucket_lifecycle_configuration.config

The reverse is true too, and usually forgotten: on destroy, Terraform walks those same edges backwards, so the sub-resources are removed before the bucket. You get that ordering for free by referencing aws_s3_bucket.config.id instead of re-typing the bucket name as a string.

⚠️ Common Mistake: assuming that two resources which both reference the bucket also depend on each other. They do not. aws_s3_bucket_public_access_block and aws_s3_bucket_ownership_controls have no reference between them, so Terraform creates them concurrently, in an order you do not control. When AWS requires a specific sequence, the graph cannot infer it — you have to state it explicitly. That specific case appears below, and the general rule lives in Control Flow and Lifecycle: for_each, count, Dependencies, and Meta-Arguments.

💡 Mental Model: read a resource block as three columns — what I send, what I receive back, and what I point at. The third column is what builds the graph.

🤔 Did you know? For most AWS resources .id is not a numeric identifier at all. It is whatever the provider's schema chose as the unique handle: aws_s3_bucket.config.id is the bucket name, aws_iam_role.app.id is the role name, and for VPCs it is vpc-.... Never guess which one a downstream resource wants — reference the precise attribute (.arn, .name, .id) and let the schema type-check it for you.

Worked Example: Hardening an S3 Bucket, End to End

If you learned S3 from an older tutorial, you saw versioning, acl, and server_side_encryption_configuration written as arguments inside aws_s3_bucket. The AWS provider has since split every one of those into a standalone resource. The inline forms are deprecated, and mixing the two styles produces a persistent diff: Terraform reads the bucket, sees the setting written by the sub-resource, and the root resource reports a change on every plan forever. The standalone resources are the pattern — and their ordering constraints are the interesting part.

# data.aws_caller_identity is declared once at the stack edge (shown later).
resource "aws_s3_bucket" "config" {
  bucket = "${var.name_prefix}-config-${data.aws_caller_identity.current.account_id}"

  tags = {
    Name = "${var.name_prefix}-config"
  }
}

# 1. Ownership controls first. BucketOwnerEnforced disables ACLs entirely, so
#    nothing downstream can hand out grants that contradict this setting.
resource "aws_s3_bucket_ownership_controls" "config" {
  bucket = aws_s3_bucket.config.id

  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

# 2. Public access block. All four flags set — partial blocking is how buckets
#    end up readable through a bucket policy rather than an ACL.
resource "aws_s3_bucket_public_access_block" "config" {
  bucket = aws_s3_bucket.config.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# 3. Versioning. Must exist before any lifecycle rule that acts on noncurrent versions.
resource "aws_s3_bucket_versioning" "config" {
  bucket = aws_s3_bucket.config.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "config" {
  bucket = aws_s3_bucket.config.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.config.arn
    }

    # Bucket keys cut KMS request volume and cost on high-request-rate buckets.
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "config" {
  bucket = aws_s3_bucket.config.id

  # No attribute reference exists between lifecycle and versioning, so the graph
  # cannot infer this edge. The provider's own pattern wires it explicitly.
  depends_on = [aws_s3_bucket_versioning.config]

  rule {
    id     = "expire-noncurrent"
    status = "Enabled"

    filter {} # empty filter = every object in the bucket

    noncurrent_version_expiration {
      noncurrent_days = 90
    }

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

Read the ordering constraints as consequences of the AWS APIs, not of Terraform:

⚠️ Common Mistake: adding aws_s3_bucket_acl with acl = "public-read" to a bucket whose ownership controls are BucketOwnerEnforced. S3 rejects the call with AccessControlListNotSupported, because ACLs are switched off for that bucket. The fix is not to weaken ownership controls but to write a (narrowly scoped) bucket policy. If you genuinely need legacy ACL semantics, set object_ownership = "ObjectWriter" deliberately, in its own commit, with a comment explaining who still depends on ACLs.

⚠️ Common Mistake: setting three of the four public access block flags. The combination block_public_policy = false with restrict_public_buckets = true is legal and still lets someone attach a public bucket policy; the only thing holding it back is restrict_public_buckets, which limits access through a public policy to AWS service principals and authorized users in the bucket owner's account. If the requirement is "never public," set all four — and enforce it in CI as well, which is what the policy gates in Production Reality: Pitfalls, Guardrails, and Automated Pipelines are for.

A lifecycle rule with no action is invalid; every rule needs at least one transition, expiration, or abort action. The empty filter {} block means "apply to every object" and is the current idiom — the older prefix argument on the rule is deprecated and generates a diff against rules created by the console.

IAM Policy Documents Without Hand-Written JSON

The hand-written heredoc is where IAM configurations go wrong, and the failure mode is a property of the approach rather than of the author:

# The pattern to move away from.
resource "aws_iam_role" "app" {
  name = "app-instance-role"

  assume_role_policy = <<-JSON
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Principal": { "Service": "ec2.amazonaws.com" }
        }
      ]
    }
  JSON
}

Three problems compound. First, every ARN you need inside the document must be interpolated as a string, so a typo in ${aws_s3_bucket.config.arn}/* becomes a runtime AccessDenied instead of a validation error. Second, the provider validates nothing inside that string — "Effect": "Aallow" passes terraform validate and fails at apply with MalformedPolicyDocument. Third, a one-word edit re-renders the entire string, so the plan diff shows the whole document as changed and the reviewer has to eyeball it line by line.

aws_iam_policy_document eliminates all three, because it is a data source with a real schema that renders JSON:

data "aws_iam_policy_document" "ec2_assume_role" {
  statement {
    sid     = "EC2AssumeRole"
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "app_permissions" {
  statement {
    sid    = "ReadConfig"
    effect = "Allow"

    actions = [
      "s3:GetObject",
      "s3:ListBucket",
    ]

    # Real references: the ARN resolves at apply time and the graph edge is free.
    resources = [
      aws_s3_bucket.config.arn,
      "${aws_s3_bucket.config.arn}/*",
    ]
  }

  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.config.arn,
      "${aws_s3_bucket.config.arn}/*",
    ]

    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

resource "aws_iam_role" "app" {
  name               = "${var.name_prefix}-app-instance-role"
  assume_role_policy = data.aws_iam_policy_document.ec2_assume_role.json
}

resource "aws_iam_policy" "app" {
  name   = "${var.name_prefix}-app-permissions"
  policy = data.aws_iam_policy_document.app_permissions.json
}

resource "aws_iam_role_policy_attachment" "app" {
  role       = aws_iam_role.app.name
  policy_arn = aws_iam_policy.app.arn
}

resource "aws_iam_instance_profile" "app" {
  name = "${var.name_prefix}-app-profile"
  role = aws_iam_role.app.name
}

The dependency chain is explicit and readable:

data.aws_iam_policy_document.ec2_assume_role  →  aws_iam_role.app
data.aws_iam_policy_document.app_permissions  →  aws_iam_policy.app
aws_iam_role.app + aws_iam_policy.app         →  aws_iam_role_policy_attachment.app
aws_iam_role.app                              →  aws_iam_instance_profile.app

What you gain, concretely: effect is validated against Allow/Deny at plan time; a duplicate sid is rejected when the document is rendered rather than by an AWS error string at apply; and because rendering is deterministic, editing one statement produces a diff of that statement rather than of the entire document. You still assign .json, and terraform plan still prints rendered JSON — reviewers read JSON either way. You simply author in block syntax.

💡 Pro Tip: prefer aws_iam_policy plus aws_iam_role_policy_attachment over inline aws_iam_role_policy. Inline policies are embedded in the role, share its size limit, and changing one rewrites the role in the plan — which both muddies the diff and makes an unrelated change look like a role modification during review.

Where this approach has limits — and it does — is worth knowing before you hit the wall. aws_iam_policy_document validates less than it appears to — the condition operator in test is an unchecked string, and nothing stops a statement from setting both actions and not_actions — and when you merge documents with source_policy_documents the data source can reorder or coalesce statements in ways that surprise you. If the merging gets in your way, jsonencode() on a local map is a legitimate fallback: it keeps deterministic rendering and still lets you interpolate ARNs, at the cost of losing per-statement provider validation. What you should not do is go back to a heredoc string.

⚠️ Common Mistake: assuming a role is usable the instant it exists. IAM is eventually consistent, so a Lambda or EC2 resource created in the same apply that assumes a brand-new role can fail with a role-cannot-be-assumed error. That invisible coupling is what depends_on exists for, and it is covered in Control Flow and Lifecycle: for_each, count, Dependencies, and Meta-Arguments.

EC2 Bootstrap: AMI Lookup, user_data, and Launch Templates

Hard-Coded AMI IDs Versus Discovery

# Works — and fails quietly for years.
resource "aws_instance" "app" {
  ami           = "ami-0abcdef1234567890"
  instance_type = "m6i.large"
}

That AMI ID is region-scoped: copy the config into another region and the apply fails immediately, which is the good failure. The bad one is silent. The region stays the same, the image ages out of patching, and every new instance launches on an image with known vulnerabilities. The rollout is green and the fleet is stale.

Discovery replaces the hard-coded value with a query:

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }

  filter {
    name   = "state"
    values = ["available"]
  }
}

⚠️ Common Mistake: omitting owners. Without an owner filter, a name-pattern match can return a community AMI published by someone else that happens to satisfy the same pattern — and you will launch it. Always scope the lookup to amazon, self, or a specific account ID you control.

Discovery carries its own tax, though: most_recent = true turns the AMI into a moving target. Any plan run after AWS publishes a new image shows a change you did not write, and depending on where the reference sits, that change can cascade into a replacement. Two ways to take control back:

• Pin the ID in a variable that a human promotes on a schedule. Predictable plans; you own the cadence.
• Publish the approved ID to SSM Parameter Store from your image pipeline and read it with data "aws_ssm_parameter". Discovery stays in the graph, and the change becomes an explicit, reviewable event in the pipeline rather than ambient drift appearing in every plan.

user_data Versus user_data_base64

user_data takes a plaintext string; the provider base64-encodes it before the EC2 API call. user_data_base64 takes content that is already encoded and forwards it untouched — the natural fit for filebase64("${path.module}/bootstrap.sh"), which keeps the script as a real file you can lint and test, and for base64gzip(...) when the script grows, since EC2 caps user data at 16 KB measured before encoding.

resource "aws_launch_template" "app" {
  name_prefix   = "${var.name_prefix}-app-"
  image_id      = data.aws_ami.al2023.id
  instance_type = var.instance_type

  iam_instance_profile {
    name = aws_iam_instance_profile.app.name
  }

  vpc_security_group_ids = [aws_security_group.app.id]

  # Already-encoded content: the provider forwards it unchanged.
  user_data_base64 = filebase64("${path.module}/bootstrap.sh")

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required" # IMDSv2 only
    http_put_response_hop_limit = 1
  }

  tag_specifications {
    resource_type = "instance"
    tags          = { Name = "${var.name_prefix}-app" }
  }
}

resource "aws_autoscaling_group" "app" {
  desired_capacity = var.desired_capacity
  min_size         = var.min_size
  max_size         = var.max_size

  vpc_zone_identifier = var.private_subnet_ids

  launch_template {
    id = aws_launch_template.app.id
    # A real reference: new template versions update the ASG's configuration.
    version = aws_launch_template.app.latest_version
  }

  instance_refresh {
    strategy = "Rolling"

    preferences {
      min_healthy_percentage = 90
    }

    triggers = ["launch_template"]
  }
}

instance_refresh with triggers = ["launch_template"] is the mechanism that makes a template change actually reach running instances: the ASG replaces them in batches while respecting min_healthy_percentage, instead of waiting for a scale event that may never come. The lifecycle meta-arguments that make this safe (creating the new template before destroying the old one) belong to Control Flow and Lifecycle.

⚠️ Common Mistake: writing the literal string version = "$Latest" in the ASG's launch_template block. AWS accepts it, so the apply succeeds — but Terraform sees only a static string with no dependency on the template's versions. Template changes then never update the ASG's configuration, and the running fleet quietly diverges from what your code claims. Use the latest_version attribute instead (or default_version if you promote versions deliberately); both create the reference edge.

⚠️ Common Mistake: editing user_data on a live aws_instance and expecting the bootstrap to re-run. The provider's user_data_replace_on_change argument defaults to false, so Terraform plans an in-place update, the provider stops and starts the instance, and the apply reports success — but by default user data scripts run only during the boot cycle when the instance is first launched, so the restart does not re-run them. You get a green apply, a restart, and an unchanged box. This is the single strongest argument for the launch-template pattern: put the bootstrap on the template, let instance_refresh roll the fleet, and stop mutating instances and hoping.

Discovery Data Sources and the Coupling Tax

Not all data sources carry the same risk. They split into two families, and the split is the rule you actually apply.

Provider-scoped lookups read the provider's own configuration — the credentials and region you set. data.aws_caller_identity.current returns account_id, arn, and user_id for whoever is running the apply; data.aws_region.current returns the resolved region; data.aws_partition.current returns the partition. There is exactly one correct answer for each, so they cannot resolve to "the wrong thing." Use them freely: they are how you build account-qualified bucket names and partition-aware ARNs without hard-coding an account ID that breaks the moment someone runs the same config in a test account.

Resource-scoped lookups read live AWS objects that other people can change or delete: aws_vpc, aws_subnet, aws_ami, aws_security_group, aws_iam_role. Every plan and apply re-resolves them, which means your plan depends on ambient account state rather than on your configuration. Three costs show up in practice:

Silent retargeting deserves the detail. data "aws_vpc" "default" { default = true } returns whatever VPC is currently flagged as the default in that account — if someone promotes a new default VPC, your stack now targets it, and nothing in the diff announces the switch. Tag-based lookups are riskier still: tags = { Name = "prod-vpc" } errors once two VPCs carry that tag, and lookups that tolerate multiple matches pick an arbitrary one. The failure mode is not an error; it is a subnet association on the wrong network, discovered weeks later.

The fix is placement, not avoidance: use resource-scoped lookups at the edge of a stack — in the root module, resolved once — and pass the resolved value down as a variable. That keeps the lookup reviewable in a single place and keeps modules on explicit inputs.

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}
data "aws_partition" "current" {}

data "aws_vpc" "selected" {
  id = var.vpc_id # explicit input; the lookup adds CIDR and owner metadata
}

locals {
  arn_partition = data.aws_partition.current.partition
  # AWS provider v6 deprecates `.name` in favor of `.region`. Under this lesson's
  # `~> 5.x` pins `.name` is the attribute to use; on v6, write
  # data.aws_region.current.region instead.
  region = data.aws_region.current.name
}

🎯 Key Principle: a data source that reads mutable infrastructure is an implicit input to your module. If a downstream module contains one, its behavior depends on account state the caller cannot see — which is precisely the contract problem that Composing AWS at Scale with Modules: Contracts, Layering, and Versioning solves with typed, validated variables.

The patterns above decide what Terraform writes into an account; what happens when that set of resources has to grow, shrink, or be re-identified is a question of resource identity and lifecycle, which is where the lesson goes next.

Control Flow and Lifecycle: for_each, count, Dependencies, and Meta-Arguments

A plan against AWS is really a list of four decisions — create, update, replace, destroy — and almost every one of them traces back to a single question the planner asks about each resource: is this the same object as last time? The answer comes from the resource's address, the string like aws_iam_user.team[1] or aws_s3_bucket.logs["eu"] that Terraform uses as the key into state. Choose the addressing scheme well and plans become boring. Choose it badly and Terraform cheerfully proposes replacing a healthy EC2 fleet because you deleted an unrelated list element.

That is the whole game here: resource identity, repeated nested configuration, the lifecycle meta-arguments that decide create-before-destroy ordering, and the dependency machinery — implicit edges plus the depends_on escape hatch — that decides what happens first.

Resource Identity: count vs for_each

With count, Terraform keys instances by position: [0], [1], [2]. With for_each, it keys them by a map key or a set member, and that key becomes part of the address. Everything else in this comparison follows from that difference.

Watch what happens when a list shrinks. Start with three IAM users:

variable "team_members" {
  type    = list(string)
  default = ["alice", "bob", "carol"]
}

# ❌ index-keyed: identity is positional
resource "aws_iam_user" "team" {
  count = length(var.team_members)
  name  = var.team_members[count.index]
}

State after the first apply, then the plan after removing "bob" from the list:

State:  aws_iam_user.team[0] = alice
        aws_iam_user.team[1] = bob
        aws_iam_user.team[2] = carol

List becomes: ["alice", "carol"]

Plan:
  [0]  alice -> alice    no change
  [1]  bob   -> carol    UPDATE    (renamed in place: name is not ForceNew)
  [2]  carol -> (gone)   DESTROY

Net effect: the plan renames bob's user to carol, so it keeps bob's
            access keys, group memberships, and inline policies, and
            it destroys carol's ORIGINAL user. Bob keeps his access; she loses hers.

Nothing in that plan destroys bob. The resource Terraform destroys had nothing to do with the change you made. Now the same input with for_each:

# ✅ key-keyed: identity is the name itself
resource "aws_iam_user" "team" {
  for_each = toset(var.team_members)
  name     = each.key
}

Addresses become ["alice"], ["bob"], ["carol"]. Remove bob and the plan is one line: destroy ["bob"]. Alice and carol are never touched, because their addresses never changed. Inside the block you read each.key (the map/set key) and each.value (the value), and when for_each is a map of objects, each.value.some_attribute gets you the fields.

🎯 Key Principle: Terraform destroys the resource whose address disappeared, not the resource whose value disappeared. count gives every instance an address borrowed from its neighbors, so neighbor churn becomes your churn.

The cost of getting this wrong is not abstract. Replace an index-keyed aws_db_instance and you get a new empty database plus a destroyed old one. Replace an index-keyed aws_instance behind a load balancer and you take a capacity hit plus a full re-bootstrap. Replace an index-keyed S3 bucket and the objects go with it. Index churn is a mechanism for converting a one-line config edit into downtime and data loss.

Where count is still the right answer

count is not deprecated, and reaching for for_each reflexively costs readability. Three cases genuinely favor it:

🧠 Toggles. count = var.enable_logging ? 1 : 0 creates-or-doesn't. There is no key to invent.

🧠 Genuinely homogeneous sets with no independent identity. Three NAT gateways, one per AZ, where you address them by AZ index and never remove the middle one. The identity risk is real but the code is simpler.

🧠 Splat access. With a toggle, reference the result with one() rather than a brittle [0]:

resource "aws_cloudwatch_log_group" "api" {
  count = var.enable_logging ? 1 : 0
  name  = "/app/api"
}

# length 0 -> null, length 1 -> the element. No index error when disabled.
output "api_log_group_arn" {
  value = one(aws_cloudwatch_log_group.api[*].arn)
}

⚠️ Common Mistake 1: using count for anything a human names — IAM users, S3 buckets, Route 53 records, ECS services, DynamoDB tables. If the element has a name in the real world, give it that name as its key via for_each, and deletions stop being neighborhood events. Section 5, "Brownfield Reality: Importing, Refactoring, and Adopting Existing AWS," covers the moved blocks that let you switch an existing resource from count to for_each without destroy/recreate.

⚠️ Common Mistake 2: passing a list to for_each. It only accepts a map or a set of strings, so toset(var.things) is the usual fix — but toset silently deduplicates. A list with a repeated email address yields fewer resources than its length, with no warning. count would at least have failed loudly on the duplicate name.

⚠️ Common Mistake 3: deriving for_each keys from values Terraform won't know until apply — a VPC ID invented by another resource in the same run, for example. You get an Invalid for_each argument error because the planner cannot know the full key set. Derive keys from variables, locals, or values already in state; if that's impossible, you may need two applies or a depends_on that defers the read. A resource cannot set both count and for_each — Terraform rejects it outright.

Dynamic Blocks: Repeating Nested Configuration

Some AWS resources take repeated nested blocks rather than a list argument — security group ingress rules, ALB listener default_action blocks, container definitions. A dynamic block generates those from a collection, so the rule set lives in a variable instead of being copy-pasted into HCL.

variable "ingress_rules" {
  type = list(object({
    from_port   = number
    to_port     = number
    cidr_blocks = list(string)
    description = string
  }))
  default = []
}

resource "aws_security_group" "app" {
  name        = "app-sg"
  description = "app tier"
  vpc_id      = var.vpc_id

  # The iterator is named after the block label ("ingress") by default.
  dynamic "ingress" {
    for_each = var.ingress_rules
    content {
      from_port   = ingress.value.from_port
      to_port     = ingress.value.to_port
      protocol    = "tcp"
      cidr_blocks = ingress.value.cidr_blocks
      description = ingress.value.description
    }
  }

  tags = { Name = "app-sg" }
}

You can rename the iterator with iterator = rule and then read rule.value — worth doing when the block label is generic and ingress.value.from_port reads poorly. dynamic only produces nested blocks, so it cannot generate top-level meta-arguments such as lifecycle; those must be written literally.

⚠️ Common Mistake 4: nesting dynamic inside dynamic three levels deep because it's technically possible. Deeply nested dynamic blocks produce plans that are hard to read and configurations nobody can debug at 2 a.m.

🎯 Key Principle: A dynamic block is a loop over configuration, not a loop over resources. Every generated rule shares one resource address, so you cannot target, import, move, or destroy a single rule independently. The moment rules need their own lifecycle, promote them to a resource with for_each:

# One address per rule: aws_vpc_security_group_ingress_rule.app["443-api"]
resource "aws_vpc_security_group_ingress_rule" "app" {
  for_each = { for r in var.ingress_rules : "${r.from_port}-${r.description}" => r }

  security_group_id = aws_security_group.app.id
  cidr_ipv4         = each.value.cidr_blocks[0]
  from_port         = each.value.from_port
  to_port           = each.value.to_port
  ip_protocol       = "tcp"
  description       = each.value.description
}

This is the current AWS provider idiom: prefer the standalone aws_vpc_security_group_ingress_rule / ..._egress_rule resources over inline ingress blocks, because each rule then has its own address and its own destroy path. Adding rule 14 no longer rewrites rules 1–13. Note that inline rules and standalone rules for the same security group conflict — the provider will fight you if both exist.

The decision rule is short. Use dynamic when the nested block is genuinely a value of the parent — a listener's action list, an ECS container definition — and the whole set is created, validated, and replaced together. Promote to a resource when a human would ever want to change one item on its own.

Lifecycle Meta-Arguments

Four settings inside lifecycle change what a plan is allowed to do. All four exist because Terraform's default behavior — destroy the old object, then create its replacement, in dependency order — is occasionally the wrong order for AWS.

create_before_destroy

create_before_destroy = true inverts the ordering: build the replacement first, then destroy the original. The canonical AWS case is an Auto Scaling group whose replacement would otherwise drop all capacity at once. The default ordering tears the ASG down before the new one exists, so the load balancer briefly has zero healthy targets:

resource "aws_launch_template" "app" {
  name_prefix   = "app-"   # generated name, not a fixed one
  image_id      = data.aws_ami.al2023.id
  instance_type = "m7g.large"
  # ...
}

resource "aws_autoscaling_group" "app" {
  name_prefix         = "app-asg-"
  vpc_zone_identifier = var.subnet_ids
  min_size            = 3
  max_size            = 12

  launch_template {
    id      = aws_launch_template.app.id
    version = aws_launch_template.app.latest_version
  }

  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 90
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

⚠️ Common Mistake 5: enabling create_before_destroy on a resource that has a hard-coded unique name. Both objects exist simultaneously, so the create fails with an already-exists error. Use name_prefix (or let the provider generate) whenever CBD is on.

The published launch template in this example is worth a note, because the guidance "CBD for launch templates" needs a caveat. In the current AWS provider, most changes to aws_launch_template produce a new version in place — the resource is not replaced, so no create/destroy dance occurs at all. CBD matters when the template is genuinely replaced (changing name or name_prefix, both ForceNew) and when you're on the legacy aws_launch_configuration, which the provider replaces on nearly any change while the ASG still references it.

Also notice what this example deliberately does not do: it does not use replace_triggered_by to rebuild the ASG whenever the template changes. Destroying and recreating an ASG to roll out a new AMI is the sledgehammer; the instance_refresh block rolls instances with a health threshold while the ASG itself lives on.

prevent_destroy

prevent_destroy = true makes Terraform fail the plan if the resource would be destroyed. Not a runtime guard — a planning refusal.

The AWS cases are the ones with irreplaceable state or slow recovery: aws_db_instance, aws_rds_cluster, aws_dynamodb_table, aws_s3_bucket, aws_kms_key with a scheduled deletion window. Two things matter for how you actually use it.

First, lifecycle arguments are static configuration, not runtime switches. You cannot write prevent_destroy = var.is_prod; a variable reference there is a parse error. The workable pattern is a separate protected configuration or a duplicated resource block, not a conditional.

Second, prevent_destroy protects against Terraform destroying the object. It does nothing about a console deletion, a terraform state rm, or a mistaken aws s3 rb. Pair it with the AWS-native control so you have two independent layers: deletion_protection = true on an RDS instance, versioning plus an MFA-delete or lifecycle policy on a bucket. And remember it blocks terraform destroy for the whole stack — a prevent_destroy on a sandbox bucket will annoy whoever is tearing down the sandbox, so scope it to the resources that genuinely hurt.

ignore_changes

ignore_changes tells Terraform to stop comparing specific attributes against state. Every attribute listed becomes a permanent blind spot: drift in it is never reported and never corrected.

resource "aws_autoscaling_group" "app" {
  name             = "app-asg"
  min_size         = 3
  max_size         = 12
  desired_capacity = 6

  lifecycle {
    # A target-tracking scaling policy moves this constantly.
    # Without this line, every plan wants to drag capacity back to 6.
    ignore_changes = [desired_capacity]
  }
}

resource "aws_s3_bucket" "data" {
  bucket = "acme-data-prod"

  lifecycle {
    # Written by a cost-allocation tool, not by you.
    # Scope to the key — ignoring all of `tags` also hides YOUR tag drift.
    ignore_changes = [tags["LastModified"]]
  }
}

ATTR references are also accepted — tags["LastModified"] targets one key, while ignore_changes = all freezes the entire resource. The whole-resource form is the right tool for adopting a resource someone else manages and then leaving it alone; it is the wrong tool for anything you still want Terraform to enforce.

The ECS analogue of the ASG case is aws_ecs_service.desired_count when Application Auto Scaling owns it — same mechanism, same perpetual-diff symptom if you skip it. The tag case is broader than people expect: cost-allocation tooling, backup services, and security scanners all attach tags to AWS resources, and the AWS provider will keep proposing to remove them until you either ignore those keys or move the tooling to a tag namespace you also manage.

❌ Wrong thinking: "The plan shows a diff, so I must have a mistake in my config." ✅ Correct thinking: "The plan shows a diff because something outside Terraform changed the object. Either Terraform should own it — fix the config — or it shouldn't, and I add ignore_changes scoped to that exact attribute."

replace_triggered_by

replace_triggered_by forces replacement of a resource when a referenced object changes or is itself replaced. It exists for cascades the dependency graph cannot infer, because there is no attribute reference to hang the edge on.

The clean AWS pattern uses a terraform_data resource as a replace signal — a small built-in resource whose triggers_replace argument exists purely to make "this value changed" visible to the planner:

# Fires a replacement whenever var.app_release changes.
resource "terraform_data" "app_release" {
  triggers_replace = [var.app_release]
}

resource "aws_instance" "app" {
  ami           = data.aws_ami.al2023.id
  instance_type = "m7g.large"
  subnet_id     = var.subnet_id

  lifecycle {
    # A launch template change does not touch running instances.
    # This is the explicit "rebuild the pet when the release moves" signal.
    replace_triggered_by = [terraform_data.app_release]
  }
}

The same pattern replaces a null_resource with triggers in older code; terraform_data requires Terraform 1.4 or later and is the current idiom. The meta-argument accepts references to resources or their attributes, not arbitrary expressions — which is exactly why the terraform_data shim exists. Reach for it when the trigger is a value that has no home as a resource attribute: an application version string, a config hash, a rotated secret's version.

Dependencies: Implicit Edges and Invisible Couplings

Terraform builds a dependency graph and applies it in topological order. Most edges are implicit: when you write role = aws_iam_role.task.arn, the function resource now depends on the role, and the planner knows it without being told. Every attribute reference is an edge. Most of the time that is all you need, and most depends_on in real codebases is redundant noise that serializes the apply and hides what actually depends on what.

depends_on earns its place only when a coupling exists that no attribute reference expresses. On AWS there are two recurring categories.

IAM propagation

IAM is eventually consistent. A role, its trust policy, and an attached policy can all report as created while an AWS service still cannot see them. Terraform orders the API calls correctly and the service still fails:

resource "aws_iam_role" "task" {
  name               = "app-task"
  assume_role_policy = data.aws_iam_policy_document.task_assume.json
}

resource "aws_iam_role_policy" "task_logs" {
  name   = "logs"
  role   = aws_iam_role.task.id
  policy = data.aws_iam_policy_document.task_logs.json
}

resource "aws_lambda_function" "api" {
  function_name = "app-api"
  role          = aws_iam_role.task.arn   # implicit edge to the ROLE only
  # ...

  # The role ARN reference does NOT order this after the inline policy,
  # and even correct ordering does not beat IAM propagation. Without this,
  # creation intermittently fails with "the role cannot be assumed".
  depends_on = [aws_iam_role_policy.task_logs]
}

Read the comment carefully: the reference to aws_iam_role.task.arn creates an edge to the role, not to the policy attached to it. The Lambda can be created in the window where the role exists and the policy does not. depends_on fixes the ordering; it does not fix propagation delay itself. If a first apply still fails on a cold account, the durable answer is a retry (or a short time_sleep between the attachment and the consumer), and CI pipelines that apply once with no retry should assume this class of error will appear occasionally.

Service-side validation

Some AWS APIs validate that a permission exists before they will accept a configuration. S3 bucket notifications are the classic:

resource "aws_lambda_permission" "allow_s3" {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.api.function_name
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.data.arn
}

resource "aws_s3_bucket_notification" "data" {
  bucket = aws_s3_bucket.data.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.api.arn
    events              = ["s3:ObjectCreated:*"]
  }

  # The notification block references the FUNCTION, not the PERMISSION.
  # S3 validates the permission at put-time and rejects the config without it.
  depends_on = [aws_lambda_permission.allow_s3]
}

🎯 Key Principle: depends_on is justified only when you can name the invisible coupling out loud. "The Lambda needs the policy attached" and "S3 validates the permission" are sentences you can say. "Terraform might get the order wrong" is not — find the reference you're missing instead. And keep the blast radius small: depends_on on a module or a broad resource can serialize large parts of the graph and slow applies measurably.

One Module Call, Many Accounts and Regions

The tempting next step is to fan a module out across regions or accounts with for_each. It does not work the way people expect, and the reason is worth internalizing: provider configuration is resolved statically, at parse time, before any for_each key exists. You cannot write provider = aws[each.key]. The provider argument must be a fixed reference.

So the shape is one module call per account/region, each receiving a distinct statically-declared alias, with for_each used inside each call for everything that shares that provider:

provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  alias  = "eu_west_1"
  region = "eu-west-1"
}

# One call per (account, region). The provider is fixed for the whole call.
module "app_us" {
  source    = "./modules/app"
  providers = { aws = aws }
  cidr      = "10.10.0.0/16"
}

module "app_eu" {
  source    = "./modules/app"
  providers = { aws = aws.eu_west_1 }
  cidr      = "10.20.0.0/16"
}

Inside ./modules/app, for_each works normally for anything within that one provider's scope — one bucket per environment, one subnet per AZ:

resource "aws_s3_bucket" "logs" {
  for_each = toset(var.environments)
  bucket   = "${var.name}-logs-${each.key}"
}

The constraints this places on what can share a module call are concrete:

🔒 Provider selection is static. Anything that varies by account or region cannot be a for_each key that drives provider choice. Region and account belong in separate module calls, not in a loop.

🔒 All instances of one call share a provider. module "app" { for_each = ... } with a single providers map gives every instance the same account and region. A module-level for_each across accounts is not expressible this way.

🔒 A module needing two provider configurations must declare the extra alias. If a child module both creates regional resources and reads something from a global region, it declares the alias in required_providers:

terraform {
  required_providers {
    aws = {
      source                = "hashicorp/aws"
      version               = "~> 5.0"
      configuration_aliases = [aws.us_east_1]
    }
  }
}

The parent then passes both: providers = { aws = aws.eu_west_1, aws.us_east_1 = aws }. Without the configuration_aliases declaration, the child module has no way to accept the second configuration, and the call fails at init.

💡 Pro Tip: In practice, multi-account fan-out is usually handled above Terraform rather than inside it — the same root module applied once per account by a pipeline that assumes a different role each time. Section 6, "Production Reality: Pitfalls, Guardrails, and Automated Pipelines," covers how that pipeline should assume those roles without stored keys. Multi-region fan-out within one account is where the static-alias module-call pattern above pays off, because both regions share the same credentials and differ only by provider configuration.

💡 Mental Model: for_each moves identity into your hands; count leaves it with the planner; lifecycle arguments move ordering into your hands; depends_on moves edges into your hands. Every one of those is a deliberate trade of convenience for control, and each is worth reaching for only when you can state what the default would get wrong.

Composing AWS at Scale with Modules: Contracts, Layering, and Versioning

Your first VPC lives inline in main.tf and it works. The second environment means copying that block and changing a few strings. By the fifth application you have five near-copies, and a security-group fix has to land in five places — or, realistically, in four, because someone already patched one by hand in the console. Modules replace those copies; but a module another engineer cannot call safely is worse than the copy-paste it removed, because it hides the coupling instead of eliminating it. The skill in this section is deciding what a module promises, how stacks relate to one another, and how that promise gets versioned.

Module Contracts: Every Version Is an Interface

Treat a module the way you would treat a published library. The caller sees inputs and outputs; everything else is implementation. If someone has to read your module's internals to use it correctly, the contract has failed.

Three properties make a module callable sight-unseen:

• 🧠 Typed inputs — every variable declares a type, so a string passed where a list belongs fails at plan time rather than mid-apply.
• 🔒 Validated inputs — a validation block catches values that are the right type but the wrong domain: a tier name that does not exist, a CIDR larger than the VPC.
• 📚 Documented outputs — every output carries a description and a stable name, because outputs are the module's public surface.

variable "name" {
  description = "Prefix applied to every resource this module creates"
  type        = string
}

variable "tier" {
  description = "Subnet tier this block wires up: public, private, or data"
  type        = string

  validation {
    condition     = contains(["public", "private", "data"], var.tier)
    error_message = "tier must be one of: public, private, data."
  }
}

variable "single_nat_gateway" {
  description = "Share one NAT gateway across all private subnets (non-prod only)"
  type        = bool
  default     = false
}

The description strings are not decoration — they surface as prompts during an interactive terraform plan, in registry documentation, and in editor autocomplete. The type gives early failure. The validation block converts a class of runtime surprises — "why is there a subnet called data-1 in the wrong VPC?" — into a plan-time error that prints your own message. The default on single_nat_gateway encodes a decision most callers should not have to make, while leaving the one caller who cares free to override it.

🎯 Key Principle: a module version is an API contract, and semantic versioning is how you communicate changes to it:

• MAJOR — breaking: a removed output, a renamed variable, a newly required input.
• MINOR — additive and compatible: a new optional variable with a default, a new output.
• PATCH — internal fixes that change no input, no output, and no created resource identity.

⚠️ Common Mistake: adding a required variable and shipping it as a patch. Every caller pinning the previous minor now fails to plan on its next init, and they discover it in the middle of an unrelated change. If the value is genuinely required it is a MAJOR bump; if it is not required, give it a default and ship MINOR.

The Three Composition Shapes

Factoring AWS at scale happens in one of three overall shapes, and the choice is not stylistic — it decides blast radius, apply time, and how tightly teams are coupled. The comparison below is the crux of this section.

Shape 1: The Monolithic Root Stack

Everything — networking, databases, compute, IAM — in one root module with one state file. This is the correct default for a single small system, and it is genuinely pleasant: one plan shows the entire delta, there is no cross-stack ordering to reason about, and any resource can reference any other by direct attribute reference.

It stops being pleasant as the system grows. The blast radius is the entire account: because every resource is in one plan, a single change can put a database replacement and a subnet edit in the same apply, with no way to ship one without the other. Apply time grows with resource count, because a plan refreshes every managed resource and each refresh is a read call against an AWS API — subject to throttling and eventual consistency, as covered in "Terraform on AWS: Execution Model, Credentials, and the Apply Loop." A plan that takes twelve minutes is a plan people begin avoiding, and avoided plans are exactly where drift accumulates. Finally, everyone who needs to change a security group edits the same directory, so there is no natural review boundary.

Shape 2: Layered Stacks

Split the system into ordered layers, each with its own state file, wired so the compute layer consumes the network layer's published outputs.

network stack        (state: prod/network)
   |  outputs: vpc_id, subnet IDs by tier, route table IDs
   v
data stack           (state: prod/data)
   |  outputs: db endpoint, db security group ID
   v
compute stack        (state: prod/compute)

The compute layer reads what the network layer published instead of re-deriving it:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tfstate"
    key    = "prod/network/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_db_subnet_group" "main" {
  name       = "acme-prod"
  subnet_ids = data.terraform_remote_state.network.outputs.database_subnet_ids
}

Blast radius is now one layer — a bad compute change cannot touch the VPC. Apply time is bounded by the layer, not the system. The cost is coupling through ordering: the network stack must apply before anything that reads it, and a renamed output breaks every consumer on their next plan. That is not a defect; it is the contract doing its job.

💡 Pro Tip: terraform_remote_state reads the entire remote state and hands you its outputs map. When you want a narrower, explicitly named handoff, publish the value as an SSM Parameter and read it with the aws_ssm_parameter data source — the consumer then depends on a named parameter rather than the producer's state layout. The trade-off runs the other way too: an SSM parameter can drift from the state it was derived from, while a remote-state read cannot. Pick per handoff, not per project.

Cross-stack ordering is not something Terraform infers from your file layout — it is a rule you enforce in the apply pipeline, which is why layering and pipeline design travel together (see "Production Reality: Pitfalls, Guardrails, and Automated Pipelines").

Shape 3: The Shared Module Library

A repository — or a private registry namespace — of independently versioned modules, consumed by many root stacks. Blast radius is one consumer stack per apply, and coupling is interface-only: a stack pins a module version and is unaffected by upstream changes until it chooses to bump. This is the shape that scales past one team.

The risk migrates wholesale into versioning discipline. A module is now a dependency of stacks you have never seen, so an accidental breaking change in a MINOR release propagates the next time each consumer runs init — quietly, and at whatever moment they happen to be making an unrelated change. This is why the contract rules above matter more in a library than anywhere else.

AWS-Specific Module Boundaries: What a Network Module Must Output

A network module earns its existence the moment downstream modules stop re-deriving networking logic. That only happens if it publishes everything the next layer needs, by name, with stable meaning:

output "vpc_id" {
  description = "ID of the VPC created by this module"
  value       = aws_vpc.this.id
}

output "private_subnet_ids" {
  description = "Private (egress via NAT) subnet IDs, one per AZ"
  value       = aws_subnet.private[*].id
}

output "database_subnet_ids" {
  description = "Isolated subnet IDs for RDS and ElastiCache"
  value       = aws_subnet.database[*].id
}

output "private_route_table_ids" {
  description = "Route tables associated with the private subnets"
  value       = aws_route_table.private[*].id
}

Each output exists because something downstream would otherwise re-derive it badly. private_subnet_ids and database_subnet_ids are deliberately separate rather than one combined list, because a data-tier module that attaches an RDS instance to a subnet it should never reach is a mistake worth making structurally impossible rather than merely discouraged. private_route_table_ids is published because VPC-endpoint and NAT-adjacent resources must associate with the correct table, and which table is correct is a networking decision the network module already made. The CIDR decisions — how much address space each tier receives and which AZs are covered — also belong here, because a downstream module that allocates its own subnets will eventually collide with a range the network layer is already using, and the collision surfaces as a routing fault rather than a plan error.

⚠️ Common Mistake: publishing only vpc_id and one flat subnet_ids list, then letting the compute module index it as subnet_ids[0] and subnet_ids[1] to mean "the two private subnets." Reorder the AZs in the network module — a perfectly legitimate internal change — and the compute tier silently starts placing instances somewhere else. Publish tiered, named outputs and index them by meaning, not position. (The identity mechanics that keep a list stable across insertions belong to for_each over a keyed map rather than count, which is covered in "Control Flow and Lifecycle: for_each, count, Dependencies, and Meta-Arguments.")

Sourcing and Versioning: Pin Everything

Module source comes in three flavors, and each affects reproducibility differently.

The rule is simple and unforgiving: pin the module source, and pin it so a later reader can identify exactly what was consumed.

• Registry: set version = "5.21.0" for a hard pin, or a constraint like version = "~> 5.21" to permit patch and minor updates within the 5.x line.
• Git: always supply ?ref=, and point it at a tag rather than a branch. ref=v1.4.0 is reproducible; ref=main is not.
• Local: fine while the module and its callers share a repository. The moment a second repo needs it, a local path stops being a source and becomes copy-paste with extra steps.

⚠️ Common Mistake: source = "git::https://github.com/acme/terraform-aws-vpc.git" with no ?ref=. This resolves to the default branch at the moment terraform init runs. Someone merges a refactor on Monday; your unrelated staging apply on Friday plans a VPC replacement, and nothing in your repository changed. The breakage is uncorrelated with your work, which is precisely what makes it expensive to diagnose.

💡 Remember: the dependency lock file records provider selections only — Terraform does not remember module versions, so with a range constraint a fresh terraform init selects the newest matching module release; an exact version or a tag is what makes a module reproducible. terraform init -upgrade deliberately re-resolves module and provider versions. That flag is a version bump wearing a disguise: run it on a branch with a plan you actually read, never as a reflex in CI. (Provider version constraints specifically are covered in "Production Reality: Pitfalls, Guardrails, and Automated Pipelines.")

Reuse Decisions: Module, Stack, Wrap, or Fork

Two questions decide where code belongs. First: would you instantiate this more than once, or version it independently of its caller? If yes, it is a module. Second: is this composition — wiring modules together, configuring providers, setting environment-specific values? If yes, it is a stack. The healthy default is an ordinary stack: create modules only for units you can imagine a second caller of. A module invoked exactly once, by exactly one stack, is indirection with no payoff — it makes simple changes harder to follow and adds a file boundary that buys nothing.

There is one legitimate exception worth naming out loud: a boundary drawn for ownership or testing rather than reuse. If two teams need separate change control over a unit, or you want to version and test that unit on its own, splitting into a module is justified even with a single current caller. State that reason when you do it; a split with a named reason ages well, and a split without one becomes the accidental indirection above.

For consuming public modules — terraform-aws-modules and its peers — the decision is a ladder, and you should stop at the lowest rung that solves your problem:

1. 🧩 Consume directly when your requirements match the module's defaults and you only need to set documented inputs. You inherit upstream maintenance and the authors' handling of AWS quirks at no cost.
2. 🎁 Thin-wrap when your organization has a convention the module must satisfy — a naming scheme, a standard tag set, a required encryption posture. The wrapper takes your inputs, applies the convention, and delegates upward. Upstream fixes flow through on the next version bump, and every caller benefits at once.
3. 🍴 Fork only when you must change behavior the module does not expose — a resource argument it never sets, an ordering that is wrong for your workload. A fork means you now own that code's maintenance and its security patches indefinitely; a fork never re-synced with upstream is a slow-motion drift machine, and the drift is invisible until an upstream CVE forces you to look.

🎯 Key Principle: wrap before you fork. A wrapper is a handful of lines and keeps the upstream module as a live, upgradeable dependency. A fork is a standing commitment, and it should be the answer to "the module cannot do this," not "the module's inputs are mildly inconvenient."

💡 Mental Model: a stack is the call site; a module is the function. You would not extract a function called from exactly one place, with no reuse and no interface worth naming — and the same restraint applies to modules. Useful as a starting heuristic, not a rule that covers every case, since the ownership exception above is real and common in larger organizations.

Where the stack-versus-module line lands is also an operations decision, because it determines what your pipelines apply and in what order — which is why the pipeline and guardrail mechanics live in their own territory ("Production Reality: Pitfalls, Guardrails, and Automated Pipelines").

At scale, the discipline that pays is unglamorous: small contracts, pinned sources, and layers that publish exactly what the next layer needs — so that a change in one place has a boundary at which it stops.

Brownfield Reality: Importing, Refactoring, and Adopting Existing AWS

Greenfield is what you demo in an interview; brownfield is what you do on Monday. Nearly every real AWS account starts in the console, accumulates a year of aws CLI one-liners and a half-migrated CloudFormation stack, and only later grows a Terraform repo that describes a fraction of what actually exists. Adoption is the discipline of pulling those resources under Terraform's control without deleting the thing that keeps the business running.

The central tension is that Terraform wants to be the source of truth, and the account already has a truth. Adoption is the negotiation between them.

Inventory First, Import Last

The instinct when you discover terraform import is to point it at everything and let Terraform sort it out. That path ends with a plan of several hundred confusing diffs and an apply you are afraid to run. The correct order is discovery → matching → import → reconcile → expand, and the first two steps have nothing to do with Terraform.

1. Inventory   →  Resource Explorer + Resource Groups Tagging API
        ↓
2. Match       →  map real names/IDs to Terraform addresses
        ↓
3. Import      →  import block + generated configuration
        ↓
4. Reconcile   →  edit config until plan shows "0 to change"
        ↓
5. Expand      →  next tag boundary, then next tier

Inventory means asking AWS, not your memory, what exists. AWS Resource Explorer gives you a searchable cross-region index of the account; the Resource Groups Tagging API gives you a tag-filtered slice. Both are read-only, so run them freely:

# Everything tagged as part of the production payments workload, any service
aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Project,Values=payments Key=Environment,Values=prod \
  --region us-east-1 \
  --query 'ResourceTagMappingList[].ResourceARN' \
  --output table

# What VPCs exist, and are they tagged at all?
aws ec2 describe-vpcs \
  --query 'Vpcs[].{Id:VpcId,Cidr:CidrBlock,Name:Tags[?Key==`Name`]|[0].Value}' \
  --output table

Tagging is what turns a flat list of ARNs into a boundary. "The payments production workload" is a boundary you can plan, review, and apply in one sitting. "The account" is not.

🎯 Key Principle: An import records the past. Your configuration decides the future. Importing a resource does not change it — it only tells Terraform "this real object is now the thing behind address X." If your config disagrees with reality, the next apply reshapes AWS to match your config.

⚠️ Common Mistake: importing an entire account at once. Importing 400 resources produces a plan with hundreds of unrelated diffs, no reviewer can reason about which changes are safe, and a single wrong attribute turns into a production incident. Adopt in tiers — typically data layer (S3 buckets, DynamoDB tables, RDS), then network, then compute — and make terraform plan clean after each tier before moving on.

Declarative import Blocks

Modern Terraform lets you declare imports in HCL rather than issuing imperative commands. You write an import block naming the target address and the real AWS identifier:

# The `to` address does not need a resource block yet.
# Run: terraform plan -generate-config-out=generated_logs.tf
import {
  to = aws_s3_bucket.app_logs
  id = "acme-prod-app-logs"
}

The id is AWS's own identity for the object, and its format varies by resource type: a bucket uses its name, an EC2 instance uses i-..., an IAM role uses the role name, and several association resources use a slash-separated composite such as subnet_id/route_table_id. Check the provider docs for the type you are adopting — this is the single most common source of "import failed" errors.

Because the resource block does not exist yet, you pair the import block with -generate-config-out, which tells Terraform to read the live resource and write a best-effort resource block for you:

terraform plan -generate-config-out=generated_logs.tf

The generated file must not already exist. On a successful plan you will see the resource counted as an import rather than an add — the plan summary reads roughly 1 to import, 0 to add, 0 to change, 0 to destroy.

When ad-hoc terraform import is still the right tool

The imperative command has not gone away, and there are real cases where it wins:

• Fast triage. You are in a shell at 2 a.m., a resource is unmanaged, and you want it in state now without editing files. terraform import aws_s3_bucket.app_logs acme-prod-app-logs is one command.
• No plan artifact wanted. The imperative form mutates state immediately with no preview; when you have already hand-written the resource block and just need the state entry, that bluntness is convenient.
• Tooling that cannot emit import blocks. Older wrapper tooling and some generated-pipeline setups still drive imports through the CLI.

The reason the block form is preferred is not elegance — it is that the block produces a plan you can review before state changes. The imperative form does not.

💡 Pro Tip: Import blocks support for_each in recent Terraform versions, so a single block can adopt a set of instances keyed by an id map. Keep the map in a .tf file under review rather than pasting a shell loop into a terminal.

Note that once an import block has been applied successfully, it is idempotent: left in place, it does nothing on later plans. Remove the block after the apply, or keep it as a record of where the resource came from.

Reconciling Generated Configuration

Generated config is a starting draft, not a destination. It faithfully reproduces what the live resource looks like, which means it reproduces exactly the things you do not want in your codebase: literal account IDs, hard-coded ARNs, and tags written by other systems. Compare a generated block against what you would actually write:

# GENERATED — faithful to reality, wrong for a codebase
resource "aws_s3_bucket" "app_logs" {
  bucket = "acme-prod-app-logs"
  tags = {
    "Project"      = "acme"
    "LastModified" = "2024-11-02"
  }
}

# RECONCILED — references and shared locals instead of literals
resource "aws_s3_bucket" "app_logs" {
  bucket = var.logs_bucket_name
  tags   = local.common_tags
}

Three reconciliation moves cover most cases. First, replace literals with references — a subnet id becomes aws_subnet.private[0].id, a bucket name becomes a variable, and cross-resource dependencies become implicit graph edges that the generator could not see. Second, decide who owns tags written by other systems: once they are out of your block, every plan will propose removing them until you add ignore_changes for those keys. (AWS's own aws:-prefixed tags, such as the aws:cloudformation:* family, never appear here — the provider filters them out on read.) Third, decide what defaults to make explicit — the generator often omits optional attributes that are set to provider defaults, but it will faithfully include any attribute that has a non-default live value, which is how you discover that someone quietly enabled a setting in the console years ago.

⚠️ Common Mistake: applying immediately after the first import plan. The first plan is usually clean because state was just populated from live values. The second plan is where reality appears — if your reconciled configuration omits something the live resource has, that second plan proposes to remove it. Always drive the plan to "0 to change" before you consider a resource adopted.

🧠 Mnemonic: Import, reconcile, zero the diff, then expand. Skipping step three is how adoptions turn into outages.

Relocating Addresses with moved Blocks

Refactoring is the other half of brownfield work. You adopted a flat root module; now you want the resources inside module.compute. In older Terraform this meant terraform state mv — an imperative state rewrite you had to remember to run in every workspace. The declarative replacement is the moved block:

moved {
  from = aws_instance.web
  to   = module.compute.aws_instance.web
}

Nothing in AWS is changed. Terraform reads the block during plan, recognizes that the object currently at the old address is the same object that should live at the new address, and rewrites the state entry instead of destroying and recreating. The plan either shows no changes for that resource or explicitly reports the move; either way, the running instance is untouched.

moved blocks travel with your code, which is the property that matters. A teammate who pulls your branch, or a CI pipeline that applies from a fresh clone, gets the move automatically. terraform state mv does not travel — it is a state edit performed once, in one workspace, and anyone with a different state copy has to be told about it. Use moved for anything permanent; reach for state mv only for one-off local surgery.

Relative addressing and count / for_each keys

The tricky case is changing a resource's identity scheme. If you switch a resource from count to for_each, index 0 must become some map key, and Terraform cannot guess which key. There is no wildcard in moved, so you state the mapping per instance:

moved {
  from = aws_instance.web[0]
  to   = aws_instance.web["blue"]
}

moved {
  from = aws_instance.web[1]
  to   = aws_instance.web["green"]
}

Without these blocks, switching the resource to for_each means Terraform sees index 0 disappear and key "blue" appear — two unrelated changes — and it will destroy one instance and create another. With the moves declared, both instances survive the refactor with new addresses and no downtime.

⚠️ Common Mistake: deleting the moved block as soon as your local apply succeeds. Other workspaces, other branches, and any teammate whose state has not yet seen the move still need it. Keep moved blocks in the codebase permanently, or remove them only after you can confirm every state copy has applied past them. They cost nothing at plan time once the move has been recorded.

One limit worth knowing: a moved block can rename an address and relocate it into a module, but it cannot turn a managed resource into a data source, and it can change the type only for the pairs a provider explicitly supports (Terraform 1.8 and later). Rewriting an aws_instance as an aws_launch_template, or moving a resource from the default provider to an aliased one, is not a move — it is a create-and-destroy, and it needs a migration plan, not a block.

When AWS Changes Things Behind Your Back

Every plan begins with a refresh: Terraform calls read APIs for each managed resource and compares the live values against what state recorded. For adopted resources, this is where you meet the other controllers already operating in your account.

A short tour of the usual suspects:

• Auto Scaling owns desired_capacity on an ASG. Scaling policies or a manual console change move it; your next plan proposes to set it back to whatever is in your config. If scaling is supposed to own that number, Terraform must stop managing it.
• IAM grows attachments outside your repo. A console operator attaches a managed policy to a role you manage, and the next plan removes it — because from Terraform's perspective, the live state no longer matches your declared set.
• Tag writers. EKS, cost tooling, and security scanners add their own tags. Plans that propose deleting those tags are pure noise and will recur forever unless you handle them. (Tags with the reserved aws: prefix, such as CloudFormation's, are the exception: the provider ignores them.)
• Console edits. An S3 bucket policy tightened by hand, an RDS parameter group tweaked during an incident, deletion_protection toggled off — all of these surface as drift on the next refresh.

🎯 Key Principle: for each attribute of an adopted resource, decide which system is authoritative. If Auto Scaling is authoritative for capacity, your config is authoritative for the launch template and the min/max bounds. If a third party (an EKS add-on manager, a CloudFormation stack that still exists) is authoritative for a resource, do not adopt that resource at all.

⚠️ Common Mistake: adopting resources that another controller will keep mutating. CloudFormation-managed resources and resources managed by platform add-ons are the classic case — you end up arm-wrestling the other controller, with each system reverting the other's changes on every apply. Ownership must be exclusive, or the resource should stay unmanaged.

The adoption-specific toolkit here is narrow: lifecycle { ignore_changes = [...] } for the individual attributes another controller owns, and a deliberate decision to leave a resource out of scope. Full ongoing drift detection, scheduled refresh, and remediation workflows are their own topic and are covered in the dedicated drift lesson — here, the point is only that you should expect drift on your first refresh of an adopted resource and plan for it rather than being surprised.

Safe Teardown

Adoption gets the attention; teardown is where the money and the data are lost. Three tools, three different jobs.

prevent_destroy is a lifecycle meta-argument that makes plan and apply fail if the resource would be destroyed — including when a forced replacement would destroy and recreate it:

resource "aws_db_instance" "orders" {
  identifier          = "acme-orders-prod"
  deletion_protection = true
  instance_class      = "db.r6g.large"
  # ...

  lifecycle {
    prevent_destroy = true
  }
}

Its limits matter as much as its protection. prevent_destroy does not stop terraform state rm, does not stop someone deleting the database in the console, and does not stop a different stack from deleting it. It is a seatbelt, not a lock. It also makes terraform destroy on the whole stack fail — which is the point, because you now have to edit the code deliberately before you can wipe an environment.

Removing a block from config versus terraform state rm is the distinction that burns teams:

The safe way to retire a resource while keeping it alive in AWS is state-first:

Keep it in AWS:
  terraform state rm <addr>  →  delete the block  →  plan: no change

Delete it:
  delete the block  →  plan: "1 to destroy"  →  apply  →  AWS deleted

Since Terraform 1.7 the same retirement can be declared in code, which gives you a plan to review, like import and moved. Replace the resource block with a removed block:

removed {
  from = aws_s3_bucket.app_logs

  lifecycle {
    destroy = false
  }
}

The next apply drops the object from state and leaves it running in AWS.

⚠️ Common Mistake: deleting the resource block first and expecting "no plan" because you "didn't run apply." The block is gone but the resource is still in state, so plan reports a destroy, and the next person who runs apply — often a CI pipeline — deletes the live object. Run terraform state rm first, then remove the configuration, and confirm the plan no longer mentions the address.

Finding orphans before you destroy a stack

Before terraform destroy, or before you believe a stack is fully gone, compare three sources:

1. terraform state list — everything Terraform believes it owns. This is your authoritative inventory of managed resources.
2. A tag-based sweep — the same Resource Groups Tagging API query you used for discovery, re-run after the destroy. Anything still tagged with the project boundary but no longer in state is an orphan.
3. AWS-side deletion blockers — S3 buckets must be emptied (or carry force_destroy), deletion_protection must be off on databases, and networking resources with lingering ENIs will fail to delete.

Orphans come from three predictable places: resources that were state rm'd earlier and forgotten; resources with a retain policy on the CloudFormation side of a hybrid stack; and child resources created by AWS itself (log groups, default security groups, ENIs) that were never in your configuration to begin with.

💡 Real-World Example: the textbook failure is tearing down a Terraform-managed environment and abandoning an S3 bucket that was excluded from the state years earlier, or that the delete failed on because it was not empty. The VPC vanishes, the compute vanishes, and the bucket quietly bills you for storage and requests for the next two years because nothing in the repo pointed at it anymore. The post-destroy tag sweep is what catches this.

Adoption, refactoring, and teardown are the same skill applied at different moments — deciding what Terraform owns, proving the ownership is real with a clean plan, and never letting a code change silently become an API deletion; the guardrails that catch these mistakes automatically in CI are the subject of the next section.

Production Reality: Pitfalls, Guardrails, and Automated Pipelines

A configuration that applies cleanly from one engineer's laptop has proved almost nothing. Production Terraform adds a second engineer, a second AWS account, a second year of accumulated decisions, and a CI runner that has no memory of what any of it meant. The failures below are boring, repeatable, and each one has a pipeline check that costs seconds to run and prevents a rollback that costs an evening.

The Five Pitfalls That Actually Reach Production

Pitfall 1: Identifiers frozen into the code

A hard-coded AMI ID or a bare 12-digit account number is correct in exactly one account at exactly one point in time. Code review rarely catches it, because the plan looks healthy — the value resolves fine in the account where it was written. The failure appears later, when the same module is called in another account: it either dies at apply, or worse, succeeds against a stale image nobody intended.

You do not need a policy engine for this. A regex gate does the job:

#!/usr/bin/env bash
# scripts/hardcoded-ids.sh — fail on identifiers that only work in one account
set -euo pipefail

# Matches bare AMI IDs and ARNs carrying a literal 12-digit account number.
# Exempt an intentional line by ending it with the comment: # allow-hardcoded
matches=$(grep -REn --include='*.tf' \
  '(ami-[0-9a-f]{8,17}|arn:aws:[a-z0-9-]+:[a-z0-9-]*:[0-9]{12}:)' infra/ \
  | grep -v 'allow-hardcoded' || true)

if [ -n "$matches" ]; then
  echo "$matches"
  echo "::error::hard-coded AWS identifier — use a variable or data source"
  exit 1
fi
echo "OK: no hard-coded AWS identifiers"

The allow-hardcoded escape hatch is the part people skip, and it is why these scripts get deleted three weeks after they are written. A check with no documented exemption path will be routed around; a check with a greppable exemption comment survives, because reviewers can see why a line is exempt in the diff.

The second pattern the regex catches — an ARN with a literal account number — is the more dangerous one, because account IDs sneak into IAM trust policies and KMS key policies where they read like configuration rather than hard-coding. (Section 2 covered replacing these with aws_ami and aws_caller_identity data sources and input variables. The pipeline check exists because review is not a reliable substitute for a rule that never gets tired.)

Pitfall 2: Secrets passed as variables and persisted in state

sensitive = true on a variable redacts the value in CLI output. It does not remove the value from state, and it does not remove it from a saved plan file. Your database password therefore lives in at least four places: the HCL, the state object, the plan artifact, and the CI log of whoever once ran terraform show to debug something.

// state, abridged to one resource — the password is plaintext
{
  "type": "aws_db_instance",
  "name": "orders",
  "instances": [{
    "attributes": {
      "username": "app",
      "password": "correct-horse-battery-staple",
      "manage_master_user_password": false
    }
  }]
}

⚠️ Common Mistake: answering a secrets finding with "the variable is marked sensitive." Sensitivity controls display. Storage is a different problem with a different fix.

Three fixes, in order of preference:

• 🔧 Let the provider own the secret. Setting manage_master_user_password = true on RDS has RDS generate the password, store it in Secrets Manager, and rotate it; your state holds a secret ARN, not a password, and your config holds neither.
• 🔧 Use write-only arguments or ephemeral values where your Terraform and provider versions expose them. That class of argument exists precisely to keep a value out of state; older versions simply do not have it, which is one more reason the version pins below matter.
• 🔧 Generate at runtime. The compute layer fetches from Secrets Manager at boot using an IAM-scoped role (the workload-identity patterns from Section 1). Terraform never sees the value at all.

Whichever you pick, treat every artifact accordingly: KMS-encrypted backend, a bucket policy that does not hand out s3:*, and a CI artifact retention measured in days rather than the default ninety. A plan file is as sensitive as state, because it contains the same shape of data.

Pitfall 3: The implicit default provider

A provider block with no region argument, or a child module that declares required_providers but no configuration, silently inherits whatever the root module configured — and if the root did not configure anything, the AWS provider resolves the region from the environment: AWS_REGION, AWS_DEFAULT_REGION, a shared config profile, or instance metadata. On a workstation that is usually fine. On a CI runner it means your infrastructure's target region is decided by an environment variable someone set for an unrelated reason.

The characteristic symptom is a plan where everything wants to be replaced, because Terraform is diffing the state in eu-west-1 against a desired state resolved in us-east-1.

# versions.tf — the boring file that prevents most cross-environment incidents
terraform {
  required_version = "~> 1.9.0"          # patch upgrades only

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"                # major pinned; minor floats inside it
    }
  }
}

provider "aws" {
  region = var.region                    # declared in code, never inferred at apply time

  default_tags {
    tags = local.common_tags
  }
}

# A second account or region is an alias passed explicitly to the module call.
provider "aws" {
  alias  = "workload"
  region = var.workload_region
}

✅ Correct thinking: region and identity are inputs. They are declared, reviewed, and versioned like any other input.

❌ Wrong thinking: "It works on my machine, so the provider config is fine." Your machine has an SSO profile with a default region. The runner has whatever the previous step exported.

Pitfall 4: Missing version constraints

With no required_version and no required_providers constraint, terraform init resolves to whatever the registry offers that day. Your laptop, a colleague's laptop, and CI can end up running three different provider versions against the same state file — and provider majors change defaults. The AWS provider's v4- and v5-era releases are the canonical example: bucket settings that used to live as arguments on aws_s3_bucket moved out into dedicated resources, so a floating major version can rewrite your S3 configuration underneath you during an otherwise unrelated change.

Three pins, all boring, all mandatory:

• 🎯 Constraint in code — required_version plus required_providers (shown above). ~> 5.60 accepts 5.61 and 5.99 but refuses 6.0, which is the trade-off you want: bug fixes arrive, behavior changes require a deliberate commit.
• 🎯 Commit the lock file — .terraform.lock.hcl records exact provider versions and per-platform hashes. Generate hashes for every platform that runs Terraform: terraform providers lock -platform=linux_amd64 -platform=darwin_arm64. CI on a platform absent from the lock file will either fail outright or re-resolve, which is the failure you were trying to prevent.
• 🎯 Pin the CLI in CI — terraform_version: ${{ vars.TF_VERSION }} from a repository variable, not latest. Floating the CLI means your pipeline can adopt a release with new validation rules on a random Tuesday.

Unpinned module sources are the same disease at the module boundary; the sourcing and semver-pinning mechanics are covered in "Composing AWS at Scale with Modules."

Pitfall 5: count-index churn, now with approval fatigue

Section 3 analyzed why count shifts identity when an element is deleted, and for_each does not. In production that becomes a diff where removing one subnet from a list proposes replacing eight unrelated resources, because their indices all shifted. The identity damage is the visible half. The invisible half is what it does to review culture: once a team sees three 400-line destroy-and-recreate diffs that were fine, they stop reading 400-line destroy-and-recreate diffs. The next one is the incident.

The pipeline fix is a hard number, not a guideline: any plan with a destroy count above zero fails the automated gate and demands a named second approver. That keeps the scary diff rare and therefore read.

Guardrails: Cheap Checks First, Policy Last

Runs in layers, ordered by cost. A check that takes two seconds belongs ahead of one that takes four minutes, so the fast failure is the one developers see.

The most important row is the last one, and the reason is a limitation worth stating plainly: HCL-level scanners cannot see what this run's plan resolved. They evaluate what the configuration itself gives them — variable defaults, a tfvars file if you pass one — but not the values your pipeline actually supplies or anything a data source returns, so they can see var.instance_type's default rather than m5.large, and can flag code inside a count = 0 branch as if it were live. That produces false positives, and a guardrail with false positives gets disabled. Plan-based policy runs against terraform show -json output, where variables, data-source lookups, and conditionals have already been resolved to concrete values — which is why the two layers are complements, not alternatives. (Aqua folded the tfsec checks into Trivy; pipelines that used to run tfsec now commonly run Trivy with the Terraform and misconfiguration checks enabled.)

Pin the linter the way you pin providers, because a ruleset upgrade is a change to which failures block your build:

# .tflint.hcl — the lint ruleset is a dependency; treat it like one
plugin "aws" {
  enabled = true
  version = "0.35.0"        # pin: a ruleset upgrade adds new failing rules
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}

rule "aws_instance_invalid_type" {
  enabled = true
}

rule "terraform_deprecated_interpolation" {
  enabled = true
}

Run tflint --init once per CI job to fetch the plugin, then tflint in each stack directory. Provider-aware rules like aws_instance_invalid_type are the reason tflint earns its place next to the generic scanners: it checks the value against the list of valid EC2 instance types built into the ruleset.

Generic scanners encode the world's rules. Policy-as-code encodes what only your organization knows — your tag schema, your approved regions, your tolerance for destruction. A worked policy file, in Rego v1 syntax (OPA 1.x, the syntax current Conftest versions expect):

# policy/org_rules.rego — evaluated against plan JSON, not HCL
package terraform.org

required_tags := {"Owner", "Environment", "CostCenter"}

# 1) Generic misconfig, phrased as an org rule: no public ACLs, ever.
deny contains msg if {
  rc := input.resource_changes[_]
  rc.type == "aws_s3_bucket_public_access_block"
  rc.change.after.block_public_acls != true
  msg := sprintf("%s: block_public_acls must be true", [rc.address])
}

# 2) Anything created here must carry the ownership tags.
deny contains msg if {
  rc := input.resource_changes[_]
  rc.change.actions != ["no-op"]
  rc.change.after != null
  tags := object.get(rc.change.after, "tags", {})
  missing := [t | t := required_tags[_]; not tags[t]]
  count(missing) > 0
  msg := sprintf("%s: missing required tags %v", [rc.address, missing])
}

# wire it into the job, after the plan is written
terraform show -json tfplan > plan.json
conftest test plan.json            # reads ./policy by default

🤔 Did you know? Conftest is the CLI wrapper around OPA — the same Rego policy can be evaluated by the opa binary locally, by Conftest in CI, or pushed to a policy service. Sentinel, by contrast, is specific to HCP Terraform and Terraform Enterprise; the Rego route is what you use when your pipeline is GitHub Actions and your state lives in your own S3 bucket.

One honest limit: IAM policy wildcards are the rule people most want to write here, and plan JSON is a poor place for them, because the provider normalizes policy JSON between configuration and state, so exact substring matching is brittle. Catch Action: "*" on the HCL side with Checkov's IAM checks or by using the aws_iam_policy_document data source from Section 2, and save Rego for structural properties — tags, public-access flags, allowed regions — that survive normalization unchanged.

The Pipeline: Plan on PR, Apply That Exact Plan on Merge

PR opened
   ↓
fmt -check → validate → tflint → checkov/trivy        (fast checks first)
   ↓
terraform plan -out=tfplan  →  terraform show -json tfplan > plan.json
   ↓
policy gate (conftest) + destructive gate + cost delta ← merge blocked here
   ↓
tfplan artifact (encrypted, short retention) + human review of the plan text
   ↓
merge to main
   ↓
apply job: OIDC → write role → terraform apply tfplan
   ↓
post-apply: tag sweep, cost report, scheduled drift run

Five design rules make that flow safe, and each one maps to a specific failure it prevents.

🎯 Rule 1: The plan file is the contract. Apply the artifact, never re-plan. A re-planned apply runs against a tree that may have changed since review, which means the approver approved something else. A saved plan also freezes the provider versions and the resource graph the reviewer actually saw.

🎯 Rule 2: The plan file is a secret. It contains the same class of values as state. Encrypt it in the artifact store, cap retention, and delete it after apply.

🎯 Rule 3: No long-lived keys in CI. Use OIDC: GitHub mints a short-lived token, AWS STS exchanges it for a role session. No AWS_ACCESS_KEY_ID in repository secrets, nothing to rotate, nothing to leak.

🎯 Rule 4: Separate read and write roles. The plan job assumes a role that can Describe/List but not mutate; the apply job assumes the write role. A malicious pull request that modifies the workflow can, at worst, read your account.

🎯 Rule 5: Destructive diffs need a human. Environment protection rules provide the approval; the destructive gate decides when it fires.

name: terraform-prod

on:
  pull_request:
    paths: ["infra/**"]
  push:
    branches: [main]
    paths: ["infra/**"]

permissions:
  id-token: write        # required for OIDC — replaces stored AWS keys
  contents: read
  pull-requests: write   # lets the job comment the rendered plan

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/envs/prod
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ vars.TF_VERSION }}   # pinned, never "latest"

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/gha-tf-plan   # read-only
          aws-region: eu-west-1

      - run: terraform fmt -check -recursive
      - run: terraform init -backend-config=backend.hcl -lock=false
      - run: terraform validate
      - run: tflint --init && tflint

      - run: terraform plan -lock=false -out=tfplan
      - run: terraform show -json tfplan > plan.json
      - run: conftest test plan.json
      - run: ./scripts/destructive-gate.sh plan.json

      - uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: infra/envs/prod/tfplan
          retention-days: 3    # the plan holds secrets — keep it short-lived

  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: prod      # environment protection rule = required human approval
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ vars.TF_VERSION }}

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/gha-tf-apply  # write role
          aws-region: eu-west-1

      - uses: actions/download-artifact@v4
        with:
          name: tfplan
          path: infra/envs/prod

      - name: Apply the reviewed plan, not a fresh one
        working-directory: infra/envs/prod
        run: |
          terraform init -backend-config=backend.hcl
          terraform apply -auto-approve tfplan

Two practical notes on that workflow. First, fmt and tflint here only cover infra/envs/prod because of the working directory — in a real repository you run those two in a separate, fast job from the repo root so they cover every stack. Second, the apply job cannot download the artifact without the originating run id; use actions/download-artifact with run-id and a token, or trigger the apply workflow from a workflow_run event on the plan workflow. Cross-run artifact retrieval is the fiddly part of this design, not the Terraform.

The destructive gate is a few lines of jq against the plan JSON:

#!/usr/bin/env bash
# scripts/destructive-gate.sh — block merges that destroy or replace resources
set -euo pipefail
PLAN_JSON="$1"

# Terraform records actions as an array: ["delete"], ["create"], and for a replace
# ["delete","create"] or ["create","delete"]. Anything containing "delete" destroys an object.
DESTROYS=$(jq '[.resource_changes[]? | select(.change.actions | index("delete"))] | length' "$PLAN_JSON")
REPLACES=$(jq '[.resource_changes[]? | select(.change.actions | index("delete") and index("create"))] | length' "$PLAN_JSON")

echo "destroys=$DESTROYS (of which replaces=$REPLACES)"
if [ "$DESTROYS" -gt 0 ]; then
  echo "::error::$DESTROYS resource(s) would be destroyed or replaced — needs a second approver"
  exit 1
fi

For the approval to be meaningful, the approver has to see the diff. Render it with terraform show -no-color tfplan and post it as a PR comment, truncating the middle if it is long — a reviewer approving a job name rather than a resource list is not reviewing anything.

Cost-Aware Review and Ownership From Day One

Every plan answers two questions that used to surface a month later on an invoice: what does this change to the monthly bill, and who owns it? Any tool that reads plan JSON can post a cost delta as a PR comment — Infracost is the common one — and the useful discipline is to make the delta visible before it becomes a blocker. Show it for a few sprints, learn which deltas are noise, then fail the build on changes above a threshold you chose deliberately.

The ownership half is the one teams regret skipping. Tagging via the provider's default_tags (Section 1) covers most taggable resources automatically, which is exactly why the exceptions matter: some resource types accept no tags at all, and some tags written by other tools will fight your configuration until you add ignore_changes for them (Section 3).

⚠️ Common Mistake: "We will add tags in the next sprint." Cost allocation tags must be activated in the Billing console before they appear in cost reports, and activation is not retroactive on its own — you have to request a backfill (up to twelve months), and it only covers resources that already carried the tag at the time. A fleet of instances that ran untagged for its first month is permanently unattributable. A backfill cannot invent a tag that was never there.

The enforcement mechanism is the tag rule in the Rego policy above, running against plan JSON. That same schema then serves three other purposes for free: AWS Config required-tag rules, IAM conditions that restrict who can modify tagged resources, and Resource Groups for operations. One schema, four consumers — which is why it belongs in the pre-apply gate rather than in a wiki page.

Pre-Apply and Post-Apply Checklist

Use this as the pull-request template so the review is enforced by the process rather than by memory. It doubles as the quick reference for everything in this section.

Two entries deserve a comment. "Immediate re-plan is empty" is the fastest drift detector you have: run a plan right after a successful apply, and if anything wants to change, either AWS is mutating something you must ignore_changes or your configuration states a value differently than the API normalizes it. "Scheduled drift run booked" matters because the pipeline only protects what flows through it — the console-created resource from a bad night and the CloudFormation stack someone else owns only surface on a schedule. The mechanics of detecting and remediating that are covered in the dedicated state and drift material referenced earlier.

⚠️ Remember these three:

• 🔒 State and plan files are secrets. Every variable you pass is persisted in both. Marking a variable sensitive changes nothing about storage.
• 🔒 Every unpinned thing is a silent upgrade path. Provider majors, module sources, the Terraform CLI in CI, and your lint ruleset all change behavior between runs if you let them float.
• 🔒 Guardrails belong in the pipeline, not in documentation. A rule that is not a failing check is a preference.

Three things to do next, in the order that pays back fastest:

1. Add the three-cheap-checks job today. fmt -check -recursive, terraform validate after init, and tflint --init && tflint. No policy engine, no approval workflow, maybe thirty minutes of work — and it stops the identifier and formatting classes of defect entirely.
2. Convert CI credentials to OIDC with two roles. One read-only role for plan, one write role for apply, both trusted only for your repository and branch. This removes the single highest-value static credential in most AWS accounts.
3. Add the destructive gate and the environment approval before your next production apply. Then extend the same pipeline with the tag policy and the cost delta once the basics are green — the incremental guardrails are easy to add to a pipeline that already runs, and nearly impossible to retrofit onto one that does not exist.