AWS Foundations & Core Services

📹Watch the Lesson Video28 min

AWS as an API-First Platform: Control Plane, Data Plane, and Resource Identity

You've probably already hit this one: a script creates an S3 bucket just fine, and then the very next call — the one that uploads a file — fails with AccessDenied, even though "S3 is allowed" in the policy. Or a launch command works in eu-west-1, and then the next script can't find the instance, because the resource lives somewhere else. Neither failure means IAM or the network is broken. Both are symptoms of not yet having a model of what an AWS call actually is.

Every AWS action — a console click, an aws CLI command, a terraform apply, a boto3 call — resolves to the same thing: a signed HTTPS request to a service endpoint, naming a specific resource with a specific identity. The console is just another client of the API you call from a terminal. Once you can read any call as (operation, endpoint, resource identity, credentials), most confusing AWS errors become diagnosable, and the ones that aren't tell you exactly where to look next.

This section builds that reading skill around five ideas: the split between control plane and data plane operations, the anatomy of an ARN, how endpoints get selected, how resources are identified, and the API mechanics — signing, idempotency, pagination, and quotas — that decide whether your automation survives contact with production.

Control Plane vs Data Plane: Configure vs Serve

Control plane operations create, configure, or delete resources. They change how a system is shaped: CreateBucket, RunInstances, CreateFunction, CreateTable, AuthorizeSecurityGroupIngress. The verbs are management verbs, and they run at human or automation speed — dozens to thousands of calls.

Data plane operations serve the workload. They move bytes, run code, and read or write data: GetObject and PutObject on S3, Invoke on Lambda, GetItem and Query on DynamoDB, and all the ordinary TCP traffic reaching an Application Load Balancer. Data plane traffic runs at application speed — potentially millions of calls per minute.

CONTROL PLANE — configure
  CreateBucket, RunInstances, CreateFunction, CreateTable
  ↓
  Low volume · rate-limited · throttled
  ↓
  Typical failure: ThrottlingException / RequestLimitExceeded

DATA PLANE — serve
  GetObject/PutObject, Invoke, GetItem/Query, app traffic to an ALB
  ↓
  High volume · designed to scale with load
  ↓
  Typical failure: 503 SlowDown, connection timeouts

🧠 Mnemonic: Control = Configure, Data = Deliver.

The split matters because three things segment along it.

1. Permissions. IAM policies are written against individual actions, so a workload's role is typically granted data plane access — s3:GetObject and s3:PutObject on arn:aws:s3:::my-bucket/* — and not s3:CreateBucket. "Can this app read the file?" and "can this app create a bucket?" are two different grants, and answering one tells you nothing about the other. The policy language itself belongs to the IAM lesson; what matters here is that the plane boundary is where you draw the permission boundary.
2. Failure modes. Control plane calls are low-volume and rate-limited, so they typically fail as ThrottlingException or, on EC2, RequestLimitExceeded. Data plane calls are built to scale with load, so they fail as 503 SlowDown, connection timeouts, or elevated latency. When you see a throttle on a management API, you learned something structural: some automation is hammering management APIs, not user traffic.
3. Blast radius. If a control plane is impaired, you can't launch new capacity — but the capacity already running keeps serving. That asymmetry is why "go edit the infrastructure during an incident" is a design smell: the fix path and the failure path compete for the same throttled surface.

💡 Real-World Example: An autoscaling group reacts to a traffic spike by calling RunInstances — a control plane operation — while existing instances absorb user requests on the data plane. If the control plane throttles, scale-out stalls but users are still served. The incident is real, the outage isn't yet — and the clock is running.

💡 Pro Tip: When a permission "works in the console but not in code," your first check is which plane you're exercising. A human with broad console permissions silently covers control plane calls that an application role was never granted.

🎯 Key Principle: Manage the control plane deliberately — it is throttled, permission-scoped, and separate from the traffic path you are trying to keep alive.

ARNs: The Address of a Resource

An ARN (Amazon Resource Name) is how AWS names a specific resource unambiguously. You see it in IAM policy Resource fields, in CloudTrail events, in EventBridge rule targets, in SNS subscriptions, in KMS key references. Its shape is fixed:

arn:partition:service:region:account-id:resource

Six fields, separated by colons. Here is the reason people misparse them: fields are frequently empty, but the colons are not.

arn:aws:s3:::my-bucket/reports/summary.csv
    aws  s3  (no region)  (no account)  my-bucket/reports/summary.csv

arn:aws:iam::123456789012:user/alice
    aws  iam (no region)  123456789012  user/alice

arn:aws:ec2:eu-west-1:123456789012:instance/i-0abc123def456
    aws  ec2  eu-west-1   123456789012  instance/i-0abc123def456

arn:aws:states:us-east-1:123456789012:execution:orderFlow:order-4821
    aws  states  us-east-1 123456789012  execution:orderFlow:order-4821

The partition is the field beginners ignore and then get burned by: hardcoding arn:aws:... breaks in GovCloud or China partitions. In Python, don't hardcode it — read it from the caller's own ARN:

import boto3

arn = boto3.client('sts').get_caller_identity()['Arn']
print(arn.split(':', 5)[1])
# 'aws' in the standard Regions — the partition of your credentials may not be 'aws'

A naive parser is a classic mistake. Splitting on every colon destroys ARNs whose resource portion itself contains colons (see the Step Functions execution above).

def parse_arn(arn: str) -> dict:
    """Split an ARN into its six fields.

    maxsplit=5 keeps everything after the fifth colon inside 'resource',
    which is required for ARNs like:
      arn:aws:states:us-east-1:123456789012:execution:orderFlow:order-4821
    """
    parts = arn.split(':', 5)
    if len(parts) != 6 or parts[0] != 'arn':
        raise ValueError(f'not an ARN: {arn}')
    fields = ['arn', 'partition', 'service', 'region', 'account_id', 'resource']
    return dict(zip(fields, parts))


parsed = parse_arn('arn:aws:ec2:eu-west-1:123456789012:instance/i-0abc123')
print(parsed['region'])     # eu-west-1
print(parsed['account_id']) # 123456789012

⚠️ Common Mistake: arn.split(':') without maxsplit. It works on your EC2 test ARN and silently corrupts any ARN with colons inside the resource — including log group ARNs and Step Functions executions.

Where ARNs show up in practice, and what each sighting means:

• 🧠 Policies: the Resource element is a list of ARNs, with wildcards. arn:aws:s3:::my-bucket/* matches objects but not the bucket itself; arn:aws:s3:::my-bucket matches the bucket but not its contents. Both lines are usually needed.
• 🔍 Audit trails: CloudTrail records userIdentity.arn (who called) and resources[].ARN (what was touched). Reading a trail starts with reading ARNs.
• 🔗 Cross-service wiring: an EventBridge target, a Lambda environment variable holding a table name versus a table ARN, an SNS subscription — all carry identity across service boundaries. This is the same contract idea that resurfaces when we compose services in "Composing Core Services: A Minimal Workload Walkthrough."

🤔 Did you know? The S3 ARN has both region and account empty — arn:aws:s3:::bucket. That's not a documentation shortcut: S3 bucket names are unique across every account and Region of a partition, so region and account are redundant to the identity. EC2, whose instance IDs are unique only within a region, needs every field.

Endpoints: Where the Call Actually Lands

Once an operation and a resource are named, the request still needs a destination. That destination is the endpoint, and its default shape is predictable: https://{service}.{region}.amazonaws.com. ec2.eu-west-1.amazonaws.com. dynamodb.us-east-1.amazonaws.com.

This is where the target stack matters. Both the AWS CLI v2 and boto3 resolve the region for you, and both produce a different class of error if they can't: CLI v2 stops with You must specify a region, and boto3 raises botocore.exceptions.NoRegionError with the message You must specify a region. If you have ever moved a boto3 client creation from a laptop into a CI job and gotten that exception, this is why: the laptop had a default region in ~/.aws/config, the job's environment did not. (A Lambda function does not hit this: its runtime sets AWS_REGION and AWS_DEFAULT_REGION for you.) The precedence rules for how region and credentials resolve — CLI flags, environment variables, profiles, instance roles — are the subject of "Programmatic Access in Practice: CLI, SDK, and IaC Entry Points." Here, the point is narrower and more useful: whatever wins that resolution becomes an endpoint, and the endpoint determines which copy of the service you are talking to.

Global services break the {service}.{region} pattern:

For IAM, this has a consequence people find surprising: passing --region eu-west-1 does not create a "European" IAM user. The user is account-global — there is exactly one alice in your account, and arn:aws:iam::123456789012:user/alice has no region field because there is no region to put there. Route 53 is the same shape of oddity: a global service whose whole purpose is to answer questions about regional and global endpoints.

S3 is the interesting middle case. Buckets live in a region, but the service accepts addressing that doesn't name that region. When you ask for a bucket in the wrong region, S3 responds with a redirect and a region hint, and botocore follows it — which is why aws s3 ls often "just works" without you specifying the bucket's region. That convenience quietly becomes a cost and latency problem later; the failure mode of getting it wrong shows up in "Common Pitfalls and a Foundation Readiness Checklist."

Finally, the plain endpoint is not the only endpoint. There are FIPS variants, dual-stack variants, Local Zone and Outpost endpoints, and — most importantly for private networking — VPC endpoints, which keep traffic to S3 or DynamoDB off the public internet entirely. Those are covered in detail in the VPC lesson; what belongs in your mental model now is that the endpoint is a choice your architecture makes, not a fixed constant.

💡 Pro Tip: A signature mismatch that looks like an authentication failure is often a region mismatch. SigV4 signs the region and service into the credential scope, so signing for us-east-1 while sending to eu-west-1 produces a signature error — not a permissions error. If your credentials are provably fine, check the endpoint next.

Resource Identity: Names vs IDs vs Tags

The practical question this whole section answers is: what string do I put in my script? AWS gives you three kinds of answer, and they are not interchangeable.

• 📛 Names — human-chosen strings: an S3 bucket name, an IAM role name, a Lambda function name, a DynamoDB table name. Some are globally unique (buckets), some account-wide (IAM roles), some per-region (Lambda function names).
• 🆔 IDs — service-assigned, immutable strings: i-0abc123def456, vpc-0a1b2c3d, vol-0f9e..., subnet-04d1.... Not friendly, but stable.
• 🏷️ ARNs — fully qualified identity, composing account, region, service, and the name or ID.

The EC2 case teaches this in one shot. An EC2 instance has no name. The console displays a "Name" column, but that value is literally a tag — Key=Name — that you or your launch template set. The identity is the instance ID. So a script like aws ec2 terminate-instances --instance-ids web-01 fails, and it always will: the API has never heard of web-01.

import boto3

ec2 = boto3.client('ec2', region_name='eu-west-1')

# 'Name' is a tag, not a field. Resolve it to an ID once, then use the ID.
response = ec2.describe_instances(
    Filters=[{'Name': 'tag:Name', 'Values': ['web-01']}]
)

instance_ids = [
    instance['InstanceId']
    for reservation in response['Reservations']
    for instance in reservation['Instances']
]

print(instance_ids)  # ['i-0abc123def456'] — pass these IDs everywhere downstream

(That call paginates — a detail we get to below, and a pattern you'll meet again in "Programmatic Access in Practice: CLI, SDK, and IaC Entry Points." Simplified here by omitting the paginator for readability.)

Tags are metadata, not identity. They are mutable, non-unique, and any number of resources can share them. That's a feature — tags are how you group, attribute, and automate — and a trap when you treat them as a key. Their role in ownership and cost attribution belongs to "Guardrails for Foundations: Quotas, Tagging, Cost Signals, and Blast Radius." Here, the rule that matters is narrower: tags are excellent for finding a resource and dangerous as the only thing you remember about it.

🎯 Key Principle: Pick the identity that survives a rename. Names change, regions change, accounts get rebuilt — a service-assigned ID does not. An ARN is only as stable as the name inside it: rename an IAM user and its ARN changes, while its unique ID stays the same.

In practice, that means: resolve names to IDs at runtime rather than baking names into deployment scripts; discover your account ID from the caller identity instead of hardcoding it; and prefer ARNs when you're handing a resource reference to another service. A script that hardcodes 123456789012 works perfectly until the day someone runs it against the staging account.

API Mechanics That Matter in Automation

Four mechanics separate scripts that work in a demo from scripts that work at 3 a.m.

1. Signed requests (SigV4). Every call is signed with your credentials, and the signing scope includes the date, region, and service. The AWS CLI v2 and boto3 do this automatically, which is exactly why it's worth knowing what it implies: a clock that is too far off gets the request rejected (on EC2, RequestExpired once the time stamp is more than 15 minutes out); signing for one region and sending to another produces a signature mismatch; and a pre-signed URL is just a signature with an embedded expiry, which is why a pre-signed URL stops working when it expires, and sooner than that if the credentials that signed it expire or are revoked.

2. Idempotency tokens. Networks retry. If a create call is retried after a timeout, you can end up with two instances. EC2's RunInstances accepts a ClientToken: resend the same token and you get the same reservation back instead of a second instance. Not every API supports a token, and the fallback differs per service — S3's CreateBucket, for example, is not uniformly "retry returns 200": outside us-east-1, repeating it returns BucketAlreadyOwnedByYou, which your automation must treat as success rather than an error (in us-east-1 the repeat returns 200 for legacy reasons, and resets the bucket's ACLs). When no token exists, idempotency comes from deterministic naming plus tags, a pattern developed further in "Programmatic Access in Practice" and "Guardrails for Foundations."

3. Pagination. List and describe calls return a page plus a continuation token — NextToken, Marker, ContinuationToken, depending on the service — and a truncation flag such as IsTruncated. One call is never "all resources." This is one of the most common silent script bugs in AWS: a fleet-wide cleanup script passes every test in a dev account with three instances and then misses a large fraction of a production account with hundreds, because nobody followed the token.

⚠️ Common Mistake: Treating a single describe call as a complete inventory. The script doesn't error; it just does less than you think. Always iterate pages.

4. Quotas and throttling. Two families: request-rate limits (throttling — ThrottlingException, HTTP 429, RequestLimitExceeded) and resource-count limits. boto3 and the CLI retry throttling automatically with exponential backoff and jitter, but the default retry budget is tuned for interactive use, not for a batch job walking a large inventory. Raise it deliberately and say so in code:

import boto3
from botocore.config import Config

# Default retries suit interactive calls. A batch job needs a bigger budget.
config = Config(
    retries={'max_attempts': 10, 'mode': 'standard'},
    connect_timeout=5,
    read_timeout=30,
)

ec2 = boto3.client('ec2', region_name='eu-west-1', config=config)

# Idempotent create: the same ClientToken on a retry returns the same reservation.
reservation = ec2.run_instances(
    ImageId='ami-0123456789abcdef0',
    InstanceType='t3.micro',
    MinCount=1,
    MaxCount=1,
    ClientToken='order-4821-launch-1',
    TagSpecifications=[
        {
            'ResourceType': 'instance',
            'Tags': [{'Key': 'Name', 'Value': 'web-01'}],
        }
    ],
)
new_id = reservation['Instances'][0]['InstanceId']

# Paginate: one describe_instances call is not the whole fleet.
paginator = ec2.get_paginator('describe_instances')
running = []
for page in paginator.paginate(
    Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
):
    for res in page['Reservations']:
        for inst in res['Instances']:
            running.append(inst['InstanceId'])

print(new_id, len(running))

Three things are happening here that you will reuse constantly: a deliberate retry configuration for a non-interactive workload, an idempotency token so a network retry can't double-launch, and a paginator so the inventory is genuinely complete. The resource-count side of quotas — how to inspect limits, and why quota planning belongs in architecture even for small workloads — is handled in "Guardrails for Foundations."

💡 Mental Model: The control plane is the building's electrical panel; the data plane is everything plugged into it. You rarely touch the panel, but when you do, you want to know exactly which breaker you're pulling — and the panel has a much lower tolerance for abuse than the sockets do.

With operations, addresses, endpoints, identities, and mechanics in hand, the next question is which service family a given problem actually belongs to — which is where "Core Service Capability Map: Choosing the Right Family for a Problem" picks up.

Core Service Capability Map: Choosing the Right Family for a Problem

A problem arrives as a sentence, not a service name: "Receipts get uploaded, we extract the totals, and three teams need to know about it." AWS offers hundreds of services, and searching the catalog by keyword is a poor way to answer that sentence. The skill this section builds is classification — sort the problem into a family, take that family's default starting service, then check the cues that would push you off the default.

The five families, and the question each one answers

Every core service you will meet in this course belongs to one of five families, and each family exists to answer one question.

A useful consequence of this map: most AWS design arguments are really family arguments in disguise. When two engineers disagree about whether an order pipeline should use SQS or Kinesis, they are not debating service features — they are disagreeing about whether the system needs a queue or a replayable log. Naming the family first turns a feature comparison into a requirements conversation.

🎯 Key Principle: choose the family from the problem, then let constraints pick the service inside the family. Choosing a service first inverts the process, and the symptom is always the same — a key-value store being asked to answer relational questions, or a queue being asked to replay history.

Compute: match the runtime to the work

Compute answers "where does code run," and the three options differ less in raw power than in what they make you responsible for.

EC2 gives you machines. You pick the AMI, the instance type, the kernel, the driver versions, the network interfaces. That control is the product: a licensed commercial solver pinned to a specific OS build, a GPU driver that must match a particular compute library version, a long-lived process holding an in-memory cache, a custom kernel module for packet inspection — these are EC2 problems. The scaling unit is the instance, and making many instances behave like one service is a separate layer of work.

Containers (ECS, EKS, and Fargate as a way to run either without managing nodes) give you a packaging and orchestration model. The artifact is an image. You get scheduling, placement, rolling replacement, service discovery, and a clean separation between "what the process is" and "where it runs." Choose containers when the deliverable your CI pipeline already produces is a container image, or when you need many small services with independent deploy cadences.

Lambda gives you a function contract. A trigger arrives, your handler runs, it ends. There is no host to patch, no long-lived connection to own, and billing follows invocations. In exchange you accept constraints: an invocation has a maximum duration, the execution environment is not the place to keep state between invocations, and concurrency — not CPU size — is the scaling dial.

A decision order that resolves most cases:

1. Does each unit of work finish in seconds and start from an event?  → Lambda
2. Is the artifact a container image, with no OS-level control needed? → ECS / EKS (with Fargate if you want no nodes)
3. Do you need OS, kernel, license, or hardware control?               → EC2

The non-obvious part is that steps 1 and 2 are not about the artifact format at all. You can package a Lambda as a container image; you can run a container built from the same base image on EC2. The real axis is the execution model — a function that starts, runs, and exits versus a long-running service that must stay up. Duration and state, not packaging, decide this.

When you are unsure what a family's limits actually are, query the API instead of guessing. Both Lambda and EC2 expose their constraints programmatically:

import boto3

# Lambda: the account-level limits describe the runtime contract you are buying.
lam = boto3.client("lambda", region_name="us-east-1")
limits = lam.get_account_settings()["AccountLimit"]
print(limits["ConcurrentExecutions"])   # account concurrency ceiling
print(limits["CodeSizeZipped"])         # max zipped deployment package, in bytes

# EC2: instance types are a queryable catalog, not a fixed menu.
ec2 = boto3.client("ec2", region_name="us-east-1")
info = ec2.describe_instance_types(InstanceTypes=["t3.micro"])
spec = info["InstanceTypes"][0]
print(spec["VCpuInfo"]["DefaultVCpus"])  # 2
print(spec["MemoryInfo"]["SizeInMiB"])   # 1024

Run that once and you have replaced three documentation pages with two calls you can repeat in any account. The exact numbers are per-instance-type facts, not opinions — which is why the API is the better source.

⚠️ Common Mistake: comparing EC2 and Lambda on "cost per hour." They bill on different units — instance-hours versus requests plus compute duration — so the comparison only becomes meaningful once you know the utilization shape. The pricing trade-offs themselves live in "Guardrails for Foundations: Quotas, Tagging, Cost Signals, and Blast Radius."

Storage and data: start from the access pattern, not the data volume

The most useful cue in this family is a single question: what shape is one unit of read? Answer it and the service mostly picks itself.

• 🗂 S3 stores objects: one whole blob, read or written over HTTP, addressed by a key you choose. Capacity is effectively unlimited, with versioning and lifecycle policies as first-class features. It is not a filesystem, and treating a key prefix as a directory tree works right up until something expects POSIX semantics — file locking, atomic rename, or partial writes into a file.
• 💽 EBS provides block volumes: a virtual disk attached to EC2 instances in the same Availability Zone. The operating system sees a device and puts a filesystem on it. Boot volumes, databases you run yourself, and anything that needs a real block device live here.
• 🗄 EFS is a shared file system over NFS, reachable from many instances across Availability Zones at once. Its reason to exist is the word shared: the moment two or more machines must see the same files simultaneously, a single block volume stops being an option.
• 🧮 RDS and Aurora are managed relational engines. You keep the schema, the queries, the joins, and the transactions; AWS runs the host, the engine patching, and the backup machinery. Client connections are stateful and the query language is relational.
• ⚡ DynamoDB is key-value and document storage where the access patterns are the design input. You choose partition and sort keys so that every query the application needs becomes a key lookup or a range scan within one partition. If you cannot list the access patterns, you cannot design the table.

The identity model is where this family stops looking like a menu and starts looking like a design decision. Storing "the same" record in S3 and in DynamoDB produces two completely different API calls, because what identifies the record differs in each:

import json
import boto3

s3 = boto3.client("s3", region_name="us-east-1")
# Object storage: identity is the key path inside a bucket.
s3.put_object(
    Bucket="orders-raw-123456789012",
    Key="orders/raw/order-4471.json",
    Body=json.dumps({"order_id": "4471", "total": 129.50}),
    ContentType="application/json",
)

ddb = boto3.client("dynamodb", region_name="us-east-1")
# Key-value: identity is the partition key, and the low-level client
# requires you to declare the type of every attribute.
ddb.put_item(
    TableName="Orders",
    Item={
        "pk": {"S": "ORDER#4471"},
        "sk": {"S": "META"},
        "total": {"N": "129.50"},
    },
)

Note the difference in obligation. With S3 you invent a key and the object exists. With DynamoDB through the low-level boto3 client, you must supply a type descriptor for every attribute ("S", "N"), because the table has a declared key schema and each item must fit it. The boto3 resource interface infers those types for you — convenient in scripts, and a liability when you want to see exactly what crosses the wire.

💡 Mental Model: the storage family is a set of answers to "what is one unit of read?" — one object (S3), one block (EBS), one file in a shared tree (EFS), one row across joined tables (RDS), one item by key (DynamoDB). Match the answer to your workload and the service follows.

Networking and content delivery: a chain, not a menu

These four services are usually taught as separate topics, which hides the fact that in a real request path they are sequential layers, each answering a different question.

User request
   ↓
Route 53        → resolves the name, and can fail over between endpoints
   ↓
CloudFront      → terminates TLS at an edge location; may answer from cache
   ↓
API Gateway     → authenticates, throttles, validates, routes the API call
   ↓
Compute in a VPC → the isolated network where your resources actually live

Read the chain as a set of boundaries:

• 🌐 VPC answers who can reach whom. It is an isolation boundary with subnets, route tables, and gateways. Designing it properly is its own lesson — see "VPC Deep Dive (Subnets, Routing, NAT, Endpoints)" — but the cue to keep here is that a VPC choice is a reachability decision, not a performance decision.
• 📇 Route 53 answers what does this name resolve to. It is DNS with routing policies attached — simple, weighted, latency, failover, among others — which makes it the natural control point for DNS-level failover and traffic shifting between endpoints.
• 🌍 CloudFront answers how do bytes get closer to users. It caches at edge locations, terminates TLS, and fronts origins that can be an S3 bucket, a load balancer, or an API Gateway stage.
• 🚪 API Gateway answers what is the contract of this API. Authentication, authorization, throttling, request validation, API keys, usage plans, and routing to Lambda, HTTP backends, or private integrations.

⚠️ Common Mistake: treating CloudFront and API Gateway as alternatives. They compose. CloudFront is about content delivery and edge TLS; API Gateway is about API contract management in a region. The shape CloudFront → API Gateway → Lambda uses both, and each is doing something the other cannot.

If you keep only one discriminator from this family: reachability is VPC, naming is Route 53, distance is CloudFront, and API policy is API Gateway.

Integration and messaging: choose by guarantee, not by name

Four services, four genuinely different problems. The names are easy to confuse; the guarantees are not.

• 📥 SQS is a queue. A producer writes a message, it waits, a consumer polls, processes, and deletes it. A queue retains messages until they are successfully processed or the retention window expires, and a dead-letter queue captures messages that repeatedly fail. This is the default answer to "decouple this producer from that consumer."
• 📢 SNS is publish/subscribe. One publish fans out to every subscriber — other queues, functions, HTTP endpoints, email. It does not store messages for a consumer that has not yet subscribed.
• 🧭 EventBridge is an event router. You publish an event with a source, a detail type, and a JSON detail, and rules match on those fields to route to many targets. It also handles scheduled rules and supports archiving and replaying events.
• 🌊 Kinesis Data Streams is an append-only log partitioned into shards. Records are ordered within a shard, retained for a configurable window, and every consumer reads the same records independently — which is exactly what makes replay and multiple consumers possible.

Five questions separate them, and the order matters:

1. 🔄 Does the consumer pull work when it is ready, or is work pushed at it?
2. 🛡 Must a message survive until something successfully processes it?
3. 🔢 Does ordering matter, and at what granularity — globally, per customer, per shard?
4. 👥 Do multiple independent consumers need to see the same data?
5. 🧭 Is routing decided by the content of the event, or by which topic it was published to?

The API shapes make the difference concrete. In boto3, sending to a queue, publishing to a topic, and routing an event are three distinct contracts:

import boto3

sqs = boto3.client("sqs", region_name="us-east-1")
sns = boto3.client("sns", region_name="us-east-1")
events = boto3.client("events", region_name="us-east-1")

# SQS: a queue holds the message until a consumer deletes it.
sqs.send_message(
    QueueUrl="https://sqs.us-east-1.amazonaws.com/123456789012/orders",
    MessageBody='{"order_id": "4471"}',
)

# SNS: fan out to whatever is subscribed right now.
sns.publish(
    TopicArn="arn:aws:sns:us-east-1:123456789012:order-events",
    Message='{"order_id": "4471"}',
)

# EventBridge: you describe the event; rules decide who receives it.
events.put_events(Entries=[{
    "Source": "com.example.orders",
    "DetailType": "OrderPlaced",
    "Detail": '{"order_id": "4471"}',
    "EventBusName": "default",
}])

Two details in that block change correctness. First, in the EventBridge call the detail is a JSON string on the wire — passing a Python dict raises a parameter validation error rather than serializing it for you. Second, if the queue in the first call is a FIFO queue, send_message rejects the request unless you supply a MessageGroupId, and ordering is guaranteed only within a message group. That is precisely the mechanism that lets a FIFO queue scale past a single consumer, and it is why "strict ordering" is always a per-group claim rather than a global one.

⚠️ Common Mistake: reaching for SNS when you need durability. A publish with no subscribers is simply gone, and a subscriber that is down is retried only as long as its protocol's delivery policy allows — three retries by default for an HTTP/S endpoint — after which the message is discarded unless the subscription has its own dead-letter queue. If a message must be processed eventually, it needs a queue behind the topic — the classic fanout pattern is SNS distributing to several SQS queues, each with its own consumer and dead-letter queue.

💡 Real-World Example: an order service that publishes directly to SNS and expects the warehouse system's HTTPS endpoint to catch up later can quietly lose orders during a warehouse deployment that outlasts those retries. Routing SNS → SQS → warehouse consumer turns that silent loss into a growing queue depth you can alarm on, plus a dead-letter queue you can inspect.

The streaming side of this family — Kafka on MSK, shard scaling, consumer groups — is deep enough to get its own treatment in "Batch, Queues & Streams (SQS, SNS, Kinesis, MSK)."

Security and management: who, with what key, recorded where

The last family serves no traffic at all — it governs everything else, which is why it is usually configured first and forgotten first.

• 🔐 IAM answers who can call which API against which resource, under what conditions. Every other service's authorization decision resolves here, and every design in the previous four families assumes an identity exists. Policy structure gets its own treatment in the IAM material; for this map, note that IAM is the family's center of gravity.
• 🔑 KMS manages keys. Most "encrypted at rest" checkboxes in AWS resolve to a KMS key, and the key policy is a permission surface people forget they own. An S3 bucket encrypted with a customer-managed key, read by a Lambda function, requires the function's role to have access to the key — not just to the bucket.
• 📈 CloudWatch holds metrics, logs, and alarms: the operational view of what is happening right now.
• 📜 CloudTrail records API activity: who did what, and when. The distinction matters because the two answer different questions. CloudWatch tells you an instance's CPU is pegged; CloudTrail tells you who changed the security group five minutes before it was.
• 🏢 AWS Organizations structures accounts. It is what makes "separate environments" real rather than nominal, and it is where consolidated billing and service control policies live.

The shared responsibility model is easiest to hold as a moving line. As you move up the stack from EC2 to containers to Lambda, AWS takes on more layers — and the line never reaches "you own nothing."

🎯 Key Principle: serverless moves the responsibility line; it does not remove it. Lambda patches the runtime, but not your third-party dependencies, not the permissions on its execution role, and not the data it writes. Every family has a row in that table with your name in it.

📋 Quick Reference Card — the first question to ask per family:

Practice: four problems, four families

Classify each before reading the answer.

1. A nightly simulation runs 25 minutes and uses a commercially licensed solver pinned to a specific OS build.
2. A mobile app uploads profile photos that users on three continents must load quickly.
3. Order events must be processed in per-customer order, and three teams each need their own independent copy of the same event history.
4. A third-party system must be notified on every order state change, and each of four internal consumers wants a different subset of those events.

Answers, with the cue that decides each one:

1. EC2 — duration and licensing control defeat Lambda (invocation duration limit), and containers add orchestration without solving the license. The constraint is machine-bound control.
2. S3 plus CloudFront — two families, and that is normal. Object storage solves the upload; edge caching solves the distance. Neither service alone answers the whole sentence.
3. Kinesis Data Streams — per-shard ordering plus multiple independent consumers reading the same retained records. An SQS FIFO queue would give per-group ordering and deduplication, but a second team reading that queue competes for messages instead of getting its own copy.
4. EventBridge — routing is decided by content, with a different target set per rule. SNS would require either one topic per subset or a filter policy maintained on every subscription.

None of those four problems was solved by asking "which service is best." Each was solved by finding the constraint that eliminates the alternatives: duration, distance, replay, and content-based routing.

That is the whole map: identify the family from the problem, take the family's default, then let one decisive constraint choose the service.

Programmatic Access in Practice: CLI, SDK, and IaC Entry Points

The console teaches you what an API looks like; the terminal and the SDK teach you what it does when nobody is watching. The first time you run the same create command twice and end up with two resources — or realize the script you have been running all week was pointed at the wrong account — you learn that programmatic access is mostly about controlling three things: which identity, which region, and how many times a mutating call actually executes.

Credential Resolution: Who Is This Shell, Really?

Every aws command invocation and every boto3 client resolves credentials through a provider chain — an ordered list of places to look. The first entry that yields a complete credential set wins, and nothing further down the list is consulted.

| 🥇 Order | 🔑 Source | 🎯 Typical setting | | 1 | Explicit constructor args | Keys passed in code | | 2 | Environment variables | CI runners, containers | | 3 | Web identity / SSO | Federated humans | | 4 | Shared credentials file | ~/.aws/credentials | | 5 | Shared config file | ~/.aws/config | | 6 | Container credentials | ECS task role | | 7 | Instance metadata | EC2 instance profile |

Two behaviors fall out of that ordering. An environment variable silently overrides the profile you thought you were using — a CI job that exports AWS_ACCESS_KEY_ID will use that key for credentials even when AWS_PROFILE=prod-deploy is set as well. (An explicit --profile prod-deploy on the command line does beat the exported key.) And an instance role only works while the instance metadata service is reachable and permitted; a container that blocks IMDS falls through the chain to nothing at all. When a script behaves differently on your laptop than on a build agent, the provider chain is the first thing to inspect.

⚠️ Common Mistake: Hardcoding an access key and secret into a script or a repository. The problem is not only hygiene — a key that exists outside IAM is a key that cannot be scoped to a workload, rotated on a schedule, or traced to a specific caller. The remediation path (roles, permission boundaries, least privilege) belongs to the IAM material; here the rule is narrower and sufficient: the credential must come from the environment the code runs in, never from the source you ship.

Region resolution is a separate chain, and conflating the two produces the classic "it worked yesterday" bug. For the AWS CLI the documented order is the --region flag, then the AWS_REGION environment variable, then AWS_DEFAULT_REGION, then the region value in the active profile. boto3 follows the same shape with one difference: an explicit region_name on the Session or client always wins; otherwise botocore consults AWS_DEFAULT_REGION (it does not read AWS_REGION) and then the profile. The safe habit is identical in both tools — name the region explicitly in anything you intend to re-run.

# 1. Who am I, and where did each setting come from?
aws sts get-caller-identity
aws configure list     # prints each value with its source: env, config file, IMDS...

# 2. In scripts, never inherit ambient state — name the profile and region.
aws --profile prod-deploy --region eu-west-1 sts get-caller-identity \
  --query 'Account' --output text

aws configure list is the fastest diagnostic in the toolkit: it prints every setting next to a source column (env, shared-credentials-file, config-file, iam-role), turning "why is this hitting the wrong account" into a one-line answer. sts get-caller-identity is the other half — it returns the account, the caller ARN, and the unique user or role ID, so any script that mutates infrastructure should call it first and log the result.

💡 Pro Tip: Make sts get-caller-identity the first line of any destructive script and abort unless the returned account ID matches an expected constant. It costs one API call and catches the entire class of "forgot to set AWS_PROFILE in a new terminal" incidents before they touch anything.

AWS CLI Patterns That Survive Production

Three flags change what a command returns, and two command families change when it returns.

--query evaluates a JMESPath expression against the response before it is rendered, which keeps a lot of jq plumbing out of your scripts:

# Just the instance IDs, one per line, ready for a shell loop
aws ec2 describe-instances \
  --filters 'Name=instance-state-name,Values=running' \
  --query 'Reservations[].Instances[].InstanceId' \
  --output text

--output selects rendering: json (the default), yaml, text (tab-separated, ideal for cut, awk, and shell loops), and table (for human eyes only — never parse it, column widths shift with content).

--dry-run deserves its own paragraph because it inverts intuition. On the subset of API operations that support it — mostly EC2 and a handful of others — the service performs all request validation and authorization but takes no action. If you would have been allowed, the call fails with a DryRunOperation error; if you lack permission, it fails with UnauthorizedOperation. The error code is the answer, which means a successful dry run exits non-zero and looks like a failure to a naive script.

⚠️ Common Mistake: Running a dry run inside a set -e script and assuming success means exit 0. It means the opposite. Capture the exit code deliberately, read Error.Code, and branch on it. Also note the coverage gap: --dry-run exists on API-level commands, not on the high-level aws s3 family. For those, the high-level commands expose their own --dryrun flag on sync, cp, and mv — no second hyphen.

Waiters (aws ec2 wait instance-running, aws rds wait db-instance-available) poll an API on a built-in delay-and-attempts schedule. They exit 0 once the condition holds and exit 255 on timeout, which removes hand-rolled sleep loops and their inevitable drift.

Pagination is where the CLI earns its keep. The CLI auto-paginates: it issues repeated calls and concatenates the pages before rendering, so aws s3api list-objects-v2 returns everything by default rather than the first 1,000 keys. Three controls matter. --page-size changes how many items each underlying API call requests — lowering it avoids timeouts on very large collections without changing the total returned. --max-items caps the combined output and appends a NextToken you must feed back as --starting-token on the next invocation. --no-paginate returns only the first page, which is sometimes exactly what you want to avoid burning API calls on a huge listing.

📋 Quick Reference Card: CLI flags for scripting

| 🚩 Flag | 🎯 Effect | ⚠️ Watch | | --query | JMESPath projection | quoting matters | | --output text | Tab-separated, script-safe | not JSON | | --output table | Human reading only | never parse | | --dry-run | Permission probe | subset of APIs | | --page-size | Items per API call | total unchanged | | --max-items | Cap plus NextToken | resume manually | | --no-cli-pager | Disable pager | needed in CI |

Those pieces compose into a script that is safe to re-run:

#!/usr/bin/env bash
set -euo pipefail

PROFILE="prod-deploy"; REGION="us-east-1"
AWS=(aws --profile "$PROFILE" --region "$REGION" --no-cli-pager)

# 1. Prove which account we are about to mutate.
ACCOUNT_ID=$("${AWS[@]}" sts get-caller-identity --query 'Account' --output text)
echo "operating in account $ACCOUNT_ID"

# 2. Select the newest AMI by creation date, not by list order.
AMI_ID=$("${AWS[@]}" ec2 describe-images \
  --owners amazon \
  --filters 'Name=name,Values=al2023-ami-2023*-x86_64' \
  --query 'reverse(sort_by(Images, &CreationDate))[0].ImageId' \
  --output text)

# 3. Launch with a deterministic token and tags applied at creation time.
INSTANCE_ID=$("${AWS[@]}" ec2 run-instances \
  --image-id "$AMI_ID" --instance-type t3.micro --count 1 \
  --client-token "probe-$ACCOUNT_ID-$AMI_ID" \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=probe},{Key=ManagedBy,Value=cli}]' \
  --query 'Instances[0].InstanceId' --output text)

# 4. Block until it is genuinely running, then report.
"${AWS[@]}" ec2 wait instance-running --instance-ids "$INSTANCE_ID"
echo "running: $INSTANCE_ID"

The AMI query is the instructive line. describe-images returns images in no guaranteed order, so taking [0] picks an arbitrary one; sort_by(Images, &CreationDate) followed by reverse(...) makes "newest" explicit. Combining that with a --query that returns a bare string and --output text means the shell variable holds an AMI ID and nothing else — no jq, no second parse.

boto3: Client, Resource, and the Objects in Between

Three objects carry the whole boto3 model. A Session owns credentials, region, and profile; clients and resources built from it inherit all three. A client maps one-to-one onto the service API: client.list_objects_v2(Bucket=...) corresponds to that API operation, returns raw dictionaries, and supports paginators, waiters, and per-service configuration. A resource is an object-oriented wrapper — s3.Bucket("x").objects.all(), dynamodb.Table("t").put_item(...) — which reads nicely but covers an uneven subset of each service's operations.

🎯 Key Principle: reach for the client in automation and the resource for exploration. Client responses are the same shape as the API documentation, which means every Stack Overflow answer, every CloudTrail record, and every AWS doc page maps directly onto your code. Resources hide that mapping, and when an operation you need is not wrapped, you drop back to the client anyway.

import logging
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError, WaiterError

log = logging.getLogger("inventory")

# One session owns credentials and region; every client made from it inherits both.
session = boto3.Session(profile_name="prod-readonly", region_name="us-east-1")

# botocore's default retry mode is "legacy". Ask for "standard" explicitly:
# jittered exponential backoff on throttling and transient 5xx responses.
config = Config(
    retries={"max_attempts": 10, "mode": "standard"},
    connect_timeout=5,
    read_timeout=30,
)
s3 = session.client("s3", config=config)


def list_all_keys(bucket: str) -> list[str]:
    """Every key in the bucket — paginator instead of a hand-rolled NextToken loop."""
    paginator = s3.get_paginator("list_objects_v2")
    keys: list[str] = []
    for page in paginator.paginate(Bucket=bucket, PaginationConfig={"PageSize": 1000}):
        for obj in page.get("Contents", []):   # empty buckets omit Contents entirely
            keys.append(obj["Key"])
    return keys


def wait_for_bucket(bucket: str) -> None:
    """Block until the bucket is readable, or fail loudly after ~60 seconds."""
    try:
        s3.get_waiter("bucket_exists").wait(
            Bucket=bucket, WaiterConfig={"Delay": 5, "MaxAttempts": 12}
        )
    except WaiterError as exc:
        raise RuntimeError(f"{bucket} never became readable") from exc


def head_or_none(bucket: str, key: str) -> dict | None:
    """head_object, branching on the two failure shapes callers actually care about."""
    try:
        return s3.head_object(Bucket=bucket, Key=key)
    except ClientError as exc:
        code = exc.response["Error"]["Code"]
        if code in ("404", "NoSuchKey", "NotFound"):
            return None                  # expected: the object is not there yet
        log.warning("head_object(%s, %s) failed: %s", bucket, key, code)
        raise                            # throttling, access denied, and friends

Several deliberate choices sit in that snippet. The Session is created once and shared, so region and identity are decided in exactly one place. The paginator replaces the ContinuationToken/NextToken loop you would otherwise hand-write and get wrong on empty buckets — note page.get("Contents", []), because a bucket with no objects returns a response without a Contents key rather than an empty list. The waiter turns "is it ready yet" into a declarative call. And head_or_none separates an expected 404 from an unexpected failure: a HEAD request returns no body, so botocore reports the missing object as the string code "404" rather than the NoSuchKey you would see from get_object.

On retries, the defaults differ from the CLI in a way that bites people. The AWS CLI v2 defaults to standard retry mode; botocore's default is legacy. So the two tools do not retry the same way: by default legacy makes up to five attempts and standard three, and standard adds a retry quota that legacy does not have. Name the setting — Config(retries={"mode": "standard", "max_attempts": N}), or the AWS_RETRY_MODE and AWS_MAX_ATTEMPTS environment variables — rather than assuming the two tools behave alike.

Failure Modes Your Code Must Actually Handle

Three failure classes show up constantly in programmatic AWS work, and each has a distinct response.

Throttling. You are calling faster than your account's quota allows. The error code varies by service: Throttling and ThrottlingException are common, DynamoDB returns ProvisionedThroughputExceededException, S3 can return SlowDown, and some services emit RequestLimitExceeded. Standard retry mode handles these automatically with backoff; what it cannot do is fix a caller that issues thousands of unbatched requests per second. When retries alone are not enough, the answer is batching or a queue, not a longer max_attempts.

Eventual consistency. A resource you just created may not appear in a list or describe call yet. Handle it with a waiter or a bounded, backed-off retry — never a bare time.sleep(5) scattered through the code, which is either too short on a bad day or wasted time on a good one.

Transient connection and server errors. Timeouts and connection resets live outside ClientError. They surface as BotoCoreError subclasses such as EndpointConnectionError, ConnectTimeoutError, and ReadTimeoutError, plus WaiterError from waiters and ParamValidationError from malformed input. A 5xx response that the service did send still arrives as a ClientError, with a code such as InternalError.

⚠️ Common Mistake: Catching bare Exception and retrying everything. AccessDenied, ValidationException, and MalformedPolicyDocument will never succeed on the eleventh attempt; retrying them wastes quota and hides the real message. Retry only the retryable taxonomy, and only for operations that are safe to repeat.

That last qualifier is the important one. Retrying run_instances without an idempotency token can leave you with two instances and one puzzled operator. Which brings us to the two remaining pieces of the puzzle.

From Ad-Hoc CLI to IaC: Where the Corner Is

The rule of thumb is blunt: the second time a human needs to reproduce a resource, move it into code. One-off CLI is excellent for inspection, debugging, and probing permissions. The moment a step must be repeated — a new environment, a rebuilt account, a teammate following your runbook — shell history stops being documentation and starts being a liability, because it captures what you did without capturing what the result should be.

| 🧩 Entry point | 🎯 Best for | ⚠️ Cost | | Ad-hoc CLI | Inspection, debugging | Not reproducible | | Scripted CLI | Gluing CI steps | Ordering lives in bash | | CloudFormation | Native, drift detection | Verbose for big graphs | | AWS CDK | Typed, testable app code | Synth step to learn | | Terraform | Plan preview, multi-cloud | State file is the truth |

CloudFormation is the native option and the only one whose drift detection and stack events are first-class in the console. The CDK synthesizes to CloudFormation from typed, testable application code, which suits teams that already review code. Terraform offers a plan/apply workflow with a preview of changes, at the cost of treating a state file as the authoritative record of what exists. This lesson takes no side; the practical point is that all three beat a shell script the moment more than one person or environment is involved.

💡 Real-World Example: A team's runbook says "create the bucket, enable versioning, add the lifecycle rule." Two engineers follow it by hand and end up with two subtly different buckets — one missing the lifecycle rule, one with versioning off. The same steps expressed as a template or CDK construct produce identical buckets every time, and the diff between environments becomes visible in a code review rather than discovered during an incident.

Traceability is what makes the move worthwhile. Give every resource a deterministic name, apply tags at creation time, and know how to ask the tool what it actually created. With CloudFormation that question has a direct answer: aws cloudformation describe-stack-resources --stack-name <name> lists every physical ID the stack manages, which is how you connect a logical template entry to the thing sitting in your account. With Terraform it is terraform state list. Either way, the goal is the same — no resource in the account should be unaccounted for by the tool you claim manages it.

Idempotent Automation: Names, Tags, and Tokens

Idempotence means running the same automation twice leaves the system in the same state as running it once. Four techniques carry most of the weight.

🧠 Deterministic naming. Derive names from stable inputs — environment, application, resource type — rather than from uuid4() or a timestamp. ${env}-${app}-${purpose} is reproducible; my-bucket-1738 is a resource nobody can find again. Constraint: names must still fit the service's character and length rules, so keep components short and normalize case where the service requires lowercase.

🔧 Tag at creation, not after. Use --tag-specifications or TagSpecifications on the create call. Tagging afterward leaves a window where an untagged resource exists, and if the script fails between the two calls, that window never closes. What tags you require is a governance decision covered later in the lesson on guardrails; here the mechanic is simply tag-on-create.

🔁 Retry with backoff, only where it is safe. Standard retry mode handles transport-level retries. Your own retries should sit above that, add jitter, and wrap only idempotent operations.

🔒 Service idempotency tokens. Many create operations accept a ClientToken, which the service uses to recognize a repeated request and return the original result instead of creating a second resource.

import hashlib

from botocore.exceptions import ClientError


def stable_token(*parts: str) -> str:
    """Deterministic across runs, unique per input set — never uuid4()."""
    digest = hashlib.sha256("|".join(parts).encode()).hexdigest()
    return digest[:32]


def ensure_probe_instance(ec2, ami_id: str, env: str) -> str:
    token = stable_token("probe-instance", env, ami_id, "t3.micro")
    try:
        resp = ec2.run_instances(
            ImageId=ami_id,
            InstanceType="t3.micro",
            MinCount=1,
            MaxCount=1,
            ClientToken=token,          # same token -> same instance, not a second one
            TagSpecifications=[{
                "ResourceType": "instance",
                "Tags": [
                    {"Key": "Name", "Value": f"{env}-probe"},
                    {"Key": "Env", "Value": env},
                    {"Key": "ManagedBy", "Value": "boto3"},
                ],
            }],
        )
        return resp["Instances"][0]["InstanceId"]
    except ClientError as exc:
        code = exc.response["Error"]["Code"]
        if code == "IdempotentParameterMismatch":
            # Same token, different arguments: the inputs are not deterministic.
            raise RuntimeError("ClientToken reused with different parameters") from exc
        raise

The token is the load-bearing line, and its derivation is the subtle part. Hashing the inputs means the same intent always produces the same token, so a retry after a network timeout returns the instance that was already created rather than a second one. An IdempotentParameterMismatch error is diagnostic gold: it means the same token arrived with different arguments, which almost always indicates an input that changes between runs — a timestamp in the name, an unordered list, a random suffix.

Two limits are worth stating plainly, because they change how you use this. First, idempotency tokens are retained by the service for a bounded window, not forever, so the token protects retries within a run far more than it protects a manual re-run next week; long-term idempotence comes from deterministic names and from an IaC tool that tracks what it created. Second, not every create operation accepts a token — where one is missing, deterministic naming plus a lookup-before-create check (describe for the resource by its deterministic name, and only create if absent) is the fallback pattern. Both techniques depend on the same precondition: the tool must be able to tell whether the resource already exists, which is exactly what naming discipline buys you.

The through-line across all of this is that programmatic AWS work is a control problem, not a syntax problem. Name your identity and region explicitly, project responses with --query so scripts get data rather than prose, respect the failure taxonomy instead of retrying everything, and make creation repeatable through deterministic names, tag-on-create, and idempotency tokens — then hand the whole thing to an IaC tool once a second person needs to run it.

Composing Core Services: A Minimal Workload Walkthrough

A single AWS service is easy to call. The failures that cost you weekends live at the seams — where an object lands in a bucket and something else is supposed to notice, where a function writes state that a different function needs to know about, where a failure in one component quietly becomes a gap in another. This section builds one small workload end to end so you can see those seams, name them, and reason about them. The workload is deliberately minimal; hardening, VPC placement, and policy design are later concerns.

The Reference Workload: "Receipt Drop"

Imagine an expense-tracking app. A user photographs a receipt and uploads it. The system extracts the merchant, date, and total, stores the result, and notifies a downstream budgeting service. That is four responsibilities — object storage, compute, state, messaging — plus one cross-cutting one, observability.

Here is the whole thing as a vertical flow. Read it top to bottom; each arrow is a boundary where one service stops and another begins:

User (browser / mobile app)
   ↓  POST /uploads  →  API Gateway (sync)
Lambda "request-upload"  (sync invoke, user is waiting)
   ↓  returns a presigned S3 URL
User
   ↓  PUT object directly to S3
S3 bucket  receipts-inbox-<account-id>
   ↓  ObjectCreated notification  (async — nobody is waiting)
Lambda  receipt-processor
   ↓  PutItem (conditional write)
DynamoDB table  ReceiptState
   ↓  PutEvents
EventBridge bus  receipts-bus
   ↓  rule match → target
Downstream consumer(s): budgeting service, audit log

Side channel from every component:
   logs + metrics  →  CloudWatch

Five services, one user-visible feature. Notice how little of the work happens synchronously. The user waits exactly twice: once for the presigned URL, once for the upload itself. Everything downstream of the S3 notification is asynchronous, which is precisely why the failure story is more interesting than the happy path.

💡 Mental Model: think of each box as owning a small piece of truth. S3 owns the bytes. DynamoDB owns the receipt's processing status. EventBridge owns the fact that something happened. Lambda owns nothing durable at all — it is pure computation that starts, runs, and disappears. When you can say which box owns which truth, deciding where to debug becomes mechanical.

🤔 Did you know? The reason a presigned URL exists in this design is that it keeps large binary data out of your compute path. The bytes go straight from the client to S3; Lambda never sees them. That is a boundary decision made for cost and throughput reasons, not an implementation detail.

Synchronous vs Asynchronous Paths

This is the single most useful distinction to carry out of this lesson, because it determines where you handle errors.

A synchronous path means a caller is blocked, holding open a connection, waiting for a response. If it fails, the caller sees the failure immediately and can decide what to do. A asynchronous path means the caller has already moved on. The producer said "this happened" and walked away. If the consumer fails, nobody is holding a connection to tell — the platform must retry on your behalf, and eventually someone must decide what happens after retries run out.

🧠 Mnemonic: Synchronous means Someone is watching. If no one is watching, you must install a safety net yourself.

The synchronous side, concretely

When the user calls POST /uploads, API Gateway invokes the request-upload Lambda synchronously. Two things follow from that. First, the user's latency budget is your latency budget — every millisecond inside that function is a millisecond the user is staring at a spinner. Second, an unhandled exception surfaces as a 5xx response. That is a feature: the user can retry.

The compensating action for this path is trivial because the operation is small and idempotent: generating a presigned URL has no side effects, so retrying it is harmless. Contrast that with the write path below, where retrying is not harmless at all.

The asynchronous side, concretely

Once S3 fires ObjectCreated, nobody is waiting. The user already got their "upload complete" confirmation. Now consider what happens when receipt-processor throws on a malformed image.

For asynchronous Lambda invocations — which is what an S3 notification produces — Lambda retries the invocation twice by default, then stops. After that, the event is either discarded into the void or handed to whatever you configured as the on-failure destination. You configure that with put-function-event-invoke-config:

aws lambda put-function-event-invoke-config \
  --function-name receipt-processor \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:123456789012:receipt-failures"}}' \
  --maximum-retry-attempts 2

That --maximum-retry-attempts 2 value is the default, but writing it explicitly is worthwhile because it is a decision, not an accident. Without a destination, failures are invisible — the invocation simply never happened as far as your business logic is concerned. The user's receipt shows as "pending" forever.

⚠️ Common Mistake: assuming a retry policy is the same as an error-handling strategy. Retries buy you tolerance for transient failures — a throttled DynamoDB write, a momentary network blip. They do nothing for deterministic failures like a corrupted image or a schema mismatch. Those will fail all three attempts and land in the same place. Your dead-letter queue must be drained and inspected, not just configured.

⚠️ Common Mistake: forgetting that redelivery creates duplicates. S3 event notifications are delivered at least once, and Lambda's own retries compound that. Your handler will eventually see the same object twice. The fix is not a bigger retry budget; it is an idempotent write, which is exactly where the conditional PutItem below earns its place.

Service Boundaries and Contracts

A contract at a service boundary is a promise about what crosses it. Get this right and services can be replaced, rewritten, or scaled independently. Get it wrong and every change ripples through the whole workload — the classic distributed-monolith failure.

Three rules make these contracts hold up.

Rule 1: pass a pointer, not a payload. The S3 notification carries the bucket and key, not the image bytes. Lambda then calls GetObject to fetch what it needs. This keeps the event small, keeps the event schema stable even if the object schema changes, and prevents the messaging layer from becoming an accidental storage layer.

Rule 2: the event is the contract, not the table. The tempting shortcut after Lambda writes to DynamoDB is to let the budgeting service query ReceiptState directly. Resist it. That couples a consumer to your internal item schema — change a column name and you break a service in another team's repository.

❌ Wrong thinking: "The data is already in DynamoDB, so querying it is cheaper than building an event."

✅ Correct thinking: "The consumer needs to react to a fact, not inspect our storage. Publish receipt.processed and let the consumer own its own read model."

Rule 3: version the event envelope. Wrap the payload with a stable header so consumers can evolve independently:

{
  "schemaVersion": "1.0",
  "eventType": "receipt.processed",
  "occurredAt": "2026-01-15T10:22:31Z",
  "detail": {
    "receiptId": "rcpt_8f21",
    "bucket": "receipts-inbox-123456789012",
    "key": "incoming/rcpt_8f21.jpg",
    "totalCents": 4287,
    "currency": "USD"
  }
}

(To publish this with PutEvents, eventType goes in the entry's DetailType — rules see it as detail-type — and the detail object goes, as a JSON string, in Detail. EventBridge has no field for schemaVersion, so carry it inside Detail as well.)

💡 Real-World Example: a team adds a merchantCategory field to their event detail. Because consumers switch on schemaVersion and ignore unknown fields, every downstream service keeps working through the rollout. Without the envelope, adding a field is a coordination problem across four repositories.

Now the seam I want you to watch most closely: state writes and event publishing are two operations in one function. If Lambda writes to DynamoDB and then crashes before publishing, the state says "processed" but no event ever fires. There is no transaction spanning both. The honest engineering answer is either (a) publish the event after the write and accept a small window, with a reconciliation job that finds processed-but-unpublished receipts, or (b) use DynamoDB Streams so the write itself emits the fact. Option (b) is the more elegant pattern and worth knowing exists — but it is a topic for the streaming material, not this walkthrough.

💡 Pro Tip: never let an event detail include anything the consumer must look up elsewhere to use. An event that says "receipt 8f21 processed, go ask S3 what it was" has moved the coupling, not removed it.

Building It Minimally with the AWS CLI

Here is the reproducible setup. It creates the bucket, table, bus, log group, function, and the permission that lets S3 invoke it. This is intentionally light on IAM — a single role with broad permissions gets the workload running, and tightening that role is squarely the territory of the IAM material and the guardrails section later in this lesson.

ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
REGION=us-east-1

# 1. Bucket — globally unique, so suffix with the account id
test -n "$ACCOUNT" || exit 1  # fail fast if sts returned an empty account
aws s3api create-bucket \
  --bucket receipts-inbox-$ACCOUNT \
  --region $REGION

# 2. Table — on-demand, single partition key
aws dynamodb create-table \
  --table-name ReceiptState \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

# 3. Event bus
aws events create-event-bus --name receipts-bus --region $REGION

# 4. Log group FIRST, with retention — otherwise the first invocation
#    creates a group that never expires
aws logs create-log-group --log-group-name /aws/lambda/receipt-processor
aws logs put-retention-policy \
  --log-group-name /aws/lambda/receipt-processor \
  --retention-in-days 14

# 5. Role (broad on purpose; narrow this in the IAM lesson)
aws iam create-role --role-name receipt-processor-role \
  --assume-role-policy-document file://trust.json
for POLICY in service-role/AWSLambdaBasicExecutionRole AmazonS3ReadOnlyAccess \
              AmazonDynamoDBFullAccess AmazonEventBridgeFullAccess; do
  aws iam attach-role-policy --role-name receipt-processor-role \
    --policy-arn "arn:aws:iam::aws:policy/$POLICY"
done
ROLE_ARN=$(aws iam get-role --role-name receipt-processor-role \
  --query Role.Arn --output text)

# 6. Function
aws lambda create-function \
  --function-name receipt-processor \
  --runtime python3.12 \
  --role "$ROLE_ARN" \
  --handler handler.handler \
  --zip-file fileb://function.zip \
  --environment "Variables={TABLE_NAME=ReceiptState,BUS_NAME=receipts-bus}"

# 7. Let S3 call it, scoped to this bucket
aws lambda add-permission \
  --function-name receipt-processor \
  --statement-id s3-invoke \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::receipts-inbox-$ACCOUNT \
  --source-account $ACCOUNT

# 8. Wire the notification
aws s3api put-bucket-notification-configuration \
  --bucket receipts-inbox-$ACCOUNT \
  --notification-configuration file://notification.json

Order matters in two places. The add-permission call must precede the notification configuration — S3 validates that it is allowed to invoke the function before it accepts the wiring, and if you configure the notification first the API call rejects it. And creating the log group before the function means your retention policy applies from the very first invocation.

A notification.json looks like this:

{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:receipt-processor",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {"FilterRules": [{"Name": "prefix", "Value": "incoming/"}]}
      }
    }
  ]
}

The prefix filter is doing real work: it stops the notification from firing on any future keys the same function writes back into the bucket, which would otherwise create an infinite loop.

⚠️ Common Mistake: hardcoding that ARN with an account ID copied from a console screenshot. Use $ACCOUNT in the generated file, or better, move this whole thing to IaC (the entry-point comparison lives in the programmatic-access section) so the ARN is derived rather than retyped.

The same thing in CDK, briefly

Once the resource graph is stable, IaC buys you repeatability and dependency ordering for free:

from aws_cdk import (
    Stack, aws_s3 as s3, aws_lambda as _lambda,
    aws_dynamodb as ddb, aws_events as events,
)
from constructs import Construct

class ReceiptStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        table = ddb.Table(
            self, "ReceiptState",
            partition_key=ddb.Attribute(name="pk", type=ddb.AttributeType.STRING),
            billing_mode=ddb.BillingMode.PAY_PER_REQUEST,
        )
        bus = events.EventBus(self, "ReceiptsBus", event_bus_name="receipts-bus")
        fn = _lambda.Function(
            self, "Processor",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="handler.handler",
            code=_lambda.Code.from_asset("lambda"),
            environment={"TABLE_NAME": table.table_name, "BUS_NAME": bus.event_bus_name},
        )
        table.grant_read_write_data(fn)   # writes the IAM policy for you
        bus.grant_put_events(fn)

Notice that the grants replace the hand-rolled role from the CLI version. That is the real payoff of IaC here: the permission boundary between the function and its resources is generated from the same graph that creates the resources, so it cannot drift.

Reading the Workload as a Set of AWS APIs

Every box in the diagram above is a service you configure with one family of calls and use with another. From the first section, you already have the vocabulary: control plane operations create and configure; data plane operations handle real traffic. Here is the mapping for this workload.

The distinction is not academic — it tells you which failure you are looking at. A AccessDenied on PutBucketNotificationConfiguration is a setup problem: the identity calling the CLI lacks configuration permission on the bucket. An AccessDenied on GetObject from inside the Lambda is a runtime problem: the function's own execution role lacks read permission. Same error text, two completely different places to look. When something breaks, first ask which plane you are in.

The second thing this mapping gives you is the ability to see that the console is not doing anything magical. Every wizard page is a thin form over control-plane calls, in a fixed order. When a wizard succeeds but your script fails, the difference is almost always ordering or a permission granted implicitly by the wizard — exactly the two things the CLI version above makes explicit.

💡 Remember: your function's static configuration (runtime, memory, environment variables, role) is control-plane state. Its behavior is data-plane. You can redeploy a function a hundred times without ever invoking it, and you can invoke a function millions of times without ever changing its configuration. Keeping those two timelines separate is what makes deployments and incidents tractable.

A note on scope: everything here runs in one region, with one account, and no explicit quota planning. The workload will work, but as it grows, the questions that matter — how many concurrent Lambda invocations can this account sustain, what the data transfer actually costs, how to tag these resources so the bill is attributable — belong to the guardrails section that follows. For now, the win is that you can name each component, say what it owns, say who waits for it, and point at the exact API call that created it.

Practice: Trace the Failure

Pick one of these and walk it through the diagram before reading further.

Scenario A: A user uploads a receipt. The API returns success. Ten minutes later the receipt still shows "pending" in the app. Where do you look first?

Start from the seam, not the code. The upload succeeded, so S3 is fine and the presigned URL worked. That means the S3 ObjectCreated notification either did not fire (check the prefix filter and the notification configuration) or did fire and the Lambda invocation failed all three attempts (check /aws/lambda/receipt-processor for an ERROR line, then check the on-failure queue). If CloudWatch shows no invocation at all, you are in the control plane. If it shows invocations with exceptions, you are in the data plane. That single fork eliminates half the system in one step.

Scenario B: Two consumers both process the same receipt twice. What happened?

The S3 notification was delivered twice, or Lambda retried after a timeout that actually succeeded, or a consumer is not idempotent. The producer side cannot be fixed — at-least-once is the contract. The fix is on the consumer: a conditional write keyed on receiptId using attribute_not_exists(pk), which turns the second attempt into a no-op instead of a duplicate.

Scenario C: The receipt.processed event fires, but the budgeting service never sees it. Nobody is waiting, so nothing logs a user-visible error.

Check the EventBridge rule's event pattern against the actual detail-type you publish. A pattern that matches receipt.processed will silently match nothing if your code publishes Receipt.Processed. There is no error for an unmatched event — this is the asynchronous tax. Fix it by asserting the event shape in a test, not by adding more logging.

That last point is the one to take into the rest of the lesson. Synchronous failures announce themselves. Asynchronous failures do not. Every arrow below the S3 notification in that first diagram is an arrow where a silent failure is possible, and the only defense is a deliberate contract, a retry policy you chose on purpose, and telemetry you actually read.

Guardrails for Foundations: Quotas, Tagging, Cost Signals, and Blast Radius

Two failure modes end most early AWS projects: a bill that grew faster than the workload, and a mistake in one environment that damaged another. Neither needs an attacker or an exotic edge case — both happen by default, because an AWS account is an open-ended resource pool with a payment method attached and no undo button. Guardrails are the defaults you put in place before the workload becomes interesting: the limits you plan for, the metadata you attach, the signals you watch, and the walls you build between environments. This section treats the cost trade-offs here as first-class design decisions rather than something you revisit when finance asks a question you can't answer.

Service Quotas: The Constraint You Forgot to Design Around

Every AWS service enforces service quotas (historically called limits) per account and per region. They come in two flavors, and the difference matters more than the number itself.

A soft limit is adjustable: you request an increase through the Service Quotas console, the Service Quotas API, or a support case, and AWS grants it based on your use case. A hard limit cannot be raised at all — you must redesign around it. Concurrency of concurrent executions of a Lambda function, the number of EC2 vCPUs you can run in a given instance family, and the number of IAM roles in an account are all soft limits with low-ish defaults that new accounts hit quickly. Hard limits are the physical constraints of the API contract: a single SQS message cannot exceed 1 MiB, a synchronous Lambda invocation payload cannot exceed 6 MB, and a single S3 PUT cannot exceed 5 GB. No support ticket changes those.

🤔 Did you know? Not every rate limit appears in Service Quotas at all. S3 request throughput is partitioned per prefix rather than tracked as an account quota — a limit you cannot find by browsing the quota list, and one that has bitten teams doing millions of small uploads.

The practical move is to treat quotas as an architecture input. Ask three questions for every component you design:

1. What is the quota for the dimension that scales here? (concurrency, vCPUs, connections, requests/sec, buckets, rules)
2. Is it adjustable, and how long does an increase take?
3. Where in the design does hitting that quota first show up?

That third question is the one people skip. When Lambda's default of 1,000 concurrent executions per region is exhausted, the symptom is not a clean error in your dashboard — it is throttling, where invocations are rejected or retried and your downstream queue backs up. When the EC2 On-Demand vCPU limit for a family is exhausted, an Auto Scaling group simply stops adding capacity mid-traffic and reports a capacity error you have to be watching to see. Both are minutes of work to prevent and hours of support-ticket time to fix at the wrong moment.

# 1. Which quotas exist for a service, and which can be raised?
aws service-quotas list-service-quotas \
  --service-code lambda \
  --query "Quotas[?Adjustable].[QuotaName,Value,QuotaCode]" \
  --output table

# 2. Resolve the quota code by name, then inspect it in the CURRENT region
QCODE=$(aws service-quotas list-service-quotas \
  --service-code lambda \
  --query "Quotas[?QuotaName=='Concurrent executions'].QuotaCode" \
  --output text)
aws service-quotas get-service-quota \
  --service-code lambda --quota-code "$QCODE" \
  --query "Quota.[QuotaName,Value,Adjustable]"

# 3. Request an increase — asynchronous, not instant. Budget days, not minutes.
aws service-quotas request-service-quota-increase \
  --service-code lambda --quota-code "$QCODE" --desired-value 5000

The first command answers "what limits exist." The second answers "what is this one set to right now, in this region" — note the region, because quotas are scoped per region and an account that has never deployed to eu-west-1 has untouched defaults there. The third is the important operational lesson: quota increases are a service-request workflow with review, not a config toggle. If your launch plan assumes 10,000 concurrent Lambdas on day one, the quota request belongs in the week before launch, not in the incident channel.

A cheap, high-leverage habit: run these checks as a preflight step in your deployment pipeline. A five-second quota query in CI catches the class of failure where the code is perfect and the platform refuses to run it.

⚠️ Common Mistake: Assuming a quota you raised in one account applies to the next account you create. Quotas are per account, per region. Multi-account structure, covered below, multiplies your quota surface rather than sharing it.

Tagging as an Operating System, Not Decoration

A tag is a key-value pair attached to a resource. It costs nothing, it is not a security boundary, and it is the single cheapest mechanism you have for ownership, automation, and cost attribution. The failure mode is not "we forgot tags on one resource" — it is "we have forty ad-hoc tag keys, half of them misspelled, and Cost Explorer can group by none of them."

Start with a small required tag set (four to six keys is plenty): Owner, Environment, Application, CostCenter, and optionally DataClassification for anything touching regulated data. Small sets survive; large sets drift. The moment a tag key is optional in practice, some percentage of your resources will lack it, and the whole attribution scheme becomes approximate.

The tag-on-create pattern means the tags are part of the same API call that creates the resource, so there is never a window where a billable resource has no owner. The catch: the syntax differs by service.

# EC2: tags travel with the create call via tag-specifications
aws ec2 run-instances \
  --image-id ami-0abcdef1234567890 \
  --instance-type t3.micro \
  --count 1 \
  --tag-specifications \
    'ResourceType=instance,Tags=[{Key=Owner,Value=team-payments},{Key=Environment,Value=dev},{Key=CostCenter,Value=cc-4471}]'

# DynamoDB: same intent, different shape — a flat tag list, not tag-specifications
aws dynamodb create-table \
  --table-name orders \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --tags Key=Owner,Value=team-payments Key=Environment,Value=dev

# S3: a third shape — the tags ride inside the bucket configuration,
# next to the LocationConstraint (the caller also needs s3:TagResource).
# A bucket that already exists is tagged with put-bucket-tagging instead.
aws s3api create-bucket --bucket my-app-dev-logs --region us-west-2 \
  --create-bucket-configuration \
    'LocationConstraint=us-west-2,Tags=[{Key=Owner,Value=team-payments},{Key=Environment,Value=dev}]'

Three services, three shapes for one intent: that is the reason tagging discipline belongs in infrastructure-as-code rather than in a runbook. Wherever the create and the tag end up as two separate human steps, some fraction of resources will be created during an incident and never tagged. When they are one template, atomicity is free. The condition keys that let you deny creation of an untagged resource exist, but they are policy design — the IAM lesson owns those details; here you only need to know they exist as an enforcement backstop.

Two operational details make tags actually power things:

• Cost allocation tags must be activated. A user-defined tag does not appear in Cost Explorer or your billing reports until you activate it as a cost allocation tag in the Billing console, and activation takes hours to take effect, not seconds. Activate your required keys once, early: until you do, your cost data cannot be grouped by them, and the way back is a backfill request that reaches at most twelve months.
• Tag policies in AWS Organizations can standardize key casing and allowed values across accounts, reporting or blocking non-compliant resources. Without them, Owner, owner, and OwnerName become three different keys across four accounts.

Automation built on tags is where the investment pays back. The Resource Groups Tagging API finds resources that violate your scheme; a scheduled Lambda that stops Environment=dev instances outside business hours cuts a real percentage off the bill for zero engineering risk; Cost Explorer grouped by Application turns a single opaque number into a per-team conversation.

⚠️ Common Mistake: Putting sensitive values in tags. Tags propagate into billing reports, resource listings, and audit logs. A Secret tag or an owner's personal phone number is visible far more widely than the resource itself.

Cost Signals and the Trade-offs Nobody Prices

The bill is a signal with a delay. Cost Explorer and billing data lag usage by hours, sometimes a day, so if your only cost tool is a report you read on Monday, you find out about a runaway loop after it has run all weekend. The guardrail is AWS Budgets: a monthly cost budget with an alert at, say, 80% of actual spend and another at 100% of forecasted spend, delivered to a distribution list rather than one person's inbox. Forecast-based alerts are the ones that actually save you, because a forecast crosses the threshold before the money is spent.

# budget.json: {"BudgetType":"COST","TimeUnit":"MONTHLY",
#  "BudgetLimit":{"Amount":"200","Unit":"USD"},"BudgetName":"monthly-200"}
# notifications.json: ACTUAL > 80% and FORECASTED > 100%, subscriber = email list
aws budgets create-budget \
  --account-id 111122223333 \
  --budget file://budget.json \
  --notifications-with-subscribers file://notifications.json

# Which tag values are actually driving this month's spend?
aws ce get-cost-and-usage \
  --time-period Start=$(date -u +%Y-%m-01),End=$(date -u +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --group-by Type=TAG,Key=Application

Now the pricing models, at the level you need to make a foundational choice:

The rule that keeps you out of trouble: commit only to a baseline you have already observed, and cover roughly the floor of your usage, never the peak. A Savings Plan on a workload whose shape you don't know yet converts a variable cost into a fixed one and leaves you paying for capacity you stopped using.

Then there are the costs that never appear in the pricing page headline:

• Data transfer is directional and asymmetric. Traffic into AWS is generally free; traffic out is billed — to the internet, across regions, and across Availability Zones in both directions. A chatty microservice split across three AZs pays a cross-AZ tax on every call, forever.
• NAT Gateway bills twice: an hourly charge plus a per-gigabyte processing charge. Every byte your private subnet sends to S3 through a NAT Gateway pays that processing charge. A VPC Gateway Endpoint for S3 and DynamoDB is a free route that removes that traffic entirely — the VPC lesson covers the routing mechanics; the point here is that this one architectural choice can dominate a small account's bill.
• Requests cost money, not just storage. S3 GET/PUT/COPY are billed per thousand requests, and millions of tiny objects can cost more in requests than the bytes they hold. This cuts both ways: a lifecycle rule that transitions millions of small objects to a colder storage class pays a per-object transition charge and may cost more than it saves.
• Cheaper services often carry higher operational cost. Self-managing PostgreSQL on EC2 instead of using a managed relational database looks cheaper on a spreadsheet, and it is — until you price the engineer-hours for patching, backup verification, and failover runbooks. Likewise, Lambda billed per request is often more expensive than a small always-on instance for a steady 24/7 workload, because you are paying a premium for elasticity you never use.

🎯 Key Principle: the cheapest service is not the lowest cost. The lowest cost is the one where total spend — compute, transfer, requests, and human attention — is minimized for the workload's actual shape.

⚠️ Common Mistake: Optimizing for the monthly bill while ignoring unit economics. Cost per request, per order, or per active customer is the number that survives growth; a total that looks large but is falling per unit is healthy, and a small total that scales linearly with traffic is not.

Blast Radius: Containment You Choose Before You Need It

Blast radius is how much breaks when someone — or some automation — makes a mistake. It is an engineering choice, and the ladder is well defined:

Isolation ladder (blast radius shrinks downward)

One account, one region
  └─ everything shares quotas, IAM, and the bill
↓
Separate environments as separate ACCOUNTS
  └─ a sandbox cannot consume production quota
↓
Accounts + AWS Organizations SCPs
  └─ guardrails cannot be disabled by the workload owner
↓
Accounts + SCPs + per-account budgets with actions
  └─ runaway spend stops itself

Separate accounts per environment is the single highest-value containment decision, and it is cheap. Beyond the obvious (a destructive action in dev cannot touch prod), separate accounts give you independent quota budgets, independent bills, independent IAM principals, and — importantly — a clean way to delete an environment entirely without picking apart a shared one. AWS Organizations groups those accounts into organizational units and applies Service Control Policies (SCPs): organization-level guardrails that deny actions regardless of what an identity policy elsewhere allows. Useful defaults include denying the ability to leave the organization, denying attempts to disable audit trails, denying root-user API calls, and denying work in regions you don't operate in. SCPs are a ceiling on permissions, not a grant — the least-privilege details below that ceiling belong to the IAM lesson.

Budgets with actions turn a notification into a guardrail. A budget action can apply an SCP to a target or stop EC2 and RDS instances when a threshold is crossed. Give sandbox accounts a small budget with a stop action at 100%: the account fences itself, and the person who left a GPU instance running learns the lesson from an email rather than from finance.

CloudTrail is the audit half of containment. An organization trail writing to a central, locked-down S3 bucket in a dedicated log-archive account records who called which API, from where, and when — across every account, including accounts created later. Two settings worth turning on from day one: multi-region coverage (so a workload in an unexpected region is still recorded) and log file validation (so tampering is detectable).

⚠️ Common Mistake: Treating CloudTrail as monitoring. CloudTrail answers "who changed this?" after the fact. It does not tell you the system is unhealthy, and data events — object-level S3 and Lambda invocation records — are billed by volume. Enable management events everywhere; enable data events selectively on your most sensitive buckets.

Baseline Observability as a Default, Not an Afterthought

There are four things a new account should have before its first real workload, and none of them are optional add-ons.

CloudWatch metrics are mostly automatic. Managed services publish a baseline set of metrics without configuration; EC2 publishes basic metrics at five-minute resolution, with one-minute detailed monitoring as an opt-in that costs extra. Metrics answer "is it healthy right now?" — and their typical gap is granularity, which is why a burst of errors that resolves in ninety seconds can be invisible to a five-minute datapoint.

CloudWatch Logs are not automatic in most cases, and this is the trap. A Lambda function writes to /aws/lambda/<function-name> only if its execution role is allowed to create that log group; if the permissions are missing, log events are silently discarded and the function appears to do nothing. EC2 instances need the CloudWatch agent installed and configured before any application log reaches CloudWatch. ⚠️ And log groups have no retention limit by default — they retain data forever and you pay for it. Set an explicit retention period on every log group you create, including the ones you create by hand while debugging.

CloudTrail supplies the audit layer described above, plus 90 days of searchable management event history in the console at no extra configuration.

AWS Budgets supplies the financial layer — the alert that fires before the bill does.

For a foundational workload, four alarms are enough to start: a billing alarm tied to your budget, an error-rate alarm on your primary compute, a queue-depth or dead-letter-queue alarm on anything asynchronous, and a storage-headroom alarm on your primary database. That short list catches the failures that actually happen early — a code deploy that starts erroring, a consumer that fell behind, a disk filling up — without drowning you in notifications you learn to ignore.

A useful framing: observability is the mechanism that makes every other guardrail visible. Quotas fail loudly only if you are watching capacity; tags pay off only if you look at grouped cost; budgets protect you only if the alerts reach someone who can act. Configure all four baselines before the workload is interesting, and the guardrails stop being insurance you hope you never need and start being the feedback loop that tells you what to do next.

That feedback loop is exactly what the final section's readiness checklist is designed to verify, item by item, before you move deeper into IAM, VPC, and the service-specific lessons.

Common Pitfalls and a Foundation Readiness Checklist

Every pitfall in this section has the same shape: something works once, so the work that would make it repeatable never gets done. The console flow produces a running service and no rebuild path. The resource exists, but the region and account holding it live only in someone's memory. None of these mistakes announce themselves when you make them — they surface at an incident, a bill, or the moment you try to stand up a second environment.

Pitfall 1: The Console-Only Workflow

The console is not a lesser tool. It is a client of the same control-plane APIs your scripts call: the button labelled "Create bucket" issues a CreateBucket request with parameters you could have typed into the AWS CLI. That equivalence is exactly why the console is a superb place to learn a service and a bad place to keep the only copy of a working configuration.

⚠️ Common Mistake: Building a working proof of concept entirely in the console and moving on to the next service. Nothing in that environment records what was created, in what order, or with which options — and the write-up you meant to do later never happens.

What breaks, concretely:

• 🔴 No diff. A console session can't be reviewed before it runs or compared afterwards.
• 🔴 No teardown. Creation is one click; cleanup is a memory exercise with a bill attached.
• 🔴 No second environment. Rebuilding dev into staging is retyping, so staging drifts from dev the day it's created.
• 🔴 Hidden coupling. A bucket policy that works only because of an account-level setting someone toggled months ago.

You click "Create bucket" in the console
        ↓
The console calls the same control-plane API your CLI would call
        ↓
The resource now exists in ONE account and ONE region
        ↓
Nothing in version control, no tags, no named owner, no rebuild path
        ↓
Six months later: "which bucket is this, who made it, can we rebuild it?"

The fix is not "never use the console." It is: every resource that outlives the session gets a scripted equivalent, even when that equivalent is three CLI commands in a text file. Two mechanisms make this cheap.

First, the API call you already made is in CloudTrail. The regional event history retains management events for the last 90 days, including request parameters:

# Recover what a console session actually created.
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateBucket \
  --max-results 10 \
  --output json > /tmp/ct.json

python3 - <<'PY'
import json

with open('/tmp/ct.json') as fh:
    events = json.load(fh)['Events']

for item in events:
    event = json.loads(item['CloudTrailEvent'])   # the raw API request
    params = event.get('requestParameters') or {}
    print(event['awsRegion'], params.get('bucketName'), event['userIdentity'].get('arn'))
PY

Read that output as a replay recipe. bucketName plus awsRegion is enough to write aws s3api create-bucket --bucket <name> --region <region> (plus --create-bucket-configuration LocationConstraint=<region> for any region other than us-east-1). You are not reverse-engineering anything: you are reading back the request the console already sent. Note the boundary, though — CloudTrail event history covers management events. GetObject-style data-plane calls are not in it unless a trail is configured to log data events, which is a distinction worth carrying from the control-plane-vs-data-plane model.

Second, tag every resource at the moment it is created so unmanaged resources become findable instead of forgotten. The Resource Groups Tagging API answers "what exists in this account and region that I own?"

aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Owner,Values=you@example.com \
  --region us-east-1 \
  --query 'ResourceTagMappingList[].ResourceARN' \
  --output text

Anything missing from that list is either not yours or not tagged — both facts you want to know before someone asks. (What tags to require and how to wire them into chargeback is the subject of "Guardrails for Foundations: Quotas, Tagging, Cost Signals, and Blast Radius.")

💡 Mental Model: The console is a REPL, not a repository. Explore in it freely; the moment something works, promote it to a script and treat the console copy as disposable.

Pitfall 2: Region and Account Confusion

Most AWS services are regional: a resource created in one region is not visible from another, and the same object does not exist twice. The programmatic-access lesson covers the resolution order for region and credentials; the pitfall is what that resolution produces when you never look at it.

CI runner exports AWS_DEFAULT_REGION=us-west-2  (set once, forgotten)
        ↓
Your script calls create-* with no --region flag
        ↓
The resource lands in us-west-2 while your team works in us-east-1
        ↓
describe-* from a laptop in us-east-1 returns an EMPTY list, exit code 0
        ↓
"The resource disappeared." It did not — you are reading a different region.

That last line is the part people underestimate. For most list and describe calls, a wrong region and a nonexistent resource are indistinguishable: the API returns an empty result and a success exit code. You get no error to grep for. The same is true of accounts — every AWS account is a separate namespace with its own resource IDs, quotas, and bill.

⚠️ Common Mistake: Trusting the terminal you are sitting in. A profile named prod that points at a sandbox account, or a shell with AWS_PROFILE exported an hour ago, will happily accept every command you type.

Identity is one command. aws sts get-caller-identity returns the account, the user ID, and the principal ARN actually being used. Make it a reflex before any mutating call. aws configure list complements it by showing the effective region and the source it came from (environment variable, config file, or role), which is the information you need when a value came from somewhere you forgot.

Hardcoded ARNs are the second half of this pitfall. The ARN anatomy from "AWS as an API-First Platform" embeds partition, service, region, account ID, and resource. Any ARN with a literal 123456789012 in your source code works exactly until the first cross-account or multi-region deployment, then fails in a way that looks like a permissions bug. The habit that removes this class of failure entirely is: read identity-bearing values from the API instead of writing them into strings. The queue ARN in the lifecycle drill further down comes back from GetQueueAttributes, not from a template literal.

⚠️ Cross-region traffic is billable and invisible until you look. An S3 bucket in one region read by a Lambda function in another is a cross-region data transfer charge on every invocation, and a NAT gateway in the wrong subnet creates cross-availability-zone charges for traffic that "stayed inside the VPC." The trade-offs themselves belong to the guardrails section; the pitfall here is that the cost decision was made accidentally, by an unset region variable.

Here is a preflight guard worth dropping at the top of every mutating script:

#!/usr/bin/env bash
# preflight.sh - fail fast before any mutating AWS call
set -euo pipefail

: "${EXPECTED_ACCOUNT:?export EXPECTED_ACCOUNT=<12-digit account id>}"
: "${EXPECTED_REGION:?export EXPECTED_REGION=<e.g. us-east-1>}"

# Who are we, really? Resolved from env, profile, or instance/task role.
actual_account="$(aws sts get-caller-identity --query Account --output text)"

# Where would a create call land? This reflects the CLI's own resolution and
# reports the source (env, config-file, ...), so '<not set>' fails closed.
# Current CLI v2 prints "region : us-east-1 : env : ..."; older releases print no colons.
config="$(aws configure list)"
actual_region="$(printf '%s\n' "$config" | awk '$1 == "region" { print ($2 == ":" ? $3 : $2) }')"
region_source="$(printf '%s\n' "$config" | awk '$1 == "region" { print ($2 == ":" ? $5 : $3) }')"

if [ "$actual_account" != "$EXPECTED_ACCOUNT" ]; then
  echo "Wrong account: targeting $actual_account, expected $EXPECTED_ACCOUNT" >&2
  exit 1
fi

if [ "$actual_region" != "$EXPECTED_REGION" ]; then
  echo "Wrong region: '$actual_region' (source: $region_source), expected $EXPECTED_REGION" >&2
  exit 1
fi

echo "preflight ok: account=$actual_account region=$actual_region source=$region_source"

Run it against a shell with no region configured and it exits non-zero — the failure you want, at the start, instead of a silent write into the wrong place.

🤔 Did you know? S3 bucket names are globally unique, but a bucket itself lives in exactly one region. So "the bucket" is simultaneously global (while the bucket exists, nobody else in the partition can use the name) and regional (the data, the policy, and the charges are pinned to one place). That split is why S3 confuses people who treat all AWS resources as uniformly regional.

Pitfall 3: Credential and Permission Anti-Patterns

Three credential mistakes recur. Remediation in depth — scoped policies, roles, Identity Center, permission boundaries — belongs to the IAM material; the recognition patterns belong here.

⚠️ Mistake 1: Long-lived access keys that escape the credential store. An IAM user access key pasted into a CI secret, a Lambda environment variable, a README, or a screenshot does not expire on its own. A role's temporary credentials do. If a static key is the only thing making a pipeline work, the pipeline is one leak away from an incident. Treat any credential that appears somewhere other than a credential store as compromised and rotate it.

⚠️ Mistake 2: Over-broad policies, justified by velocity. "Action": "*", "Resource": "*" works, which is the problem. The real cost is not the security exposure alone — it's that the policy hides your dependency map. A script running under an admin profile tells you nothing about which permissions it actually needs, so the day you scope it down, you are doing archaeology instead of editing a list. The failure is diagnosable: your script works for you and fails for every teammate, and no one can say which action was missing.

⚠️ Mistake 3: Skipping sts get-caller-identity. This is the cheapest mistake to avoid and the most expensive to discover late, because "I edited the wrong account" and "my policy is wrong" produce identical error messages. One command separates them.

🔒 Note that identity checks and permission design solve different problems: the check tells you who you are right now; the policy decides what that principal may do. Confusing the two leads to "it worked for me" debugging, which has no resolution.

Pitfall 4: Treating Managed Services as Interchangeable

The capability map earlier gave you cues for choosing a service family. This is the failure side of that: patterns that look reasonable, work at small scale, and quietly become the wrong tool.

⚠️ Mistake 1: S3 as block storage. People mount S3 as a filesystem, or use it as a queue by listing objects and deleting them. The mechanics break it: S3 objects are written whole, there is no partial byte-range write, no file handle, no lock or seek. Changing one byte of a 1 GB object means rewriting and re-uploading the whole object. Large sequential writes are fine — that is what multipart upload is for — but databases, append-heavy logs, and continuously modified files are not. Block and shared-file needs go to EBS and EFS. (Consistency is no longer the reason to avoid S3: object operations have strong read-after-write consistency, so "I read a stale object" is a much rarer failure than the write-amplification problem above.)

⚠️ Mistake 2: DynamoDB for ad-hoc relational queries. There are no joins, and the escape hatch is Scan, which reads and bills for every item in the table. A Scan on a dev table with a few hundred items returns in milliseconds and costs nothing noticeable; the same access pattern at production volume becomes the dominant line in both the latency graph and the bill. DynamoDB wants access patterns named before schema design, which is the whole premise of single-table design. If your honest answer is "we'll figure out the queries later," that is a relational requirement in disguise.

⚠️ Mistake 3: Lambda for long-running processes. A function invocation is capped at 15 minutes (the exception is a function on Lambda Managed Instances, where asynchronous invocations and most event source mappings may run up to 90 minutes). When a job genuinely needs 40, "just raise the timeout" does not exist as an option; you either chunk the work so each invocation fits the cap and hands state to the next, or you pick a runtime without the cap. Beyond duration, the same wrong-tool pattern shows up with long-lived connections, large resident memory working sets, and CPU-heavy batch work: those belong to Fargate, Batch, or EC2. The test is one sentence: if a hard 15-minute stop changes your design, you are not building a Lambda-shaped task.

⚠️ Mistake 4: One queue where you needed a fan-out. An SQS queue delivers each message to one consumer. Sending the same event to five independent consumers by adding five readers to one queue means each consumer sees a random fifth of the traffic. Fan-out is a different service choice, and replayable streams are a third — the messaging cues in the capability map decide among them.

⚠️ Mistake 5: Reading "managed" as "nothing is mine." A managed service moves undifferentiated operational work to AWS; it does not move your data model, your access patterns, your capacity mode, or your failure handling. That boundary between what AWS operates and what remains yours is the shared-responsibility split covered in the capability-map section — the pitfall is discovering it while debugging rather than while designing.

Pitfall 5: Quotas, Tagging, and Cost Visibility Come Last

All three of these get deferred for the same reason: none of them hurt today. They hurt on the day you need the number.

⚠️ Mistake 1: Discovering a quota in production. Service quotas are applied per account and per region, and they are the reason "it worked in dev" stops being true at scale. The instance fleet that cannot launch its next instance, the function that suddenly throttles under concurrency, the vCPU ceiling that blocks a scale-out — these are quota events, and they arrive exactly when demand is highest. Inspecting your current limits and planning for them belongs to the guardrails section; the pitfall is that the first time you look at that page is during an incident.

⚠️ Mistake 2: Tagging as a later cleanup task. Untagged resources are unattributable: no owner to page, no cost line to explain, and no way to automate against them. The trap is that tagging looks retroactive and is not. A tag you attach today does not re-attribute usage that already happened — it explains future spend, not last quarter's (a cost allocation backfill reaches back at most twelve months, and only for resources that already carried the tag). Every week a resource runs untagged is a week of spend that will never be attributable.

⚠️ Mistake 3: No cost signal until the bill. Without budgets and alerts, the first cost feedback you get is a monthly invoice, which is far too late to connect to a decision. The dangerous part is that a cheaper service can be more expensive overall once you count the operational work it pushes onto you — the wrong trade-off is invisible until somebody prices the on-call time. That trade-off discussion is owned by "Guardrails for Foundations."

🎯 Key Principle: Every resource should be traceable to three things — a script, a tag, and a line on a bill. If any of the three is missing, the resource is a liability you happen to be paying for.

The Foundation Readiness Checklist

Five checks. Each has a pass criterion you can demonstrate in under a minute, and they map onto the mnemonic below: Explain it, Choose it, Script it, Tag it, Bill it.

⚠️ Common Mistake: Treating check 3 as "write a create script." A create-only script is a console session with a shebang. The lifecycle includes verification and teardown, and it must survive being run twice.

Here is the whole drill, executed against SQS. Read it as a template, not as SQS trivia — the same shape applies to a bucket, a table, or a log group.

import time

import boto3
from botocore.exceptions import ClientError

REGION = 'us-east-1'
NAME = 'foundations-drill'

sqs = boto3.client('sqs', region_name=REGION)

# 1. CREATE - re-running with the same name and attributes returns the
#    existing queue URL instead of erroring or creating a duplicate.
def create_queue() -> str:
    return sqs.create_queue(
        QueueName=NAME,
        Attributes={'MessageRetentionPeriod': '86400'},   # seconds, as a string
    )['QueueUrl']

try:
    queue_url = create_queue()
except ClientError as exc:
    # SQS refuses a name it deleted less than 60 seconds ago: the teardown
    # of the previous run. Wait it out once, then create.
    if 'QueueDeletedRecently' not in exc.response['Error']['Code']:
        raise
    time.sleep(60)
    queue_url = create_queue()
print('queue:', queue_url)

# 2. STAMP - tag in the same run as creation, while you still know the owner.
#    (create_queue also accepts tags=..., which is true tag-on-create.)
sqs.tag_queue(
    QueueUrl=queue_url,
    Tags={
        'Owner': 'you@example.com',
        'Environment': 'sandbox',
        'CostCenter': 'learning',
    },
)

# 3. VERIFY - control plane (attributes) plus data plane (a real message).
sqs.send_message(QueueUrl=queue_url, MessageBody='hello from the drill')
attrs = sqs.get_queue_attributes(
    QueueUrl=queue_url,
    AttributeNames=['QueueArn', 'ApproximateNumberOfMessages'],
)['Attributes']
print('arn:', attrs['QueueArn'])                      # read, never hardcoded
print('visible now:', attrs['ApproximateNumberOfMessages'])

# 4. TEAR DOWN - the half everyone skips.
sqs.delete_queue(QueueUrl=queue_url)

Three details matter more than the syntax. The ARN comes back from GetQueueAttributes rather than being composed from a hardcoded account ID, which is the remedy for Pitfall 2. ApproximateNumberOfMessages is exactly what its name says — expect it to trail the send_message by a moment, so a verification script that asserts on it immediately will flake. And create_queue is idempotent for a matching name and attribute set: a second run returns the same queue URL, which is precisely the property you want in automation and precisely the property a console session never gives you. The QueueDeletedRecently branch is there because this drill deletes its queue: SQS will not re-create a deleted name for 60 seconds.

📋 Quick Reference Card: preflight for any mutating script

Next Steps

Three things to do before the IAM lesson, each about thirty minutes.

1. Run the five-check drill on a service you have only ever clicked. Pick a queue, bucket, or table, write the create-verify-tag-delete script, run it twice, then confirm the resource appears in a tag-filtered inventory. If the second run fails, you have found a real portability problem early and cheaply. (One failure is expected and is not a portability problem: SQS refuses to create a queue for 60 seconds after a queue of that name was deleted.)

2. Retrofit a preflight guard onto one existing script. The account and region checks from Pitfall 2 are the highest-leverage ten lines you can add to any automation, because they convert a silent wrong-target write into a loud, immediate failure.

3. Answer the identity question out loud. For one workload you have built, state the account, the region, the ARN, and the owner without opening the console. Gaps in that sentence are the exact places where the next lesson's identity and networking work will pay off.

⚠️ Remember: AWS will always let you make the mistake. It returns an empty list instead of an error for a wrong region, accepts "Action": "*" because it is syntactically valid, and bills you for the cross-region traffic you did not know you created. Every guardrail in this section exists because the platform does not supply it for you — you supply it, in the script, before the first mutating call.