CloudWatch Beyond Metrics

Finding and Reading Logs: Where Each Service Writes and How to Tail Them

You deploy a function, send a request, and get a 500. You open CloudWatch to find out why, and there is nothing there. No error and no trace of the request. Before you can debug anything, you have to know where each AWS service is supposed to write its logs and which setting or permission stops it from doing so.

This section builds that map for Lambda, ECS on Fargate and API Gateway. It then shows how to read the logs quickly from the terminal with aws logs tail and filter patterns, and how to stop retention defaults from quietly inflating your bill.

Three nouns that organize everything

• A log event is a timestamp plus a message: one line your code printed, or one record a service emitted.
• A log stream is the ordered sequence of events from one source, such as one Lambda execution environment (the sandbox that runs your function code) or one container.
• A log group is the container of streams. Retention, encryption (KMS key) and access permissions are set on the group, so every stream inside it shares them.

log group  /aws/lambda/orders-api      <- retention, KMS, IAM live here
   ↓
log stream  one per execution environment
   ↓
log event   timestamp + message

Prediction task: where do the logs land?

Before reading further, commit to answers. You have three things deployed:

1. A Lambda function named orders-api.
2. An ECS task (one running copy of a task definition) on Fargate (AWS-managed container compute). Its container is named worker, and it uses the awslogs log driver, which ships the container's stdout and stderr to CloudWatch Logs. The driver options are awslogs-group=/ecs/orders-worker and awslogs-stream-prefix=orders.
3. A REST API in API Gateway with a stage named prod. A stage is a named deployment of the API. The API's ID is a1b2c3. Nobody has touched its logging settings.

Task: For each one, write the log group name and the stream name pattern. One of the three will have no log group at all. Say which and why.

Lambda: automatic, if the role allows it

Lambda's runtime captures what your code prints and sends it, by default, to /aws/lambda/<function-name> (a function's logging configuration can name a different group). Each execution environment writes to its own stream, so concurrent invocations scatter across several streams. This is why you search at the group level rather than guessing a stream.

The usual cause of "no logs" is that the function's execution role (the IAM role the function assumes when it runs) lacks logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents. The managed policy AWSLambdaBasicExecutionRole grants exactly these. Lambda does not fail your invocation when log delivery is denied, so the function succeeds silently and logs nothing.

ECS on Fargate: you wire it up

Nothing is logged until the container definition says where. Here is the relevant fragment of a task definition:

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/orders-worker",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "orders"
  }
}

The stream name becomes orders/worker/<task-id>, where worker is the container's name in the task definition. On Fargate the prefix option is required.

The permissions belong on the task execution role, which the ECS agent uses to pull images and write logs. They do not belong on the task role, which your application code uses for its own AWS calls. The execution role needs logs:CreateLogStream and logs:PutLogEvents. If you set awslogs-create-group to "true", it also needs logs:CreateLogGroup. Putting the permission on the wrong role is a classic mistake because both roles look similar in the console.

API Gateway: two log types, both off by default

A REST API has two separate kinds of logs:

• Execution logs record what API Gateway did internally (request validation, integration calls, errors). You switch them on per stage by setting a log level (ERROR or INFO). They go to the auto-named group API-Gateway-Execution-Logs_<api-id>/<stage>.
• Access logs record one line per request in a format you define, written to a log group you create and name.

Both need the account-level CloudWatch Logs role: a setting in API Gateway (per region) that holds the ARN of an IAM role allowing API Gateway to write to CloudWatch. Until it is set, enabling logging on a REST stage fails or produces nothing. HTTP APIs (the lighter, cheaper API type) offer access logs only, with no execution log level.

Tailing from the CLI

The console is slow when you are chasing a live problem. In AWS CLI v2, aws logs tail reads a whole group, merging all its streams:

# Live view of the last 15 minutes, then keep following
aws logs tail /aws/lambda/orders-api --follow --since 15m

# Only events containing the term ERROR
aws logs tail /aws/lambda/orders-api --since 15m --filter-pattern ERROR

# Narrow to streams of one container (prefix match on stream name)
aws logs tail /ecs/orders-worker --since 1h --log-stream-name-prefix orders/worker

A filter pattern is a small matching language the service applies server-side, so you only download matching events. Patterns are case-sensitive.

In a shell, wrap patterns in single quotes so the inner double quotes survive. Filter patterns match events; they do not compute anything. They cannot count errors per minute or average a latency, which is the job of Logs Insights (covered in the next section, 'Structured JSON Logs and Logs Insights: Tracing One Request ID Across Services').

Guided attempt: find a 5xx burst without the console

Assume the prod stage writes JSON access logs to /apigw/orders-prod/access with an unquoted numeric status field.

1. Start wide: aws logs tail /apigw/orders-prod/access --since 30m --filter-pattern '{ $.status >= 500 }' --format short. Note the timestamps. If the lines bunch around one minute, that is the burst.
2. Zoom in: rerun with --since 5m and --follow to see whether it is still happening.
3. Your turn: the access lines show 502 responses. Which log group do you tail next, and with which pattern, to see whether the Lambda behind it raised errors?

Retention: the cost you set once

New log groups never expire, including those that Lambda or awslogs-create-group creates for you. You pay for ingestion (data sent in) and storage (data kept), and ingestion is usually the larger share. Retention only controls the storage part, but a never-expiring group makes that part grow forever.

aws logs put-retention-policy --log-group-name /aws/lambda/orders-api --retention-in-days 14

Pick per group: around 14 days for debug-heavy services, longer where compliance demands it. Create groups yourself (ideally in your infrastructure-as-code) with retention declared, instead of letting services auto-create them.

Independent variation: Fargate shows no logs after deploy

A new orders-worker service deploys, but /ecs/orders-worker shows nothing. Here is what you are given.

"executionRoleArn": "arn:aws:iam::111122223333:role/orders-exec-role",
"taskRoleArn": "arn:aws:iam::111122223333:role/orders-task-role",
"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/orders-worker",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "orders"
  }
}

orders-exec-role allows only ecr:GetAuthorizationToken, ecr:BatchGetImage and ecr:GetDownloadUrlForLayer. orders-task-role allows logs:CreateLogStream and logs:PutLogEvents. Nobody created the log group.

Task: name the two possible causes and justify your fix.

Checklist: Can you name the group and stream pattern for each service? Can you say which role or setting causes missing logs for each? Can you tail a group with a filter pattern? Have you set retention on every group you own?

Structured JSON Logs and Logs Insights: Tracing One Request ID Across Services

A customer says order r-42 "just spun and failed." The request passed through API Gateway, a Lambda function and a container service, so the evidence sits in three log groups (named containers of log streams that share retention and access settings). With the tail-and-filter skills from the previous section, you can search each group one at a time. This section shows how to follow one request through all three in a single query.

The naive approach and where it stops

Searching free-text lines works inside one service. You run aws logs tail with --filter-pattern '"r-42"' (double quotes inside the single quotes, because the term contains a hyphen) and read what comes back. It breaks down across services for three reasons:

• Each service writes the ID in a different place and format (id=r-42, RequestId: r-42, [r-42]).
• You must run three searches and merge the results by eye, comparing timestamps yourself.
• Text has no schema, so you cannot ask for "latency by route" without hand-parsing every line.

The fix is to make the logs queryable by construction.

Structured logging: one JSON object per line

Structured logging means each log event is a single JSON object with consistent field names. A line from the Lambda function might look like this:

{"level":"INFO","message":"charge ok","service":"orders-api","requestId":"r-42","route":"POST /orders","latencyMs":87}

CloudWatch Logs Insights is the query tool for log groups. It auto-discovers top-level JSON fields, so filter level = 'ERROR' works with no extraction step. Nested fields use dot notation, such as ctx.userId. Plain-text events expose only built-in fields such as @timestamp, @message, @logStream and @log (the source log group). Lambda's own report lines also carry discovered fields such as @requestId, @duration, @billedDuration, @memorySize and @maxMemoryUsed. Anything else must be pulled out with parse, covered below.

Here is a minimal Python helper in the Lambda that keeps the schema consistent. It also forwards the ID downstream:

import json
import time
import urllib.request

def log(level, message, **fields):
    # One JSON object per line: Lambda sends stdout to the function's log group
    print(json.dumps({'level': level, 'message': message, 'service': 'orders-api', **fields}))

def handler(event, context):
    # The edge-assigned ID; present in REST and HTTP API proxy events
    request_id = event['requestContext']['requestId']
    start = time.time()
    log('INFO', 'received order', requestId=request_id)
    req = urllib.request.Request(
        'http://payments.internal/charge', data=b'{}',
        headers={'X-Request-Id': request_id, 'Content-Type': 'application/json'},
    )
    urllib.request.urlopen(req, timeout=3)  # pass the ID downstream
    log('INFO', 'charge ok', requestId=request_id, route='POST /orders',
        latencyMs=int((time.time() - start) * 1000))
    return {'statusCode': 200, 'body': '{}'}

Correlation ID propagation

A correlation ID is one identifier attached to every log line produced while serving a single request. Take it at the edge: API Gateway generates $context.requestId, and you can also honor an incoming X-Request-Id header. Then do two things at every hop: put it in every log line, and forward it to the next service.

Client
  ↓
API Gateway   assigns $context.requestId = r-42 (access log)
  ↓
Lambda        reads event.requestContext.requestId, logs it
  ↓  header X-Request-Id: r-42
Container service   reads header, logs it
  ↓  SQS message attribute requestId = r-42
Worker        must read the attribute and log it

The API Gateway access log format is itself JSON. This example uses HTTP API variable names, and REST APIs use others such as $context.resourcePath:

{"service":"api-gateway","message":"access","requestId":"$context.requestId","route":"$context.routeKey","status":$context.status,"latencyMs":$context.responseLatency}

⚠️ The chain usually breaks at queues and async calls. An SQS consumer or an event-driven function receives only what the producer put in the message, so the ID must travel as a message attribute or body field. It does not travel automatically.

Worked trace: rebuild one request's timeline

In Logs Insights, select all three log groups (/aws/apigateway/orders-access, /aws/lambda/orders-api, /ecs/payments) and a narrow time window. Then run:

fields @timestamp, service, level, message
| filter requestId = 'r-42'
| sort @timestamp asc
| limit 50

The story is now clear. The payments service waited about two seconds on the card processor. Notice that the gateway row comes last even though the gateway received the request first, because an access log is written when the response completes. Sorting by @timestamp shows when each line was written, not when the request entered each system.

The commands you will use most often:

Commands chain with |, and each one receives the previous one's output. The same query can run from the CLI. start-query returns a query ID, and get-query-results fetches the rows:

# date -d is GNU date (Linux, CloudShell); on macOS use $(date -v-1H +%s) for the start time
aws logs start-query \
  --log-group-names /aws/lambda/orders-api /ecs/payments \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string "fields @timestamp, service, message | filter requestId = 'r-42' | sort @timestamp asc"

Guided attempt: p95 latency by route

Task: Every orders-api line that carries latencyMs also has a route field. Write a query that shows p95 latency and request count per route in 5-minute bins. Then write a second query for error count per route.

filter service = 'orders-api' and ispresent(latencyMs)
| stats pct(latencyMs, 95) as p95, count(*) as requests by bin(5m), route

filter service = 'orders-api' and level = 'ERROR'
| stats count(*) as errors by bin(5m), route

Parsing unstructured lines

Legacy services often log plain text such as 2026-03-02T10:15:03Z INFO order created id=r-42 took 87ms. parse extracts fields from @message at query time. It supports a glob pattern, where * is a capture slot, or a regular expression with named groups:

parse @message 'id=* took *ms' as requestId, ms
| filter requestId = 'r-42'

parse @message /id=(?<requestId>[\w-]+)/
| filter requestId = 'r-42'

parse works, but it is fragile. If someone rewrites the message to order created (req r-42), the pattern stops matching and the field is silently empty, with no error. The extraction is also repeated in every query by every person. Logging JSON from the start moves that work to write time, where the schema is defined once.

Cost and pitfalls

• ⚠️ Insights bills by data scanned. Choose the narrowest time range and only the log groups you need before you run anything.
• ⚠️ limit does not reduce scanning. It trims the output after the scan, so limit 10 over 30 days costs the same as without it. (The separate limit any 10 form does stop scanning early, but it returns an arbitrary, unordered sample.)
• Multi-line stack traces are often split into one event per line (the awslogs driver does this by default; on Lambda it depends on the runtime and how you log), and then only the first line carries your JSON, so the others have no requestId. Log the stack as a single JSON string field (newlines escaped) instead.
• High-cardinality or huge fields, such as full request payloads, inflate ingestion and every later scan. Log IDs, sizes and status codes instead. (Cardinality, the number of distinct values a field takes, matters even more for metrics, as we'll see in the next section.)

Independent variation: the worker that dropped the ID

The API, a queue worker and a database proxy all handled order o-77. You are given these events, already in time order:

[api]     {"service":"api","requestId":"r-42","orderId":"o-77","message":"order accepted, enqueued"}
[worker]  {"service":"worker","orderId":"o-77","message":"charge failed: timeout"}
[proxy]   slow query 2400ms tag=o-77

Task: Find where the requestId chain breaks. Then write one query over all three groups that rebuilds the timeline for r-42 anyway.

fields @timestamp, service, message
| parse @message /tag=(?<tagId>[\w-]+)/
| filter requestId = 'r-42' or orderId = 'o-77' or tagId = 'o-77'
| sort @timestamp asc

From Logs to Numbers: Metric Filters, Custom Metrics and Embedded Metric Format

Logs Insights can answer almost any question about past events, but an alarm (a rule that watches a number and notifies someone) cannot watch a log search. It needs a metric, a named series of numeric datapoints over time. So the question for each signal is how to turn application behavior into a metric without paying for a metric you never graph. Three routes exist, and they differ in who does the work and what each unit of detail costs.

Route 1  app logs JSON ──→ log group ──→ metric filter ──→ metric
Route 2  app code ──→ PutMetricData API call ───────────→ metric
Route 3  app logs JSON with _aws block ──→ log group ──→ metric (extracted automatically)
                                            └──→ raw event stays searchable in Insights

Route 1: Metric filters on logs you already have

A metric filter is a rule attached to a log group. It uses the same pattern syntax as the filter patterns from the first section. Each time a newly ingested event matches, CloudWatch adds a datapoint to a metric. This turns the JSON error logs into an ErrorCount metric:

aws logs put-metric-filter \
  --log-group-name /aws/lambda/orders-api \
  --filter-name ErrorCount \
  --filter-pattern '{ $.level = "ERROR" }' \
  --metric-transformations \
    metricName=ErrorCount,metricNamespace=OrdersApp,metricValue=1,defaultValue=0

metricValue=1 means every matching event contributes 1. defaultValue=0 publishes a 0 for each one-minute period in which the group receives log events but none of them match, so a service that is logging without errors shows a flat zero instead of gaps. If the group receives no events at all in a minute, nothing is published, and an alarm still sees that minute as missing data.

Predict before reading on: you create this filter at 10:00. Errors were logged at 9:50. What does ErrorCount show for 9:50?

A filter can also pull a number out of the event and set dimensions. A dimension is a name/value label that splits one metric into separate series, such as Route=/orders. This filter publishes latencyMs as the value and splits by route:

aws logs put-metric-filter \
  --log-group-name /aws/lambda/orders-api \
  --filter-name RequestLatency \
  --filter-pattern '{ $.latencyMs = * }' \
  --metric-transformations '[{"metricName":"RequestLatency","metricNamespace":"OrdersApp","metricValue":"$.latencyMs","unit":"Milliseconds","dimensions":{"Route":"$.route"}}]'

Because the metric keeps real values, you can graph p95 or p99 of it later. The pattern { $.latencyMs = * } matches any event that has the field. Filters are a quick win because they need no code change. Their limits are that you can only extract what the log line already contains, and a filter that sets dimensions cannot also use defaultValue.

Route 2: Calling PutMetricData directly

The PutMetricData API publishes a datapoint you compute yourself. It is the right tool when no log line exists to derive the number from, for example a scheduled function that reads a vendor's queue depth every minute:

import boto3
cloudwatch = boto3.client("cloudwatch")

def publish_depth(depth):
    cloudwatch.put_metric_data(
        Namespace="OrdersApp",
        MetricData=[{
            "MetricName": "VendorQueueDepth",
            "Dimensions": [{"Name": "Queue", "Value": "billing"}],
            "Value": depth,
            "Unit": "Count",
        }],
    )

For request-path metrics this route has real costs. Every call is a network round trip that adds latency to the request. It needs an extra IAM permission (cloudwatch:PutMetricData) and its own error handling. Under load it can be throttled, and you can lose datapoints unless you batch and retry. That is a whole new code path to maintain. Use it for metrics with no log line behind them, and prefer Route 3 for everything an invocation or request already knows.

Route 3: Embedded metric format (EMF)

The embedded metric format is a JSON log line with a reserved _aws block that tells CloudWatch which fields to turn into metrics. CloudWatch extracts the metrics as the line is ingested, and the full event stays in the log group. Here is a Lambda handler emitting OrderPlaced and OrderLatency:

import json, time

def handler(event, context):
    start = time.time()
    order = place_order(event)  # your business logic
    latency_ms = (time.time() - start) * 1000

    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "OrdersApp",
                "Dimensions": [["Service", "Route"]],
                "Metrics": [
                    {"Name": "OrderPlaced", "Unit": "Count"},
                    {"Name": "OrderLatency", "Unit": "Milliseconds"}
                ]
            }]
        },
        "Service": "orders-api",
        "Route": "POST /orders",
        "OrderPlaced": 1,
        "OrderLatency": latency_ms,
        "requestId": context.aws_request_id,
        "userId": order["userId"],
        "level": "INFO",
        "message": "order placed"
    }))

The _aws block is the instruction sheet. Dimensions lists which top-level fields become labels, here Service and Route together. Metrics names which top-level fields are values, OrderPlaced and OrderLatency. Every other field, such as userId and requestId, is only a property: it stays in the log event, searchable in Insights, but never becomes a metric label. This example prints one line per invocation, so it costs one log event and no API call. Later you can still run filter userId = "u-123" in Insights against the same line.

Getting EMF to CloudWatch by runtime

The point is that EMF is extracted from whatever lands in a log group. In a container, either run the CloudWatch agent as a sidecar (a helper container in the same task) and point the EMF library at it, or configure the library to write to stdout and let the awslogs driver ship it.

The cost trap: cardinality

Cardinality means the number of distinct values a field takes. CloudWatch bills each unique combination of metric name and dimension values as its own custom metric, and metric filters that set dimensions follow the same rule. Take OrderLatency with a customerId dimension. The dollar figures use an illustrative tiered rate of $0.30 per metric per month for the first 10,000 metrics and $0.10 for the next 240,000 (the rate drops again above 250,000); check current regional pricing.

• 5 customer IDs: 5 metrics × $0.30 = $1.50/month.
• 50,000 customer IDs: 10,000 × $0.30 = $3,000, plus 40,000 × $0.10 = $4,000, giving about $7,000/month.

The code did not change, only the data did. Listing several dimension sets, such as [["Service"],["Service","Route"]], multiplies the count again. ⚠️ Log ingestion is billed on top of whichever route you pick, and these are metric costs only. A dimension should have a small, bounded set of values you would actually put on a dashboard axis.

Decision task

For each case, choose metric filter, EMF or logs-only (Insights queries, no metric), and justify it by cost and query needs.

1. Count 5xx responses from an access log your load balancer already writes.
2. Per-route latency percentiles for a new Lambda with 12 routes.
3. Usage per tenant across 20,000 tenants, mostly for monthly reports and occasional support questions.

Independent variation

A teammate's EMF design for checkout failures is: Dimensions: [["Service","userId"]], Metrics: [{"Name":"CheckoutFailed"}], with 200,000 active users. Rewrite it so a support engineer can still investigate one user's failures without per-user metrics.

Checklist: I can say what a metric filter will not count; I can choose between the three routes; I can read an _aws block; and I can estimate what a dimension costs before shipping it.

Alarms That Page the Right People: Alarms, Composite Alarms and Anomaly Detection

At 03:12 your phone buzzes: CPU is at 82% on a web fleet. Customers are fine and the nightly batch job just started. By Friday the team has muted the channel, and the first real outage goes unnoticed. This section is about building alarms that make the opposite trade: fewer pages, each one meaning a human should act. You will use the metrics you created in 'From Logs to Numbers: Metric Filters, Custom Metrics and Embedded Metric Format' as alarm inputs.

Alarm anatomy

A CloudWatch alarm watches one metric (or a metric-math expression, a formula over metrics such as errors divided by invocations) and moves between three states: OK (not breaching), ALARM (breaching) and INSUFFICIENT_DATA (not enough datapoints to decide). Five settings decide when it moves:

• Statistic: how raw samples are summarized (Sum, Average, p99).
• Period: the length in seconds of each datapoint (60 means one datapoint per minute).
• Evaluation periods (N): how many of the most recent datapoints form the window.
• Datapoints to alarm (M): how many datapoints in that window must breach. This is the M of N rule, and the breaching datapoints need not be consecutive.
• Threshold plus a comparison operator, which define what 'breaching' means.

# Lambda 'checkout': alarm when Errors >= 5 in at least 3 of the last 5 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name checkout-lambda-errors \
  --namespace AWS/Lambda --metric-name Errors \
  --dimensions Name=FunctionName,Value=checkout \
  --statistic Sum --period 60 \
  --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 5 --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:oncall-page

This uses a raw count to keep the flags simple. An error-rate alarm replaces the single metric with a metric-math expression such as Errors divided by Invocations times 100, supplied through --metrics.

Worked trace: 3 of 5 on an error rate

Setup: error rate above 5% counts as breaching, the period is 1 minute, N=5 and M=3. The four minutes before minute 1 were healthy, so the window is full from the start. An incident then hits.

The alarm fires at minute 3, once the third bad datapoint arrives. It recovers at minute 6, three minutes after the last bad datapoint, because the window must slide past the bad minutes. M of N therefore has memory. That memory suppresses flapping, which is repeated flipping between OK and ALARM.

Now hold a continuous outage constant and vary only the period. These times are approximate, since metric publishing adds a delay of a minute or so.

The 5-minute 3-of-5 setting is calm but slow, and a short sharp outage can average below the threshold inside a 5-minute bucket. The 5-minute 1-of-1 setting is faster, but a single noisy bucket pages someone. Choose by asking how many minutes of customer pain you can tolerate before a human knows.

Missing data: when the metric goes quiet

Metrics are only published when something happens. A Lambda function that is not invoked emits no Invocations datapoint, and an alarm needs a rule for those gaps. The treat missing data setting has four options:

• missing (the default): the alarm evaluates only datapoints that exist, and goes INSUFFICIENT_DATA if the whole window is empty.
• notBreaching: a gap counts as a healthy datapoint.
• breaching: a gap counts as a bad datapoint.
• ignore: the alarm keeps its current state.

The right choice depends on what silence means. For an error alarm on a low-traffic function, silence at 4 a.m. is fine, so notBreaching keeps it OK instead of INSUFFICIENT_DATA. For a heartbeat alarm such as 'Invocations < 1' on a function a schedule should trigger every 5 minutes, silence is the failure, so you need breaching. ⚠️ With the default, a dead scheduled job produces no data, the alarm goes INSUFFICIENT_DATA, and nobody is paged. An INSUFFICIENT_DATA alarm only notifies you if you attached an action to that state.

Alarm actions and what deserves one

An alarm can run actions on any state transition. The usual action is publishing to an SNS topic (a pub/sub channel that fans out to email, chat or a paging tool). Alarms can also trigger an Auto Scaling policy or an EC2 action (stop, terminate, reboot or recover an instance).

That split suggests a rule of thumb. Cause metrics like CPU belong in scaling actions, where a machine reacts to a machine. Symptom metrics are what users feel, so they should page people. Typical symptoms are 5xx error rate, p99 latency (the time within which 99% of requests finish) and queue age (how long the oldest message has waited). A CPU alarm that pages with no user impact is the 03:12 problem from the opening.

Composite alarms: pages only on co-occurrence

A composite alarm has no metric of its own. It evaluates a rule over other alarms, using ALARM(name), OK(name), AND, OR and NOT. The child alarms carry no actions, so only the composite notifies anyone.

# Children (checkout-p99-latency, checkout-5xx-rate) are created WITHOUT --alarm-actions
aws cloudwatch put-composite-alarm \
  --alarm-name checkout-slow-and-failing \
  --alarm-rule 'ALARM(checkout-p99-latency) AND ALARM(checkout-5xx-rate)' \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:oncall-page

Now a latency blip alone, or a handful of errors alone, stays silent, and the page means users are both waiting and failing. NOT lets you mute a page during known work, for example ... AND NOT ALARM(maintenance-window-flag). A separate actions suppressor setting can also silence a composite's actions while a named alarm is firing. ⚠️ A composite hides things: if the rule is too strict, a real single-signal outage never pages. Keep a plain static alarm for the failures that must always page, such as a very high 5xx rate.

Anomaly detection: a learned band

An anomaly detection alarm trains a model on a metric's history, draws an expected band around it, and alarms when the metric leaves the band. You set how wide the band is. It suits seasonal metrics that rise and fall on daily or weekly cycles, like request count: 40 requests per minute is normal at 03:00 and alarming at 14:00.

It works poorly in three cases:

• No history. A new metric has nothing to learn from.
• Hard limits. Disk at 90% is bad no matter what the band says.
• Flat lines. The band hugs the line, so tiny wiggles fire constantly.

Bad periods inside the training data teach the model that outages are normal. You can exclude time ranges from training.

What it costs

Exact prices change, so read the current pricing page. The shape is stable:

In units of one standard alarm, 40 anomaly alarms cost about three times as much as 40 standard ones, and only some metrics benefit. A sound plan caps the paging alarms per service, keeps most alarms static, spends anomaly detection on a few seasonal traffic metrics, and uses a composite where it removes pages. Silent child alarms still cost money, so count them.

Guided attempt: the checkout API

Task. The checkout path is ALB (load balancer) → Lambda → DynamoDB. Choose 4 alarms that can notify a human (silent children of a composite are allowed). For each, say static, composite or anomaly, and what it pages for. Available metrics: ALB HTTPCode_Target_5XX_Count and RequestCount, ALB TargetResponseTime p99, Lambda ConcurrentExecutions and Errors, DynamoDB ThrottledRequests. Think about which signals users feel and which have hard limits or seasonality.

Independent variation: 30 pages a week

Task. A web fleet in an Auto Scaling group has an alarm: average CPU > 70%, period 60 s, 1 of 1, paging on-call. It pages about 30 times a week, mostly around 02:00 when a batch job runs. During those pages p99 latency stays under 400 ms and the 5xx rate is about 0.1%. Redesign the alarms. Say what happens to the CPU alarm, which new alarms you add, and whether any is a composite.

Design Task: Cost-Aware Observability for One Service

You are handed a CloudWatch bill and an on-call rotation that has stopped trusting its pager. Fixing both means deciding, signal by signal, what to log, how long to keep it, what to turn into a number, and what deserves to wake someone. This task uses only ideas from the earlier sections: log groups and retention, structured JSON with a request ID, metric filters and EMF, and alarms.

The scenario

The order service has three parts. An API Gateway REST API (stage prod) calls a Lambda function orders-api, which validates the order and puts a message on an SQS queue (a managed message queue). An ECS Fargate service, orders-worker, consumes that queue. Current state:

• orders-api runs at DEBUG level and logs every request body. The worker logs payloads and a line for every health check.
• No log group has a retention setting.
• A latency metric is published with a customerId dimension (300 customers).
• There are 25 alarms. The loudest is orders-worker-cpu-high, which fires several times a week with no user impact. On-call has muted it, and with it every other alarm.

Step 1: Build the cost worksheet

The rates below are illustrative round numbers for the worksheet. Check the CloudWatch pricing page for your Region before quoting a real bill.

Task: compute each driver's monthly cost and rank them. Predict which lever saves the most before you open the answer.

Step 2: Apply the log cost levers

Ingestion is the largest log line item, so cut it at the source:

• Log level per environment. Set LOG_LEVEL=INFO in production and DEBUG only in dev, read from an environment variable.
• Remove payload dumps. Log the order ID, tenant and size instead of the body. Drop health-check lines in the worker.
• Retention per group. Fourteen days for the Lambda and worker groups. Thirty for access logs, assuming that window suits your audit needs.
• Infrequent Access (IA) log class. This class costs less to ingest. It suits logs you rarely query, such as access logs. It does not support metric filters, subscription filters (rules that stream events elsewhere), embedded metric format or Live Tail. It also does not support the GetLogEvents and FilterLogEvents operations, so aws logs tail (which calls FilterLogEvents) cannot read it: you read an IA group through Logs Insights. The class is fixed when the group is created. Never use it for a group you derive metrics from or need to tail.

Step 3: Decide what each signal becomes

For every signal, ask three questions in this order:

A request ID or tenant ID is unbounded, so it stays a log field. "Orders per minute" is bounded, so it becomes a metric. "Customers cannot check out" is a symptom, so it becomes an alarm. Worker CPU fails the third question, so it is a dashboard line at most. This is a starting heuristic. A rare signal can still deserve an alarm when missing it is costly.

Step 4: The harder variation

New constraint: three premium tenants (t-101, t-102, t-103) need per-tenant latency visibility. The custom metric budget cannot grow. Your design currently publishes OrderLatency by Route (4 routes), plus 1 worker error metric, for 5 custom metrics.

Task: choose between EMF with a tenant dimension for the premium tenants only, and Insights queries for everyone else. Then explain how the metric count stays at 5.

import json
import time

PREMIUM = {"t-101", "t-102", "t-103"}

def emit_latency(tenant_id, route, latency_ms, request_id):
    # Service-only metric for everyone; tenant dimension for premium only
    dims = [["Service"]]
    if tenant_id in PREMIUM:
        dims.append(["Service", "TenantId"])
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "Orders",
                "Dimensions": dims,
                "Metrics": [{"Name": "OrderLatency", "Unit": "Milliseconds"}]
            }]
        },
        "Service": "orders-api",
        "TenantId": tenant_id,   # a plain property for non-premium tenants
        "Route": route,
        "OrderLatency": latency_ms,
        "requestId": request_id
    }))

filter TenantId = "t-207"
| stats pct(OrderLatency, 95) as p95, count(*) by bin(1h)

Step 5: Deliverable and model design

Write a one-page design containing:

1. Log groups with retention.
2. The structured log schema, including the request ID field.
3. Three metrics, each with how it is produced.
4. Three alarms, one of them composite.
5. A monthly cost estimate using the Step 1 rates.

Do this before opening the model below. Your alarms and metrics may legitimately differ.

{"level":"INFO","service":"orders-worker","requestId":"c6af9ac6-7b61","message":"order stored","latencyMs":84,"tenantId":"t-207"}

Self-check rubric

• The request ID appears in every service's logs, including across the queue.
• No unbounded dimensions.
• Retention is set on every group.
• Every alarm maps to a user-visible symptom.
• Insights queries are scoped by time range and log group.

Pitfalls to review inside this design

• API Gateway logging is off until you enable it. Check the stage's access log setting, and the account-level CloudWatch role, which a REST API needs for either log type. Without them, edge failures have no request IDs to trace.
• Metric filters count only events ingested after creation. Create WorkerErrors before the incident, and do not expect it to backfill history.
• Missing data. A worker error count goes quiet when nothing fails. Choose notBreaching deliberately, and add a separate alarm if silence from a service that should always emit is itself a failure.
• Insights across every log group. The scan bill is the sum of all selected groups. Select only the order service's groups and a narrow time range.

Final check

Can you rank a bill's drivers by cost? Can you justify each signal as log, metric or alarm? Can you keep tenant visibility without per-tenant metrics for everyone? Can you explain why each of your alarms deserves a page?