Observability at Scale

Observability at Scale

An alert says checkout latency doubled. The global dashboard averages regional averages, the only retained traces are fast requests, and the log filter has silently dropped warnings. More telemetry would not automatically answer what happened. First decide which population each signal represents, how it is transported and where it can be lost.

This lesson develops four applications: bounded metric aggregation, a real tail-sampling collector, deliberate log routing and regional investigation with an anomaly alarm. Each example states its limits. The goal is useful evidence under load, with measurable cost and failure behavior.

1. Start with the question and the population

Metrics summarize measurements, logs record events, and traces connect operations along a request path. These roles overlap; a log line does not necessarily explain why a failure occurred, and a trace may omit uninstrumented work. Correlate signals using a trace/request identifier in logs, a shared service name and clear timestamps. Keep identifiers out of metric dimensions when they would create a new series for every request.

For each critical user journey, define successful outcomes, failure categories, latency measurements and the point where requests are counted. Decide how timeouts, retries and rejected requests enter that denominator. A sampled trace set that preferentially retains errors cannot directly supply an unbiased error rate. Use appropriately collected counters or another explicitly defined population. AWS observability guidance.

Separate the workload from the telemetry pipeline. Collector queues, exporter failures, dropped records, ingestion delay and missing source accounts deserve monitoring too. “No errors found” could mean a quiet service, a bad query, a missing log source or a failed collector. Make a known test request and follow its evidence end to end before trusting a dashboard.

2. Application: aggregate a metric without inventing percentiles

A CloudWatch custom metric is identified by its namespace, name and dimension combination within an account/Region. Individual dimension values can be bounded while their combinations still multiply. For one metric with 20 services, 4 Region labels, 3 tiers and 5 status classes, the full Cartesian product is 1,200 possible series. Actual series depend on combinations actually emitted. Adding one million user IDs gives a theoretical 1.2 billion combinations; it does not prove every combination occurs. Metric dimensions.

Use controlled service/environment/route-template/status values. /orders/{id} is a route template; /orders/813?q=... is not. Logs and traces can carry a necessary diagnostic ID, but still need privacy, retention and indexing limits. There is no universal safe “10,000 series per metric” threshold.

The following accumulator retains four numbers instead of an ever-growing list. It is deliberately single-threaded. The caller supplies one fixed metric identity and invokes flush independently on a 60-second schedule and at graceful shutdown. Recording traffic does not drive the timer, so idle periods do not strand the last measurements. Run concurrent producers through one owning worker or add appropriate synchronization.

import math

class WindowStats:
    def __init__(self):
        self.reset()

    def reset(self):
        self.count = 0
        self.total = 0.0
        self.minimum = None
        self.maximum = None

    def record(self, milliseconds):
        value = float(milliseconds)
        if not math.isfinite(value) or not 0 <= value <= 2 ** 360:
            raise ValueError("Expected a finite nonnegative latency")
        total = self.total + value
        if not math.isfinite(total) or total > 2 ** 360:
            raise ValueError("Window sum exceeds the metric value range")
        self.total = total
        self.count += 1
        self.minimum = value if self.minimum is None else min(self.minimum, value)
        self.maximum = value if self.maximum is None else max(self.maximum, value)

    def flush(self, cloudwatch, timestamp):
        if self.count == 0:
            return False
        cloudwatch.put_metric_data(
            Namespace="CheckoutLab",
            MetricData=[{
                "MetricName": "Latency",
                "Dimensions": [{"Name": "Service", "Value": "checkout"}],
                "Timestamp": timestamp,
                "Unit": "Milliseconds",
                "StorageResolution": 60,
                "StatisticValues": {
                    "SampleCount": self.count, "Sum": self.total,
                    "Minimum": self.minimum, "Maximum": self.maximum
                }
            }]
        )
        self.reset()
        return True

Supply an aware UTC timestamp for the intended window and an authenticated regional client with cloudwatch:PutMetricData. Values 100, 200 and 300 produce count 3, sum 600, min 100, max 300 and average 200 ms. Empty windows publish nothing; that is missing data, not a measured zero. One hundred such observations in a window replace 100 one-observation requests with one call for this example. There is no universal 1,000-fold saving, and metric-storage charges do not fall in the same ratio as call count. Publishing statistics, PutMetricData.

The buffer clears only after a successful call. A timeout may occur after AWS accepted the request; retrying can duplicate observations because this API does not provide a caller idempotency token. In-memory state also disappears on process loss. Choose an exporter/queue and failure policy appropriate to the required accuracy, and measure retries and drops. This class illustrates aggregation, not durable exactly-once delivery.

Min/max/sum/count generally cannot reconstruct p95 or p99. CloudWatch can calculate percentiles from statistic sets only in documented degenerate cases, such as all samples having the same value. When percentiles matter, preserve the distribution: publish raw values or Values/Counts batches, within the API limits, or use a suitable histogram-based telemetry path. Do not average regional p99 values to produce a global p99. Standard-resolution metrics use 60-second storage resolution; setting StorageResolution=1 enables high-resolution publication but does not recreate within-window timestamps already discarded by aggregation. Percentile requirements.

Embedded Metric Format (EMF) is another publication path: a valid structured log event contains _aws, a millisecond timestamp, CloudWatchMetrics, declared dimension sets and the corresponding values. The event must actually reach supported CloudWatch Logs ingestion. Stdout alone does not configure an ECS log driver. EMF avoids an explicit application PutMetricData call; it still incurs log-ingestion/storage and applicable custom-metric charges. Do not put request IDs into EMF dimension sets. EMF specification, EMF costs and behavior.

3. Application: retain useful traces with a real collector

Head sampling decides early, before the complete request outcome is known. It can reduce collection overhead, but it can discard a request that later fails. Tail sampling uses accumulated spans to decide what to retain later. It requires collecting and transporting the candidate spans first; it cannot resurrect spans already dropped by head sampling. Choosing a Python Boolean after a response does not change an already-exported X-Ray trace into a tail-sampled one. OpenTelemetry sampling.

Instrumentation status checked 15 September 2026: the X-Ray SDKs and daemon entered maintenance mode on February 25, 2026. AWS recommends migration to OpenTelemetry. This is not a statement that the X-Ray service has shut down. Existing code may still use segments/subsegments and indexed annotations versus non-indexed metadata; new instrumentation should follow the supported migration path. Metadata is not unlimited, and full request bodies can expose secrets or personal data. AWS migration guidance, X-Ray segment documents.

This local laboratory uses OpenTelemetry Collector Contrib 0.160.0. Save the following complete configuration as collector.yaml. The OTLP HTTP receiver binds only to localhost. The debug exporter prints retained spans locally; it does not send them to AWS. Use synthetic data.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64
  tail_sampling:
    sampling_strategy: trace-complete
    decision_wait: 10s
    num_traces: 1000
    expected_new_traces_per_sec: 50
    decision_cache:
      sampled_cache_size: 10000
      non_sampled_cache_size: 10000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 0.1}
  batch:
    timeout: 1s
    send_batch_size: 512
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [debug]

Validate with otelcol-contrib validate --config=collector.yaml, then run otelcol-contrib --config=collector.yaml. Configure the instrumented test client to export candidate spans to http://127.0.0.1:4318/v1/traces, preserving trace/parent IDs across operations. In this controlled test, collect all candidate spans upstream; a parent-based head sampler inheriting an unsampled flag would undermine the exercise.

Send three synthetic traces: a short successful one, a span marked ERROR, and a trace lasting over 1,000 ms. Wait for the decision window and exporter batching. The error and slow policies retain matching candidate traces; the baseline probabilistically selects 0.1% of the remaining trace population. ERROR is an OpenTelemetry span status, not a test that every HTTP code is at least 400. Verify the instrumentation's status mapping, including client/server semantics and application failures. Pinned tail-sampling configuration, HTTP span status.

The collector's decision timer does not prove all spans have arrived. Late spans, restarts, memory pressure and exporter failure can lose evidence. All spans of a trace must reach the same tail-sampling collector instance; adding replicas behind an ordinary round-robin load balancer breaks that assumption. Use trace-aware routing and monitor dropped-too-early traces, queue pressure and export failures. At 50 new traces/second, a 10-second window contains roughly 500 pending traces in steady traffic; the 1,000-trace limit is headroom for this lab, not a production memory guarantee. Collector scaling.

For an AWS path, replace the local exporter with the pinned distribution's awsxray exporter and its reviewed Region configuration. Supply an appropriately scoped workload role, network connectivity and mapping choices; confirm the exporter is included in the distribution. ADOT and Contrib are different distributions, so verify components/configuration against the one actually deployed. Test a cross-service trace and the resulting X-Ray document before rollout. Cross-Region operations can propagate trace context, while collection and querying remain subject to each backend's regional behavior; “traces cannot cross Regions” is too broad. Pinned X-Ray exporter.

Calculate the retained population

Assume errors are 1% and slow requests are 2% of traffic, with no overlap, and retain both groups plus 0.1% of the remaining requests. The expected retained fraction is:

1% + 2% + (97% × 0.1%) = 3.097%
50,000 requests/s × 0.03097 = 1,548.5 retained traces/s
1,548.5 × 86,400 = 133,790,400 retained traces/day

If 0.5% of all requests are both errors and slow, the union is 2.5%, so the expected retained fraction becomes 2.5975%. Without overlap information, adding 1% and 2% is unjustified. Also, a 0.1% baseline alone at 50,000 requests/second is 50 traces/second, or 4.32 million/day, not 43 million. These are expectation calculations for a defined policy/population, not guarantees of complete error capture or a quoted AWS bill.

Do not call a fixed one-second threshold “p99” without a measured distribution. If using X-Ray's sampling rules in an existing deployment, lower-numbered priorities are evaluated first; a broad rule before a health-check exclusion can shadow that exclusion. Reservoir and fixed-rate settings are not by themselves an error-aware adaptive algorithm. X-Ray sampling rules.

4. Application: route logs before paying for the wrong destination

Take an explicit workload: 1,000 requests/second, 100 lines/request and 500 bytes/line. That produces 100,000 lines/second, 8.64 billion lines/day and 4.32 trillion bytes/day, or 4.32 decimal TB. At an illustrative assumed ingestion price of $0.50 per decimal GB, the calculation is $2,160/day or $64,800 per 30-day month. Real billing depends on Region, log class, volume tiers, units and other charges; use current service pricing. The arithmetic is useful without presenting that unit price as universal. CloudWatch pricing.

Decide which records are needed before ingestion. Shortening CloudWatch retention reduces retained storage, not bytes already ingested. A subscription filter operates after CloudWatch receives logs, so it can reduce downstream indexing but not the original ingestion volume. Export tasks are asynchronous batch exports, not a dependable continuous archive pipeline; CloudWatch recommends subscriptions for ongoing delivery. Subscriptions, S3 export behavior.

Consider this deliberately chosen policy for non-audit application logs. Errors and warnings go to searchable storage; ordinary information goes to a lower-cost archive; debug records are omitted unless a separately authorized diagnostic workflow applies. Unknown formats go to quarantine so parser mistakes are visible. Security/audit records follow their own retention and access requirements and must not be dropped just because they use INFO.

Application --> local router / delivery agent
                    |-- ERROR/CRITICAL --> CloudWatch Logs --> selected downstream index
                    |-- WARN/WARNING --> CloudWatch Logs
                    |-- INFO --> archive delivery --> S3
                    |-- DEBUG --> omit under approved policy
                    `-- malformed/unknown --> quarantine + alert

This pure routing function makes the branches testable. It returns destinations, not successful delivery. Its caller owns serialization, buffering, credentials and output handling:

def destinations(record):
    if not isinstance(record, dict):
        return ("quarantine",)
    if record.get("record_kind") == "audit":
        return ("audit",)
    level = record.get("level")
    if not isinstance(level, str):
        return ("quarantine",)
    level = level.upper()
    if level in {"ERROR", "CRITICAL"}:
        return ("searchable-errors",)
    if level in {"WARN", "WARNING"}:
        return ("searchable-warnings",)
    if level == "INFO":
        return ("archive",)
    if level == "DEBUG":
        return ()
    return ("quarantine",)

In Fluent Bit, implement branching with distinct tags/routes and appropriate outputs. Chaining grep level ERROR, then grep level WARN, then grep level INFO on the same tag filters the same stream successively; it does not create three independent branches. rewrite_tag can emit records under new tags, with deliberate keep/remove behavior. Authenticate OpenSearch outputs using the chosen supported mechanism; an HTTPS hostname alone does not establish AWS authorization. Lambda@Edge is not a generic router for arbitrary host log files. Fluent Bit rewrite_tag, grep filter.

Test one record from each branch, an audit record labelled INFO, malformed data and an unavailable destination. Configure bounded memory, persistent buffers where needed, retries and monitoring for drops. Queuing is not automatically nonblocking: Python's queue.put() waits when a bounded queue is full by default; put_nowait() instead raises Full, which requires an explicit policy. Process shutdown can lose queued records unless the chosen runtime and flush procedure preserve them. Do not promise a fixed logging latency without measuring the actual handler. Python queue semantics.

For diagnostic detail, use an authorized request-scoped decision with expiration and redaction. Do not let an arbitrary HTTP header change a process-global logger to DEBUG for other users. Correlation identifiers help investigation; credentials, tokens and full request bodies generally do not belong in broadly accessible logs.

Cost exercise: suppose 10 decimal TB/day consists of 0.1 TB errors, 0.5 TB warnings, 5.4 TB information and 4 TB debug. Sending only errors/warnings to CloudWatch reduces that ingestion population to 0.6 TB/day, a 94% reduction in bytes. At the same illustrative unit price, its 30-day ingestion component changes from $150,000 to $9,000. This excludes archive delivery/storage, requests, index capacity, transfer, query, retained storage and operations. It is not a 94% reduction in the total observability bill. Shorter warning retention does not halve warning ingestion charges.

5. Application: compare Regions and alert on the intended metric

CloudWatch dashboards can show metrics from multiple Regions; Managed Grafana is another option with CloudWatch and other data sources. Cross-account access requires actual permissions and sharing configuration. A dashboard does not copy telemetry or enable collection. CloudWatch metric streams deliver selected metrics through configured Firehose delivery, not an automatically available cross-Region Kinesis Data Stream. Use the real regional resources and supported destinations when building an export pipeline. CloudWatch dashboards, metric streams.

For comparable, nonoverlapping request populations in the same window, combine sums and counts. Region A has 900 requests averaging 100 ms; B has 100 averaging 1,000 ms. The overall average is (900×100 + 100×1000) / 1000 = 190 ms, not the unweighted 550 ms. Align time windows, units and counting semantics before aggregation. Regional percentiles cannot be merged by this formula.

This helper consumes a complete result from GetMetricStatistics requested with Sum and SampleCount. It avoids assuming returned datapoints are ordered or that every expected period is present:

def observed_average(datapoints):
    rows = sorted(datapoints, key=lambda row: row["Timestamp"])
    count = sum(row["SampleCount"] for row in rows)
    total = sum(row["Sum"] for row in rows)
    return {
        "average": total / count if count else None,
        "sample_count": count,
        "observed_periods": len(rows)
    }

Use a bounded interval and valid period for the metric's age and service limits; GetMetricStatistics is not a paginator. Missing periods remain a coverage question. A five-period window with only three datapoints must not be divided by five as though the missing measurements were zero. When comparing windows, handle a missing or zero baseline rather than blindly dividing by it. GetMetricStatistics.

An anomaly model estimates an expected band from metric history; it does not establish that a deviation is an incident. For the custom metric emitted earlier, this complete alarm request monitors the five-minute Average, in the same account/Region and with the same dimensions. The SNS topic must already exist with appropriate permissions and a working subscription:

def create_latency_alarm(cloudwatch, topic_arn):
    return cloudwatch.put_metric_alarm(
        AlarmName="checkout-latency-anomaly",
        ComparisonOperator="GreaterThanUpperThreshold",
        EvaluationPeriods=2,
        DatapointsToAlarm=2,
        TreatMissingData="missing",
        ThresholdMetricId="band",
        Metrics=[
            {
                "Id": "latency", "ReturnData": True,
                "MetricStat": {
                    "Metric": {
                        "Namespace": "CheckoutLab", "MetricName": "Latency",
                        "Dimensions": [{"Name": "Service", "Value": "checkout"}]
                    },
                    "Period": 300, "Stat": "Average"
                }
            },
            {
                "Id": "band", "ReturnData": True,
                "Expression": "ANOMALY_DETECTION_BAND(latency, 2)"
            }
        ],
        AlarmActions=[topic_arn],
        InsufficientDataActions=[topic_arn]
    )

The band-width argument adjusts sensitivity. Do not infer a guaranteed normal-distribution coverage percentage from it. The alarm condition requires two breaching five-minute datapoints under complete data; it does not promise notification within 60 seconds or exactly ten minutes of incident onset. Model training, period boundaries, missing data and delivery contribute to behavior. Creating an anomaly alarm also creates the associated model; put_anomaly_detector alone creates/configures a detector, not a notification. Anomaly detection, alarm API.

For enrichment, use the alarm's actual metric-math configuration or a reviewed mapping of alarm ARN to source queries. Do not assume every SNS alarm message contains a simple Trigger.MetricName/Namespace pair; metric-math alarms use a different trigger structure. Validate event origin and shape, handle redelivery, and inspect telemetry before attaching a causal explanation. The alert should lead to the owner, runbook, relevant time window and observed coverage. Alarm notification schemas.

Finish with a rehearsal: create a known slow request and a known failure, verify the aggregate counts, follow their trace/log correlation, inspect the chosen regional metric and exercise a telemetry outage. Record what was lost as carefully as what arrived. That gives the on-call engineer a system whose evidence they can interpret under pressure.