Cost & Performance Engineering

Cost & Performance Engineering

A checkout service is cheap only if it can still complete checkouts. An instance running at 15% average CPU might be oversized, waiting on a database, or preserving necessary failure capacity. Your job is to distinguish those cases before changing the bill.

This lesson builds a repeatable optimization loop: define the useful work, find the constraint, test a change, and compare the complete cost of delivering the same outcome. You will practice four applications: a compute fleet, database queries, an archive, and a serverless API. The worked prices are invented teaching inputs, not current AWS quotes.

1. Define success before choosing a discount

Start with a workload contract. For a checkout API, that might be p95 latency below 300 ms, an agreed error-rate limit, and enough capacity to survive an Availability Zone loss. For a batch pipeline, the contract might instead be completing each day's accepted files before 06:00. Measure retries, rejected work and failed outputs alongside successful throughput.

Use a unit metric such as cost per 1,000 successful checkouts. A lower bill caused by serving fewer customers is not an efficiency gain. Compare the same traffic mix, data size, availability requirements and accounting period. Keep a rollback condition tied to customer outcomes.

The AWS Well-Architected Framework has six pillars, including Cost Optimization, Performance Efficiency and Sustainability. Optimizing one dimension can create work in another: a smaller fleet can reduce spare capacity, aggressive caching can serve stale results, and archival storage can lengthen recovery. AWS Well-Architected Framework

Business requirement + current bill
                 |
                 v
Utilization + latency + errors + useful throughput
                 |
                 v
Hypothesis -> controlled change -> comparable measurement
                 |                         |
                 |                 contract still met?
                 |                    /          \
                 |                  yes           no
                 |                   |             |
                 +<-- next review <--+          roll back

Do not stack estimated savings percentages from overlapping recommendations. Resizing an instance changes the baseline on which a commitment discount is calculated. Deleting an unused resource eliminates the same spend that a purchasing recommendation might propose discounting.

2. Choose purchasing options for the measured workload

• On-Demand: useful for uncertain demand and testing a new shape before making a commitment. Separate billing from the availability of capacity when you need it.
• EC2 Reserved Instances: primarily a billing benefit for matching usage. Regional RIs do not reserve capacity; a qualifying zonal RI includes a capacity reservation in its Availability Zone. Payment options include no upfront, partial upfront and all upfront. “Reserved” does not mean every offering is a prepaid physical server. EC2 Reserved Instances
• Savings Plans: commit eligible spend per hour at the plan's rates. Compute plans cover eligible EC2, Fargate and Lambda usage across a broad range of configurations. EC2 Instance plans bind a family and Region. SageMaker AI and Database plans cover their own eligible usage. Verify the selected plan's service coverage, term and payment options; a Compute plan does not simply cover every database bill. Plan types
• Spot: suitable when work can tolerate unavailable capacity and interruption. Diversify compatible pools, checkpoint, retry and budget for recomputation. Stop/termination warnings are normally two minutes; delivery is best effort and hibernation starts without that advance interval. Design for loss even if a warning is missed. Interruption notices

A commitment is still payable during an unused hour. A $4/hour commitment means paying for that committed amount even when only $2/hour of eligible plan-rated usage occurs. Amortize upfront payments over the term when comparing alternatives; comparing only the recurring invoice line understates cost. Savings Plans and RIs do not provide extra compute throughput.

For an interruptible fleet, use a maintained fleet mechanism such as an Auto Scaling mixed instances policy. A single low maximum Spot price can reduce access to capacity; it is not a reservation. For independent batch jobs, AWS Batch can manage scheduling and retries, but the application must produce correct output after a retry. No purchasing model promises an 8× speedup or a fixed total-bill saving.

3. Build cost visibility with the right time scale

Cost Explorer answers which services, accounts and allocated categories drive spending. AWS Budgets can alert on actual or forecasted thresholds and can run configured actions, but billing and notification delays mean an alert is not a hard spending cap. Data Exports CUR 2.0 provides configurable detailed billing data for S3/Athena analysis. Inspect resource-ID and time-granularity settings rather than assuming every exported line identifies a resource. Budgets practices, CUR 2.0 configuration

Activate relevant cost allocation tags in billing as well as applying tags to resources. Assign an owner for unallocated spend and shared costs. Account, service and cost-category views remain useful even when tags are missing. Tagging an EC2 instance does not automatically tag every dependent resource or retroactively repair every historical allocation.

Compute Optimizer recommends resource changes from observed utilization. Trusted Advisor supplies checks whose availability and freshness depend on the check and support plan. Neither is a real-time proof that a resource is safe to remove. Compute Optimizer recommendations, Trusted Advisor coverage

This read-only function retrieves all pages of daily service costs, retaining the service's units and the provisional-data flag. Supply an authorized Cost Explorer client and ISO dates; end is exclusive. Calls can incur API charges.

from decimal import Decimal

def daily_service_costs(ce, start, end):
    request = {
        "TimePeriod": {"Start": start, "End": end},
        "Granularity": "DAILY",
        "Metrics": ["UnblendedCost"],
        "GroupBy": [{"Type": "DIMENSION", "Key": "SERVICE"}]
    }
    while True:
        response = ce.get_cost_and_usage(**request)
        for day in response["ResultsByTime"]:
            for group in day["Groups"]:
                metric = group["Metrics"]["UnblendedCost"]
                yield {
                    "day": day["TimePeriod"]["Start"],
                    "service": group["Keys"][0],
                    "amount": Decimal(metric["Amount"]),
                    "unit": metric["Unit"],
                    "estimated": day["Estimated"]
                }
        token = response.get("NextPageToken")
        if not token:
            break
        request["NextPageToken"] = token

UnblendedCost is the selected metric, not a universal economic definition of cost. Reconcile credits, refunds, shared services and amortized commitments for your decision. Do not add currencies or repeat overlapping export files. Investigate incomplete or changing billing data before claiming a saving. GetCostAndUsage API

4. Application: right-size a fleet without hiding risk

Suppose a web fleet has low average CPU but unpredictable evening bursts. Gather per-instance and service-level evidence across representative peaks, deployments and failure tests. Check memory, disk latency, IOPS, throughput, network limits, connections and queueing. Standard EC2 metrics do not include guest memory utilization; install an appropriate agent or integration. A single saturated core can be hidden by a low multi-core average.

The following probe summarizes a 14-day CPU window. It reports an observed-sample mean, maximum reported CPU and the number of returned hourly periods. Missing periods are not silently treated as zero. CloudWatch does not return datapoints in chronological order. GetMetricStatistics

from datetime import timedelta, timezone

def cpu_evidence(cloudwatch, instance_id, end):
    if end.tzinfo is None or end.utcoffset() is None:
        raise ValueError("Use a timezone-aware end time")
    end = end.astimezone(timezone.utc).replace(minute=0, second=0, microsecond=0)
    response = cloudwatch.get_metric_statistics(
        Namespace="AWS/EC2", MetricName="CPUUtilization",
        Dimensions=[{"Name": "InstanceId", "Value": instance_id}],
        StartTime=end - timedelta(days=14), EndTime=end,
        Period=3600, Statistics=["Sum", "SampleCount", "Maximum"]
    )
    points = sorted(response["Datapoints"], key=lambda p: p["Timestamp"])
    count = sum(p["SampleCount"] for p in points)
    if not points or count <= 0:
        return {"status": "no usable data"}
    return {
        "status": "observed data",
        "observed_sample_mean_cpu": sum(p["Sum"] for p in points) / count,
        "max_reported_cpu": max(p["Maximum"] for p in points),
        "returned_hourly_periods": len(points),
        "requested_hourly_periods": 14 * 24,
        "latest_timestamp": points[-1]["Timestamp"].isoformat()
    }

Even 336 returned periods do not prove every expected sample was received. A recently launched or stopped instance has a different observation history. High-resolution bursts may disappear inside reported intervals. This function supplies evidence; it deliberately does not issue a downsize command at “CPU below 40%.” There is no universal ideal utilization range.

Test a smaller shape with representative traffic and a failure scenario. A CPU target of 50% can be a chosen target-tracking setpoint for one Auto Scaling group. Scaling attempts to approach it, subject to capacity, warmup, limits and metric behavior; it does not cap instantaneous CPU at 50%. Choose metrics that track demand per unit of capacity.

Now compare an explicitly defined 730-hour teaching month:

• Before: 50 instance-equivalents continuously at $0.10/hour = $3,650.
• After testing: a continuous baseline of 20 at an amortized $0.06/hour = $876.
• Additional variable usage: 10,000 instance-hours at a blended $0.04/hour = $400.
• New compute total: $1,276; compute saving $2,374, or 65.04% of the original compute subtotal. Twelve identical months save $28,488.

The extra 10,000 hours are above the baseline, so the terms do not overlap. They are an assumed measured usage total, not a consequence of setting an Auto Scaling minimum. Add EBS, networking, load balancing, observability and operational costs before calling this a total-workload saving.

For T-family instances, distinguish modes: Standard can fall back to baseline CPU when credits are exhausted; Unlimited can sustain usage through surplus credits and may incur extra charges. Inspect credit balance, surplus charges and workload duration before recommending Unlimited as a free fix. Unlimited mode

5. Application: make the database faster without changing its answer

A query optimization is valid only if the required result remains correct. Removing product joins, dropping returned columns or filtering out inactive customers changes an order report. Such a change needs a product requirement, not merely a faster benchmark.

Begin with database load, wait events, top SQL, query plans, lock contention and connection pressure. Current RDS guidance uses CloudWatch Database Insights; the Performance Insights API remains documented for metric access. Check engine support and Standard/Advanced features before choosing retention and monitoring options. Old console screenshots and old product names are not configuration instructions. RDS Database Insights

For a PostgreSQL customer-order view, consider this query and a candidate index:

SELECT id, customer_id, order_date, total
FROM orders
WHERE customer_id = 42
  AND order_date >= TIMESTAMP '2026-08-01 00:00:00'
ORDER BY order_date DESC, id DESC
LIMIT 100;

CREATE INDEX idx_orders_customer_date_id
ON orders (customer_id, order_date DESC, id DESC);

The equality key comes first, then the range/order keys. Keep the query unchanged while testing the index. Compare returned rows, order and boundary cases; id breaks equal-date ties. Review the execution plan with representative statistics. EXPLAIN (ANALYZE, BUFFERS) actually executes the statement, and index creation consumes resources and can affect concurrent work. More indexes also increase write work and storage. An index is a candidate, not a guaranteed speedup. PostgreSQL multicolumn indexes, Using EXPLAIN

Read replicas can offload appropriate reads, but the application or a suitable routing layer must use the replica endpoint. Asynchronous replication can produce stale reads; keep read-after-write operations on a path that meets their consistency requirements. A replica does not automatically accelerate a single bad query. RDS read replicas

Aurora Serverless v2 scales database compute within configured bounds; creating only the cluster does not create a usable writer instance. Provision the serverless DB instance, networking and credentials as well, then verify readiness. DynamoDB provisioned capacity with auto scaling and DynamoDB on-demand are different capacity modes. Neither removes hot-key constraints; DAX helps eligible cached reads, not every write or strongly consistent read. These choices follow the access pattern rather than a universal “serverless is cheaper” rule.

6. Application: price an archive over its actual lifetime

Choose storage using access latency, recovery requirements, object size, minimum-duration charges and resilience. Standard-IA supports immediate access but has retrieval charges. One Zone-IA stores in one AZ, so use it only when that loss model is acceptable. Glacier Instant Retrieval supports immediate GETs; Flexible Retrieval and Deep Archive require restoration. Ordinary Flexible Retrieval Standard restores commonly take hours; the often-quoted 1–5 minutes concerns eligible Expedited requests. Deep Archive Standard restoration is typically within 12 hours, with Bulk typically within 48 hours. These are not hard deadlines. Storage classes, Archive retrieval choices

A transition policy is not retention enforcement. For a versioned bucket, current-version expiration can create delete markers while older versions remain. If immutable retention is required, design Object Lock and version handling explicitly; 2555 days is not always seven calendar years. Do not derive a retention obligation from a cheap storage rate.

Example policy fragment for reviewed non-sensitive logs, with object sizes at least 128 KiB and no early deletion requirement: transition at object age 30 days to Standard-IA, at 90 days to Glacier Instant Retrieval, and at 365 days to Deep Archive. For an object following this rule from before its first transition, the scheduled thresholds are 60 days apart for IA and 275 for Instant Retrieval, exceeding their respective 30- and 90-day minimum billed durations. Deep Archive has a 180-day minimum billed duration. Days are measured from object creation, not policy installation: an existing 500-day-old Standard object is already eligible for Deep Archive. Review existing object ages, current classes and overlapping rules before applying the policy. Lifecycle for existing objects

{
  "Rules": [{
    "ID": "reviewed-log-transitions",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Transitions": [
      {"Days": 30, "StorageClass": "STANDARD_IA"},
      {"Days": 90, "StorageClass": "GLACIER_IR"},
      {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
    ]
  }]
}

This is the LifecycleConfiguration value for put_bucket_lifecycle_configuration, not the complete SDK request. That operation replaces the bucket's lifecycle configuration: merge and review existing rules first. Current lifecycle defaults skip transitions for objects smaller than 128 KiB unless configured otherwise; per-object requests and archive metadata can outweigh small-object savings. Intelligent-Tiering has its own monitoring and tier behavior, including optional asynchronous archive tiers. Lifecycle transition constraints

For arithmetic practice, ignore requests, retrievals, metadata, taxes and actual billing-day boundaries. Assume one fixed cohort of 1,000 billing GB stays for 84 equal teaching months. Storage rates are $0.023, $0.0125, $0.004 and $0.001 per GB-month. Allocate 1 month to Standard, 2 to IA, 9 to Instant Retrieval and 72 to Deep Archive:

from decimal import Decimal as D

def archive_teaching_costs():
    gb = D("1000")
    standard = gb * D("0.023") * 84
    tiered = gb * (
        D("0.023") * 1 + D("0.0125") * 2
        + D("0.004") * 9 + D("0.001") * 72
    )
    return standard, tiered, (standard - tiered) / standard * 100

Results: $1,932 continuously in Standard versus $156 tiered, a 91.93% storage-subtotal reduction under those assumptions. This month model illustrates accounting; it is not an exact simulation of the day-based policy above.

If you instead ingest another 1,000 GB at the start of every month, the first 84 months contain 1 + 2 + … + 84 = 3,570 cohort-months. Standard storage over that observation window is $82,110, not $1,932. The newest cohorts have not yet lived out their retention period. A growing archive needs a cohort model and an observation horizon.

For EBS, gp3 includes 3,000 IOPS and 125 MiB/s baseline throughput, with additional performance charged separately. A gp2-to-gp3 change that pins throughput to 125 MiB/s may reduce a busier volume's available throughput. Match workload requirements and compare the complete configuration. An unattached volume can be deliberately retained; confirm ownership, backups and recovery value before removal. EBS gp3 behavior

7. Application: compare a serverless API fairly

Assume 10 million requests, each using 1 GB of Lambda memory for 0.2 billable seconds. The compute quantity is 2 million GB-seconds. With a teaching rate of $0.0000166667/GB-second, duration costs $33.3334. At teaching request rates of $0.20/million for Lambda and $3.50/million for the chosen API Gateway API, those components cost $2 and $35. The subtotal is $70.3334, displayed as $70.33.

Keep the same database and data-access semantics in both alternatives. Do not compare EC2 plus RDS against Lambda plus DynamoDB without accounting for the database redesign and equivalent functionality. Add storage, request units, data transfer, NAT/endpoints, logs, authentication and any provisioned capacity. Free tiers, architecture, Region, API type and pricing tiers can change the result. Consult current prices for an actual estimate. Lambda pricing, API Gateway pricing

A $242 teaching subtotal for instances plus load balancing minus $70.3334 gives $171.6666, or 70.94%, for those selected components only. It does not establish a universal break-even request count. Doubling Lambda memory while halving billable duration leaves GB-seconds unchanged; faster completion can still improve user experience. Measure cold and warm invocations, downstream latency and p95/p99 tails.

Caching can reduce repeated work, but cache keys must include every relevant input and authorization boundary. CloudFront can serve dynamic HTTP content as well as static files; forwarding/cache policies and origin access need deliberate configuration. Global Accelerator supplies static anycast addresses and routes supported TCP/UDP traffic through AWS's network after entry, but does not cache content. Neither replaces application authentication or guarantees a latency figure. CloudFront cache behavior, Global Accelerator

Cross-Region replication is ongoing for new eligible writes, not a one-time payment that eliminates every transfer cost. Compare repeated remote reads with replication traffic, duplicate storage, requests and retrievals. Evaluate freshness and residency requirements as well as network charges.

8. Turn findings into an experiment

Write a short change record: hypothesis, evidence window, expected bill components, required service-level outcomes, rollout and rollback. Test one meaningful change at a time, then monitor long enough to include relevant demand cycles. Record actual useful output so a traffic decline cannot masquerade as optimization.

For a development environment approved to run 07:00–19:00 on weekdays, the intended schedule is 12 × 5 = 60 hours/week. Relative to 168 hours, that is 64.29% fewer running hours, before residual storage and other charges. A stop-only function never restarts the environment. Configure both start and stop schedules, an explicit timezone, holiday/override behavior and a check that resources reached the intended state. Commitments may remain payable while instances are stopped.

Try these decisions before looking at the answers:

1. CPU is low but p99 latency rises with disk queue depth. Investigate storage and query behavior before shrinking compute.
2. An RI recommendation promises savings for an instance scheduled for deletion. Remove the obsolete demand from the purchasing baseline first.
3. An archive needs unpredictable immediate access. Reject restore-dependent tiers unless the access requirement changes.
4. The bill falls 20% while successful requests fall 30%. Cost per success rose: 0.8 / 0.7 = 1.1429, about 14.29% worse.