Advanced Architecture & AI

Advanced Architecture & AI

An image classifier works in a demo: upload a file, call a model, save the label. Production adds harder questions. What if the upload notification arrives twice? What if the object is overwritten before processing? What if a model changes while predictions remain cached? What if two Regions both accept the last available item in stock?

Advanced architecture is the ability to answer those questions with explicit contracts. This lesson develops four applications: an event-driven inference pipeline, regional data ownership, interruptible training, and an asynchronous media workflow. Service behavior was checked in September 2026; benchmark numbers below are teaching assumptions, not measured AWS guarantees.

1. Specify the result, deadline and failure policy

For each stage, define its input identity, output schema, completion condition, retry policy and authorized caller. An accepted request is not necessarily completed work. A job ID proves submission; it does not prove the model trained or the video finished processing.

Choose a serving mode from requirements:

• Interactive inference: a caller waits for a response. Measure the entire path: authorization, feature lookup, queueing, model loading, inference and serialization. A host type alone cannot promise a 20 ms response.
• Asynchronous inference: callers can receive a job identifier and retrieve results later. Plan storage, status, deadlines and failure notification. SageMaker AI asynchronous inference can queue eligible requests and return results through S3.
• Batch inference: a finite input dataset can be processed as a job. Compare job startup, data partitioning and actual runtime with the existing online capacity. Batch is not automatically cheaper at every volume.

Lambda can orchestrate short tasks; a SageMaker AI endpoint, job or another serving platform can run the model. ECS Service Auto Scaling and Kubernetes scaling are available; “Fargate is manual only” is false. Model memory, accelerators, container support, utilization and operational ownership affect the choice. Layers package Lambda dependencies but do not guarantee shorter cold starts. Module-level initialization can be reused within a warm execution environment, not across every instance. SageMaker AI inference options, Lambda execution environment

For generative applications, add model/version, prompt template, retrieval sources, evaluation criteria and permission boundaries. Treat retrieved text and model output as untrusted data. The application authorizes tool actions and validates their arguments; a model's confident answer is not evidence of correctness. Bedrock provides access to supported foundation models; SageMaker AI supplies tools for custom model development and hosting. Evaluate your task and data rather than assuming one service guarantees factual answers. Amazon Bedrock

2. Application: process uploaded images despite retries

Use a versioned S3 bucket with a dedicated input prefix. Send direct S3 object-created notifications to a standard SQS queue, then use a Lambda event source mapping. The queue needs permission for the specific S3 source; the consumer needs S3 version-read, model-invoke and result-write access, plus any required KMS permissions. Keep result files outside the triggering prefix to prevent a loop.

Versioned S3 input
        |
        v
SQS standard queue ---> exhausted retries ---> dead-letter queue
        |
        v
Lambda: validate event, resolve tenant, identify object version
        |
        v
Read exact version -> preprocess -> invoke pinned model
        |
        v
Idempotent result record -> separate notification/outbox worker

Each arrow is configured delivery or application work. S3 does not magically wait for inference to finish. Direct S3 notification filtering for .jpg or .png needs two non-overlapping configurations, each with one suffix. File extensions are a routing convenience; validate the actual content before decoding it. S3 notification filters

This is a notification configuration value for a bucket whose approved configuration you manage. Merge existing entries before saving because PutBucketNotificationConfiguration replaces the configuration. Substitute the reviewed queue ARN; the queue and bucket must satisfy S3's destination requirements.

def image_notifications(queue_arn):
    return {"QueueConfigurations": [
        {
            "Id": "input-" + suffix[1:],
            "QueueArn": queue_arn,
            "Events": ["s3:ObjectCreated:*"],
            "Filter": {"Key": {"FilterRules": [
                {"Name": "prefix", "Value": "input/"},
                {"Name": "suffix", "Value": suffix}
            ]}}
        }
        for suffix in (".jpg", ".png")
    ]}

S3 event keys are URL encoded: a+b.jpg in the event represents a b.jpg; a literal plus is encoded as %2B. Process every nested S3 record, not just Records[0]. Require the non-null version ID for this design, and fetch that version so later overwrites cannot change the input. Keep bucket, key and version together; key alone is not globally unique. S3 event structure

The following adapter assumes messages arrive through the trusted queue and process_one is the application's idempotent processing function. It handles a standard queue; it is not the ordered-processing algorithm for a FIFO queue. Emit operational failure metrics in a surrounding wrapper without logging sensitive payloads.

import hashlib
import json
from urllib.parse import unquote_plus

def object_identity(record, model_version):
    if record.get("eventSource") != "aws:s3":
        raise ValueError("Unexpected event source")
    if not record.get("eventName", "").startswith("ObjectCreated:"):
        raise ValueError("Unexpected event type")
    obj = record["s3"]["object"]
    version = obj.get("versionId")
    if not version or version == "null":
        raise ValueError("A versioned object is required")
    bucket = record["s3"]["bucket"]["name"]
    key = unquote_plus(obj["key"])
    parts = [bucket, key, version, model_version]
    identity = hashlib.sha256(
        json.dumps(parts, ensure_ascii=False, separators=(",", ":")).encode()
    ).hexdigest()
    return {"bucket": bucket, "key": key, "version": version,
            "model_version": model_version, "identity": identity}

def process_sqs_batch(event, process_one, model_version):
    failed = []
    for message in event["Records"]:
        try:
            payload = json.loads(message["body"])
            if payload.get("Event") == "s3:TestEvent":
                continue  # Expected destination-validation message.
            records = payload["Records"]
            if not records:
                raise ValueError("Empty object event")
            for record in records:
                process_one(object_identity(record, model_version))
        except Exception:
            failed.append({"itemIdentifier": message["messageId"]})
    return {"batchItemFailures": failed}

The real handler returns that object and enables ReportBatchItemFailures on its mapping. If one SQS message contains several S3 records and a later one fails, retrying that message repeats earlier work. Idempotency is therefore still required. If the handler itself crashes or times out, partial-response handling cannot rescue an unreturned response. SQS partial batch responses

A robust process_one resolves authorization from a controlled bucket/prefix ownership mapping, reads the version with get_object(VersionId=...), enforces size/pixel limits, and applies the model's actual preprocessing contract. The endpoint's response schema is model-specific. An image model does not universally return predictions[0].class. Store an actual timestamp as well as a request/correlation ID; a UUID is not a time.

Use a durable idempotency record with conditional state transitions and a recoverable processing lease. A crash after inference but before saving can repeat inference; a crash after saving but before sending SNS can lose or duplicate a notification in a naive implementation. Write an outbox record transactionally with the result where supported, then deliver it with retry and a consumer deduplication key. This improves recoverability; it does not promise exactly-once external effects.

Control load at the consumer and the model

In standard polling mode the SQS event source mapping scales consumption. Set its MaximumConcurrency to a tested limit and provision sufficient function/account concurrency. For example, an authorized configuration fragment for an existing standard-mode mapping is:

def configure_consumer(lambda_client, mapping_uuid):
    return lambda_client.update_event_source_mapping(
        UUID=mapping_uuid,
        FunctionResponseTypes=["ReportBatchItemFailures"],
        ScalingConfig={"MaximumConcurrency": 10}
    )

This caps concurrent invocations from that mapping, not requests per second to the endpoint. Ten workers each issuing a request every 0.2 seconds could offer about 50 requests/second in a steady idealized case; concurrency, processing time, retries and batching change the rate. Keep endpoint capacity, queue age and throttling in the feedback loop.

Lambda function provisioned concurrency keeps execution environments initialized; it is not a queue-depth rate limiter. SQS event poller provisioned mode is a separate feature and cannot be combined with the mapping's MaximumConcurrency setting. Configure queue visibility, retry and dead-letter policies against actual processing time; observe backlog age as well as count. SQS scaling controls

3. Make prediction caching version-aware

Cache-aside means the application checks the cache, invokes the model on a miss, then stores the result. A prediction cache is correct only if equal keys imply interchangeable authorized results. Include the tenant, model artifact revision, preprocessing version, relevant features and inference parameters. If features are mutable, include their version or invalidate on change. Disable caching when stochastic output or freshness requirements make reuse inappropriate.

Python's built-in hash() is not a persistent, collision-resistant identifier across processes. This key function uses a stable serialized envelope and SHA-256. The trusted caller supplies a complete JSON-compatible feature object; nonfinite numbers are rejected. Extra distinctions such as 1 versus 1.0 can cause misses, but should not cause an incorrect hit.

def prediction_cache_key(tenant, model_version, preprocessing_version,
                         features, inference_parameters):
    envelope = {
        "tenant": tenant, "model": model_version,
        "preprocessing": preprocessing_version,
        "features": features, "parameters": inference_parameters
    }
    encoded = json.dumps(envelope, sort_keys=True, ensure_ascii=False,
                         separators=(",", ":"), allow_nan=False).encode()
    return "prediction:" + hashlib.sha256(encoded).hexdigest()

An opaque hash does not authorize access or anonymize guessable input. Authorize before cache lookup, protect cache transport and access, avoid exposing keys, and use a deliberate TTL. Handle cache failure without overwhelming inference through a miss storm. Multi-model endpoints can reduce idle capacity for supported workloads, but loading and eviction affect latency; there is no fixed 50–70% saving.

4. Application: regional availability with explicit write ownership

Define RTO, the targeted restoration time, and RPO, the targeted acceptable loss window. An active-passive topology might keep a warm secondary ready for promotion; an active-active application serves requests in several Regions. Neither label sets a universal RTO, RPO or availability percentage.

Separate routing from data correctness. Route 53 latency routing uses its latency measurements; geolocation routes by user location policy and is not a synonym for lowest latency. DNS caches, health detection, connection reuse and secondary readiness affect failover. A global routing layer does not make a database writable or replicate secrets and artifacts. Test a complete business transaction during a regional drill. Route 53 routing policies

Aurora Global Database has a primary write Region with secondary Regions replicating from it. Where supported, write forwarding sends secondary-originated writes to the primary; it does not create independent writers in every Region. An application can be active in multiple Regions while its relational writes still have one authority. Aurora Global Database

App in Region A -- local write ------> Aurora primary writer
                                             |
App in Region B -- forwarded write ---------->|
                                             |
                                             v
                                     Aurora secondary read copy

For current DynamoDB global tables, choose a consistency mode deliberately. MREC replicates asynchronously and resolves concurrent same-item writes using last-writer-wins; conditions check the local replica. MRSC supports cross-Region strong reads and conditions against the latest item, with different supported Region/topology and feature constraints. MRSC uses three Regions with either three replicas or two replicas plus a witness. A witness is not an application read/write endpoint. Global table consistency modes

Do not confuse the legacy create_global_table API with creating a current-version table. For an existing eligible table, current replica management uses update_table with ReplicaUpdates; MRSC creation adds further requirements. Inspect the table's version, mode and readiness before proceeding. UpdateTable, Global table versions

Conflict exercise: two MREC Regions both see inventory version 7 and stock 1. Each locally accepts “if version=7, set stock=0 and version=8.” Both local conditions can succeed. Last-writer-wins may converge to stock 0 while two orders have been accepted. Adding an application version attribute does not establish a cross-Region lock.

Choose a business solution: assign a fenced write owner for each inventory item; partition ownership so different Regions control different units; use a reservation protocol; or evaluate a suitable strongly consistent design. An owner failover needs fencing so the old owner cannot continue accepting writes. A lease stored independently in each MREC replica cannot itself prove global exclusivity. Global table design guidance

Shard a hot entity using a varying input

Hashing only user_id yields the same suffix for every event from that user. It changes the name of the hot key without spreading that user's writes. Instead, use a stable event identifier for the shard calculation. The following deterministic pattern supports retrying the same event into the same logical shard:

def event_keys(user_id, event_id, shards=10):
    if isinstance(shards, bool) or not isinstance(shards, int) or shards < 1:
        raise ValueError("shards must be a positive integer")
    value = int.from_bytes(hashlib.sha256(event_id.encode()).digest(), "big")
    suffix = value % shards
    return {"pk": f"USER#{user_id}#SHARD#{suffix}", "sk": f"EVENT#{event_id}"}

Require stable, unique event IDs and a fixed shard scheme for the data's lifetime, or version the scheme during migration. Different event IDs can distribute one user's writes across logical keys; this does not guarantee one physical partition per suffix. Reading all events requires querying all logical shards and merging results. A GSI with only the same hot user ID can recreate the bottleneck. Introduce sharding only when measured skew justifies the added read complexity. DynamoDB write sharding

TTL is a cleanup mechanism, not a precise expiry gate. Store Unix epoch seconds in the configured numeric attribute, and enforce expiration during reads/authorization because deletion is asynchronous. MREC can replicate TTL deletions with replica-side charges; MRSC does not support TTL. Do not base a session's validity on whether its item has physically disappeared. DynamoDB TTL

5. Application: train through a Spot interruption

Managed Spot Training can lower eligible training compute charges, but interruption can delay completion. Define the acceptable finish deadline and whether training is restartable. The training container must actually save and restore checkpoints; merely declaring an S3 checkpoint path does not implement recovery.

Use a supported, reviewed training image and pin dependencies. A checkpoint may need model weights, optimizer state, training progress, data position and random-number state. For distributed training, coordinate workers and choose the framework's correct writer/restore strategy. Resuming at the wrong epoch can silently skip or repeat data even if a file loads successfully.

These are selected fields to merge into a complete CreateTrainingJob request, which also needs the job name, role, image, input/output and resource configuration. Use a distinct checkpoint prefix per compatible training run. The training program writes to the configured local path and restores the same format at startup.

def managed_spot_fields(checkpoint_uri):
    return {
        "EnableManagedSpotTraining": True,
        "CheckpointConfig": {
            "S3Uri": checkpoint_uri, "LocalPath": "/opt/ml/checkpoints"
        },
        "StoppingCondition": {
            "MaxRuntimeInSeconds": 86400,
            "MaxWaitTimeInSeconds": 90000
        }
    }

Here 86,400 seconds is 24 hours and 90,000 is 25 hours. The managed Spot wait limit covers the overall wait/run allowance described by SageMaker; it is not 25 extra hours added to the runtime. It must be at least the runtime limit, and extra room accommodates waiting/interruption. These limits do not guarantee capacity or successful completion. Managed Spot Training, StoppingCondition

For a checkpoint drill, train to a known intermediate step, terminate the training process in a disposable experiment, then resume from the synchronized checkpoint. Verify restored progress and the expected learning behavior, not merely the existence of a checkpoint directory. Avoid mixing a whole-model save API with a checkpoint-discovery API that expects another file format. Atomic local file replacement can avoid a partially written local manifest; it alone does not prove all referenced files reached S3 consistently.

Calculate observed savings from job metadata rather than assuming a 70% discount and applying it twice:

from decimal import Decimal

def training_compute_summary(job, hourly_rate):
    if job["TrainingJobStatus"] != "Completed":
        raise ValueError("Use a completed comparable training job")
    training = Decimal(str(job["TrainingTimeInSeconds"]))
    billable = Decimal(str(job["BillableTimeInSeconds"]))
    count = job["ResourceConfig"]["InstanceCount"]
    rate = Decimal(str(hourly_rate))
    if training <= 0 or billable < 0 or count < 1 or rate < 0:
        raise ValueError("Invalid training cost inputs")
    return {
        "reference_compute": training * count * rate / 3600,
        "billable_compute": billable * count * rate / 3600,
        "reported_saving_percent": (1 - billable / training) * 100
    }

For a homogeneous four-instance job, teaching metadata of 3,600 training seconds and 1,080 billable seconds with an invented $3/instance-hour rate gives $12 reference compute, $3.60 billable compute, and 70% for this comparison. The count matters for dollars. Do not multiply $3.60 by another 0.3 discount. This is not a complete bill: include storage, requests, data movement and the value of waiting. A genuinely equivalent On-Demand run can have a different runtime. DescribeTrainingJob billing fields

Similarly, a hypothetical one-instance endpoint at $1/hour for a 720-hour month costs $720. Batch processing at the same rate for four hours on each of 30 days costs $120: 83.33% lower for that compute subtotal. Five batch instances for the same four hours/day would cost $600, or 16.67% lower. More instances help only if work can be partitioned effectively; there is no guaranteed linear speedup.

6. Application: wait for an asynchronous media job correctly

Amazon Elastic Transcoder's support ended in November 2025. Use a current service such as AWS Elemental MediaConvert for supported transcoding workloads. Step Functions has an optimized mediaconvert:createJob.sync integration for Standard workflows; an arbitrary API name does not acquire .sync support merely by appending the suffix. AWS service shutdown reference, MediaConvert integration

For Rekognition stored-video moderation, submit a job, retain its ID, and use a documented completion notification or poll GetContentModeration. An AWS SDK integration returns the API response; startContentModeration returns before analysis finishes. Fetching a single result page is also not the same as retrieving the entire analysis. Rekognition stored-video flow

The following complete small JSONPath state-machine definition demonstrates polling for completion, not the whole media platform. It runs as a Standard workflow. Input contains bucket, key, version and a caller-generated token stable for this job submission; the selected versioned video and service permissions are prerequisites. The token must meet Rekognition's constraints and correspond to identical request parameters when reused. The execution role needs the supported Rekognition calls; the service needs access to the video and any KMS key.

{
  "Comment": "Wait for a versioned video moderation job; return its job ID",
  "TimeoutSeconds": 3600,
  "StartAt": "Submit",
  "States": {
    "Submit": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rekognition:startContentModeration",
      "Parameters": {
        "Video": {"S3Object": {
          "Bucket.$": "$.bucket", "Name.$": "$.key", "Version.$": "$.version"
        }},
        "ClientRequestToken.$": "$.token"
      },
      "ResultPath": "$.submission",
      "Next": "Pause",
      "Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.failure", "Next": "Failed"}]
    },
    "Pause": {"Type": "Wait", "Seconds": 10, "Next": "Poll"},
    "Poll": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rekognition:getContentModeration",
      "Parameters": {"JobId.$": "$.submission.JobId", "MaxResults": 1},
      "ResultPath": "$.poll",
      "Next": "Status",
      "Catch": [{"ErrorEquals": ["States.ALL"], "ResultPath": "$.failure", "Next": "Failed"}]
    },
    "Status": {
      "Type": "Choice",
      "Choices": [
        {"Variable": "$.poll.JobStatus", "StringEquals": "IN_PROGRESS", "Next": "Pause"},
        {"Variable": "$.poll.JobStatus", "StringEquals": "SUCCEEDED", "Next": "Ready"}
      ],
      "Default": "Failed"
    },
    "Ready": {
      "Type": "Pass",
      "Parameters": {"jobId.$": "$.submission.JobId", "status": "READY_FOR_RESULT_RETRIEVAL"},
      "End": true
    },
    "Failed": {"Type": "Fail", "Error": "ModerationIncomplete", "Cause": "Inspect execution history"}
  }
}

ResultPath preserves the original input while storing responses. The success output is intentionally a job ID, not an incomplete classification result. A next-stage reader must page through GetContentModeration with NextToken and persist the results. Keep large outputs in S3 and pass references through the workflow. StartContentModeration, GetContentModeration

This teaching machine fails on API errors rather than retrying them implicitly; production can add explicit bounded retries for selected transient errors. An execution timeout requires external monitoring and reconciliation, and stopping the workflow does not by itself cancel the underlying Rekognition job. States.ALL does not catch every possible Step Functions failure. Record submission identity so an operator can reconcile a lost response. Step Functions error handling

Transcoding, thumbnailing and moderation can run as parallel branches when independent. A Parallel state waits for successful branches and normally produces an array of branch results. Map that array deliberately before a downstream task expects named fields. Catch failures where recovery makes sense, configure retries explicitly, and preserve the original video identity. A failed branch does not undo an already-created file or guarantee cancellation of every external job.

7. Protect the boundaries and test the claims

API keys identify clients for API Gateway usage plans; they are not a replacement for authentication or authorization. Validate the caller and tenant before reading inputs, results or cache entries. Use separate execution roles with the actions and resource scope each stage needs. A private network path does not authorize a request by itself.

Avoid a combined S3 GetObject/PutObject allow conditioned on an upload encryption header: ordinary GETs do not supply that upload header. Separate read and write statements and configure the intended bucket encryption and KMS grants. For a specific VPC endpoint boundary, use the applicable aws:SourceVpce condition. If you also filter private addresses with aws:VpcSourceIp, combine it with an endpoint or VPC identity condition such as aws:SourceVpce or aws:SourceVpc: unrelated VPCs can have overlapping CIDRs. A private CIDR in aws:SourceIp does not implement that boundary. Review service-mediated access paths before adding network denies. Customer-managed KMS keys are appropriate when their control model is needed; using an AWS-managed key is not inherently a mistake. S3 policy condition examples, IAM network context keys

A service catalog also has an availability date. For example, Amazon Forecast is unavailable to new customers, although eligible existing customers can continue using it. Do not present it as an unconditional starting choice for a new project. Forecast availability notice

Finish with a failure drill for each contract:

1. Deliver a duplicate upload event and then overwrite the same key. The same event should use the original version and a stable result identity.
2. Change the model revision without changing the input. The prediction cache key must change.
3. Partition two MREC Regions during an inventory sale. Verify the business ownership protocol, not just eventual item convergence.
4. Restart training mid-run. Verify the actual checkpoint contents and resume position.
5. Let moderation run beyond the polling deadline. Detect the failed execution and reconcile the still-existing job.