Advanced Architecture Patterns

Advanced Architecture Patterns

An order is saved, but its notification is lost. A payment times out, yet the customer was charged. A migration rollback restores the old application while leaving the newest orders in another database. These failures happen between successful operations, where no single service owns the whole outcome.

This lesson builds four applications: a reliable event-backed order view, a compensating workflow, an incremental migration and dependency isolation. The aim is to select a pattern for a concrete consistency or recovery problem, then test the failure boundary it claims to handle. More services do not automatically create a more reliable system.

1. Separate the patterns before combining them

Event-driven architecture connects producers and consumers through events. An event records something that happened; a command requests work that may be rejected. OrderCreated is a fact, whereas CreateOrder is an instruction. Define an event ID, aggregate ID, per-aggregate version, schema version and payload contract. An event bus routes events; durable retention, replay and consumer recovery require explicit configuration and design.

CQRS, Command Query Responsibility Segregation, separates command and query models. It does not require two databases, event sourcing or asynchronous updates. A separate asynchronously updated projection is one useful implementation, with a deliberate consistency delay. Simple CRUD can remain simpler. CQRS

Event sourcing makes the retained sequence of domain events the source of truth from which state is rebuilt. It differs from keeping a current-state database plus an audit log. Snapshots accelerate reconstruction but do not replace the authoritative event history. Domain rules, schema evolution, retention and access to sensitive historical data remain responsibilities of the application. Event sourcing

DynamoDB Streams captures item changes for 24 hours; it is useful change data capture, not a permanent event store. Ordering is per item, not a global order of every event row sharing an aggregate partition key. Lambda consumers can process records more than once. Store authoritative events durably and use domain versions when order matters. DynamoDB Streams, Lambda stream processing

Command -> validate against authoritative aggregate
                         |
                         v
              one DynamoDB transaction
              /          |          \
         advance HEAD  append event  add outbox entry
                          |                |
                     retained log     retrying relay
                          |                |
                       replay         EventBridge -> consumer
                          |                              |
                          +---- rebuild projection <-----+
                                         |
                                    query response
                              includes projection version

A failed consumer can lag without blocking a producer immediately, but finite retention, backlog and shared quotas still connect their outcomes. Monitor outbox age, rejected events and projection lag. A green publisher metric does not prove that an email was sent or a read model caught up.

2. Application one: save an order event and its publication intent together

Writing business data and then calling EventBridge creates a dual-write gap: a crash between those operations leaves committed data without its event. A transactional outbox commits the business change and a pending publication record together. A relay publishes pending records and marks/removes them after acknowledged publication. A crash after publishing but before marking can duplicate delivery, so consumers still need idempotency. Transactional outbox

For this example, an existing DynamoDB table has string keys pk and sk. Each aggregate uses one partition-key value and these sort keys: HEAD, E# plus a 20-digit version, and O# plus the stable event ID. Authoritative writers use one Region and this protocol. The caller has loaded and validated the current aggregate, including the command's business transition. It supplies the expected version, a globally unique stable event ID, a stable request token and a small JSON-compatible payload. Every JSON integer, including nested payload values and the event version, must fit signed 64-bit range for EventBridge. Encode larger numeric identifiers as strings. All records must fit DynamoDB's item and transaction size limits.

This function builds the complete low-level SDK request, without sending it. Pass the returned object to dynamodb_client.transact_write_items. The event row and outbox row are immutable inserts; the head condition checks the current aggregate version, rather than examining only a newly addressed event row.

import json

def event_integer(token):
    value = int(token)
    if not -(2**63) <= value <= 2**63 - 1:
        raise ValueError("JSON integer outside EventBridge range")
    return value

def append_request(table, aggregate_id, expected_version,
                   event_id, event_type, data, request_token):
    if type(expected_version) is not int or not 0 <= expected_version < 2**63 - 1:
        raise ValueError("version outside key format")
    for value in (table, aggregate_id, event_id, event_type, request_token):
        if not isinstance(value, str) or not value:
            raise ValueError("nonempty identifiers required")
    if not isinstance(data, dict):
        raise ValueError("event data must be an object")
    version = expected_version + 1
    envelope = {"event_id": event_id, "aggregate_id": aggregate_id,
                "version": version, "schema_version": 1,
                "type": event_type, "data": data}
    body = json.dumps(envelope, sort_keys=True, separators=(",", ":"), allow_nan=False)
    json.loads(body, parse_int=event_integer)  # Check nested integers before commit.
    update = {"TableName": table,
              "Key": {"pk": {"S": aggregate_id}, "sk": {"S": "HEAD"}},
              "UpdateExpression": "SET #v = :next",
              "ExpressionAttributeNames": {"#v": "version"},
              "ExpressionAttributeValues": {":next": {"N": str(version)}}}
    if expected_version == 0:
        update["ConditionExpression"] = "attribute_not_exists(pk)"
    else:
        update["ConditionExpression"] = "#v = :expected"
        update["ExpressionAttributeValues"][":expected"] = {"N": str(expected_version)}
    def insert(sk):
        return {"Put": {"TableName": table,
                        "Item": {"pk": {"S": aggregate_id}, "sk": {"S": sk},
                                 "body": {"S": body}},
                        "ConditionExpression": "attribute_not_exists(pk)"}}
    return {"ClientRequestToken": request_token,
            "TransactItems": [{"Update": update},
                              insert(f"E#{version:020d}"), insert("O#" + event_id)]}

The three operations address different items in the same account and Region. A rejected condition cancels the transaction; it does not append a partial event. Two writers expecting version 7 cannot both advance the same head to 8 under this single-Region protocol. An expectation of version 9 cannot jump a head still at 7. This is stronger than a condition on a new event row saying “no item exists here.” Transactions

Use a valid transaction token of at most 36 characters and preserve the exact request on a transport retry. DynamoDB's request-token idempotency window is ten minutes; it is not a permanent business-command deduplication ledger. For later retries, resolve whether that command/event was committed before executing it again. Multi-Region eventual-consistency global-table writers need a different ownership/coordination design; this local transaction is not a global lock. Transaction considerations

A relay can find O# entries using a suitably designed index or dispatch mechanism. Treat Streams as a wake-up mechanism with a recovery scan/index path, so a relay outage longer than stream retention does not strand pending work. The following function publishes one stored envelope and inspects its individual outcome. Supply an existing event bus in the client's Region and appropriate permission; a nonexistent bus can cause events to be dropped without a failed-entry response, so validate the configured bus independently.

def publish_one(events, bus_name, envelope):
    response = events.put_events(Entries=[{
        "EventBusName": bus_name, "Source": "custom.orders",
        "DetailType": envelope["type"],
        "Detail": json.dumps(envelope, allow_nan=False)}])
    entries = response.get("Entries", [])
    if (response.get("FailedEntryCount") != 0 or len(entries) != 1
            or "ErrorCode" in entries[0] or not entries[0].get("EventId")):
        raise RuntimeError("event publication was not acknowledged")
    return entries[0]["EventId"]

The EventBridge-assigned ID is distinct from the stable domain event_id. Keep the domain ID on retransmission. An acknowledgment allows the relay to finish its outbox bookkeeping; it is not evidence that every target completed. Configure and monitor delivery retries/dead-letter handling, and test consumer behavior separately. PutEvents, Publishing considerations

Replay and projection correctness

Query the base event table with strong consistency, a known aggregate and an explicit version range; consume every result page. A multi-page Query is not an atomic snapshot of a changing collection. Capture a committed head version first and replay exactly through that boundary. For a snapshot at version N, begin at N+1. Do not replay the snapshot's last event twice. DynamoDB Query

This pure reducer uses the stored envelope directly, with no hidden nested eventData serialization. The surrounding application supplies all events for one aggregate in increasing version order. It recognizes a deliberately small state machine: created, paid, shipped. Unknown schema/type, missing versions and impossible transitions fail visibly. It performs no emails, charges or other external side effects during replay.

def replay_order(aggregate_id, events):
    state = {"version": 0, "status": "ABSENT", "total_cents": None}
    for event in events:
        if (event.get("aggregate_id") != aggregate_id
                or type(event.get("version")) is not int
                or event["version"] != state["version"] + 1
                or type(event.get("schema_version")) is not int
                or event["schema_version"] != 1):
            raise ValueError("wrong aggregate, schema or event sequence")
        kind, data = event.get("type"), event.get("data")
        if not isinstance(data, dict):
            raise ValueError("event data must be an object")
        if kind == "OrderCreated" and state["status"] == "ABSENT":
            amount = data.get("total_cents")
            if type(amount) is not int or not 0 <= amount <= 2**63 - 1:
                raise ValueError("invalid integer cents")
            state.update(status="CREATED", total_cents=amount)
        elif kind == "PaymentReceived" and state["status"] == "CREATED":
            state["status"] = "PAID"
        elif kind == "OrderShipped" and state["status"] == "PAID":
            state["status"] = "SHIPPED"
        else:
            raise ValueError("unknown event or invalid transition")
        state["version"] = event["version"]
    return state

For a live projection, atomically update projected state and its last-applied version. A duplicate already applied version must not repeat the effect; a gap must trigger ordered catch-up, not silently skip ahead. Concurrent consumers require a conditional write/transaction, not “check then write” in two unprotected calls. Rebuild into a new projection version, compare representative results and high-water marks, then switch readers. A replay must not charge the customer again.

Return the accepted version from command handling. If a client needs to see its own update, wait with a bound for the projection to reach that version, read an authoritative compatible view, or display “processing.” A strong read of a lagging projection only reads that projection's current contents; it cannot manufacture the missing update. A GSI supports eventual reads, even if its base table supports strong ones. Read consistency

3. Application two: compensate a workflow without pretending it is a rollback

A saga coordinates local transactions and business compensations. Orchestration puts the flow in a coordinator; choreography distributes reactions among participants. Neither provides database-style isolation across services. Compensation can take time, fail or require a different business action: canceling a shipment before dispatch differs from recovering a package already delivered. Saga orchestration

For an order, use this deliberate sequence: reserve inventory, charge payment, confirm the order. If payment is definitively declined without a charge, release the reservation. If confirmation fails after a known charge, refund and release. If a payment call times out, the outcome is unknown; query the payment provider using the stable operation ID before retrying or compensating. A timeout does not prove that no money moved.

A Step Functions Catch is an error-handling field, not a state type. Compensation work runs in Task states. Preserve successful task outputs with ResultPath, and ensure every later step receives the order, reservation and payment identifiers it needs. The default result replacement can otherwise discard the original input. States.ALL does not catch every possible failure, including States.DataLimitExceeded and States.Runtime. Error handling, ResultPath

A useful durable saga record contains order_id, operation IDs, known outcomes and a recovery status. Record transitions conditionally so two recovery workers cannot independently refund the same charge. The provider's own idempotency key and lookup capabilities must cover its external side effect; a local “processed” flag cannot atomically commit with an unrelated payment gateway.

This pure decision function demonstrates the recovery contract. Its inputs are statuses already established by reconciliation, not merely inferred from caught exceptions. It returns the next action for a durable orchestrator; it does not execute or persist it. Re-run it after the selected action's outcome is recorded.

def next_recovery(reservation, payment, confirmation):
    if reservation not in {"HELD", "RELEASED", "UNKNOWN"}:
        raise ValueError("invalid reservation status")
    if payment not in {"CHARGED", "NOT_CHARGED", "REFUNDED", "UNKNOWN"}:
        raise ValueError("invalid payment status")
    if confirmation not in {"FAILED", "CONFIRMED", "UNKNOWN"}:
        raise ValueError("invalid confirmation status")
    if "UNKNOWN" in (reservation, payment, confirmation):
        return "RECONCILE"
    if confirmation == "CONFIRMED":
        return "COMPLETE" if payment == "CHARGED" and reservation == "HELD" else "MANUAL_REVIEW"
    if payment == "CHARGED":
        return "REFUND"
    if reservation == "HELD":
        return "RELEASE"
    return "COMPENSATED"

Example: (HELD, CHARGED, FAILED) selects REFUND. After a confirmed refund it selects RELEASE; only (RELEASED, REFUNDED, FAILED) becomes COMPENSATED. A refund failure stays unresolved and is retried or escalated according to policy. Record deadlines, attempts and who owns manual recovery. The function's state names describe a reservation still held until a separate fulfillment handoff; consuming stock belongs to a later workflow.

4. Application three: migrate one capability while keeping a rollback path

The strangler fig pattern replaces legacy capabilities incrementally behind a routing boundary. Choose a slice with clear ownership, such as the customer profile read API. Compare behavior and data before expanding traffic. Feature flags and canary routing can help; they are techniques, not mandatory definitions of the pattern. Strangler fig

Routing must match a path boundary: /api/users-old is not a child of /api/users. This pure selector assumes the front door has already canonicalized and validated the path, and the tenant comes from authentication. It uses a deterministic rollout assignment. Increasing the percentage preserves the previously selected cohort, but requests do not become exactly that percentage in every small sample.

import hashlib

def select_backend(path, trusted_tenant, percent):
    if type(percent) is not int or not 0 <= percent <= 100:
        raise ValueError("rollout percent must be 0..100")
    if not isinstance(path, str) or not path.startswith("/"):
        raise ValueError("canonical absolute path required")
    if not isinstance(trusted_tenant, str) or not trusted_tenant:
        raise ValueError("authenticated tenant required")
    migrated = path == "/api/users" or path.startswith("/api/users/")
    bucket = int.from_bytes(hashlib.sha256(trusted_tenant.encode()).digest()[:8], "big") % 100
    return "new" if migrated and bucket < percent else "legacy"

Use this decision in a correctly configured routing layer; it is not a complete Lambda@Edge handler. With Lambda@Edge, CloudFront origin changes belong to an origin-request flow with appropriate origin/Host/TLS settings and cache behavior. API Gateway integrations are a different interface. Test cache hits as well as misses so a response from one route or tenant cannot leak into another. Lambda@Edge request examples

Pick one authoritative writer during data migration. Backfill to a known boundary, apply ongoing changes with version/deduplication rules, reconcile records and switch ownership deliberately. Best-effort dual writes that merely log a failure create divergence. Falling back to legacy data after every new-system error can return deleted or stale records and disguise an outage as “not found.”

A route toggle is a complete rollback only while the old system can correctly serve all accepted changes. Before moving writes, decide how reverse synchronization, write fencing or forward recovery will work. Shadow traffic should avoid duplicating side effects and should respect data-access rules. Canary success includes result equivalence and data reconciliation, not just a lower error count.

5. Application four: keep a slow dependency from consuming the service

A timeout bounds waiting under a defined client contract. A retry attempts recoverable work again, with bounded attempts, backoff, jitter and idempotency. A circuit breaker temporarily stops attempts to a failing dependency. A bulkhead limits the resources one dependency or workload can consume. These mechanisms complement each other. SDK retry configuration, Circuit breaker

A breaker normally moves CLOSED → OPEN after its failure threshold, then HALF_OPEN for a limited recovery probe. A failed probe reopens it immediately; successful recovery closes it. Use monotonic elapsed time and cap concurrent probes. Late results from older calls must not accidentally reset the current generation of the breaker. A process-local breaker is local to that process; serverless instances do not share its state automatically.

Suppose three nested layers each make up to three attempts, including the original. One user operation can create 3×3×3=27 downstream attempts. Decide which layer owns retries and fit its attempts into the end-to-end deadline. A payment queued for later processing must return a pending outcome, not “paid.” Reject invalid input without counting every business validation failure as dependency unavailability.

For bulkheads, separate bounded capacity for critical checkout and background reporting. A ThreadPoolExecutor(max_workers=10) caps active workers but does not by itself establish a bounded submission queue. A future timing out does not stop an already-running function. Bound admission, queue length and dependency connections; release the capacity permit when the work actually exits, not when the caller stops waiting. Python executor behavior

As a planning approximation, if a dependency occupies a worker for 0.2 seconds and ten workers are continuously available, the ideal service capacity is 10/0.2=50 operations/second. Queueing, variable latency, retries and other constraints reduce usable capacity. At a sustained 80 operations/second with only 50 completions/second, an unconstrained queue grows by about 30 operations each second. After a minute that is 1,800 waiting operations, before accounting for abandonment or rejected work. Admission limits make overload explicit.

Service meshes and traffic layers can provide service discovery, telemetry, routing and selected transport policies. They cannot decide whether a timed-out payment was charged or repair application idempotency. AWS has announced App Mesh support ends September 30, 2026; evaluate current supported options such as ECS Service Connect for the relevant environment rather than adopting a retiring control plane. App Mesh notice, ECS Service Connect

6. Test the boundary, not just the happy path

For the event path, inject failure before and after transaction commit, after publication and before acknowledgment bookkeeping, and during projection update. Demonstrate one accepted aggregate version, durable pending work and replay-safe effects. For the saga, test unknown payment outcome, duplicate recovery, failed compensation and operator handoff.

For migration, compare path boundaries, stable cohorts, stale caches and rollback after a new write. For resilience, test overload with slow rather than instantly failing dependencies, bounded queues, probe concurrency and abandoned callers. Keep evidence of the useful business result and recovery time. A pattern earns its complexity when these tests demonstrate the specific guarantee the application needs.