Caching & Performance

Caching & Performance

A fast answer is useful only if it belongs to the right user and is fresh enough for the task. A catalog description might tolerate a few minutes of delay. An authorization decision, account balance or final checkout price often cannot. Start a caching design by stating which data may be reused, for whom, and for how long.

This lesson develops four applications: a public catalog API, expiring sessions, a leaderboard and DAX-backed article reads. It uses current boto3 and redis-py with Valkey/Redis OSS-compatible commands. Examples are bounded teaching components; prepare network access, authentication, TLS and deployment settings before connecting them to AWS.

1. Put each cache on the request path that uses it

CloudFront caches HTTP representations near viewers. ElastiCache is accessed by application code. DAX is a DynamoDB-specific cache accessed through a DAX client. They are alternatives or complements for particular paths, not mandatory stages in one chain:

viewer -> CloudFront --cache miss--> ALB/API origin -> application
                                                       |-- cache lookup -> ElastiCache
                                                       |-- SQL query ----> RDS/Aurora
                                                       +-- DAX request --> DAX -> DynamoDB
static asset miss: CloudFront -> S3 origin

An ALB does not act as a general-purpose response cache. An application need not call ElastiCache before DAX, and neither cache transparently intercepts arbitrary database traffic. A closer location can reduce travel time, but cache misses, network conditions, authorization and origin work determine the observed latency. Avoid fixed latency or speedup promises. CloudFront, DAX concepts.

API Gateway's managed stage cache is a REST API feature; do not generalize it to every API Gateway API type. REST API caching.

2. CloudFront: the cache key comes before the TTL

If /products?category=books&page=2 differs from page 1, the cache key must distinguish those values. Forwarding a query string to the origin without including it in the cache key can still serve the wrong cached response. Likewise, an origin response varying by tenant, language, cookie or header needs an appropriate key or a behavior that disables caching. Cache policy controls key membership and TTL; origin request policy controls additional forwarding. Cache keys, policy distinction.

Cache-Control: public, max-age=0, s-maxage=60 permits shared-cache reuse for sixty seconds while making browser reuse immediately stale. It does not authenticate users. CloudFront's minimum/default/maximum TTL settings interact with origin headers. A positive minimum TTL can override no-cache, no-store or private. For private personalized routes, use a caching-disabled behavior and appropriate forwarding; do not rely solely on adding a response header. TTL behavior, cache policy settings.

Choose long TTLs for content-addressed assets whose URLs change when bytes change. Use shorter or revalidated policies for entry pages and changing public data. Invalidation requests are asynchronous and do not purge browser caches or the application's Redis keys. Reuse one CallerReference for retries of the same logical invalidation and identical path list; a different batch must use a different reference. Invalidation, CreateInvalidation.

import boto3

def invalidate_images(distribution_id, operation_id):
    # Persist operation_id for retries of this exact batch.
    return boto3.client("cloudfront").create_invalidation(
        DistributionId=distribution_id,
        InvalidationBatch={
            "Paths": {"Quantity": 1, "Items": ["/images/*"]},
            "CallerReference": operation_id})

CloudFront does not cache objects of arbitrary size in a single response. Its documented maximum cacheable response body is 50 GB; supported range requests can cache larger objects as bounded parts. Cacheable size.

3. ElastiCache: engine behavior is not a durability promise

ElastiCache supports Valkey, Redis OSS and Memcached. Valkey/Redis OSS provide strings, hashes, lists, sets, sorted sets and other version-dependent features. Memcached stores opaque values by key, so “strings only” is a misleading application-level restriction: serialized objects and bytes are normal. Compare the engine and deployment together; node-based and serverless options do not have identical replication, scaling or recovery behavior. Engine comparison, deployment options.

Do not copy an upstream Redis “RDB/AOF persistence” checklist into ElastiCache. Managed snapshots, replication and optional node-based Valkey durability are distinct. The durability feature uses a Multi-AZ transactional log, has engine/node/cluster-mode constraints, and is not available in ElastiCache Serverless. Its synchronous and asynchronous write modes make different loss/latency tradeoffs. A plain cache should be treated as reconstructible unless the chosen configuration and application contract explicitly provide more. Durability, limitations.

“Redis is single-threaded, Memcached is multithreaded” is also too coarse for modern service choices. Command execution, networking/I/O and background work have different threading behavior; supported Valkey/Redis OSS configurations have I/O enhancements. Measure engine CPU, connection and memory limits rather than predicting throughput from a binary label. Enhanced I/O.

Use the endpoint's actual port and TLS/authentication configuration. This helper targets a TLS-enabled cluster-mode-disabled primary endpoint with a prepared ACL user. Cluster-mode-enabled deployments need a cluster-aware client and compatible key placement:

import redis

def cache_connection(host, username, password, port=6379):
    return redis.Redis(
        host=host, port=port, username=username, password=password,
        ssl=True, ssl_cert_reqs="required", decode_responses=True,
        socket_connect_timeout=0.2, socket_timeout=0.2)

Credentials come from a secure application mechanism, not source code. decode_responses=True decodes text responses; it does not parse JSON. An absent key returns None, which must be distinguished from a valid empty string or a cached JSON null. Encryption in transit, redis-py connections.

4. Understand misses, writes and local caches

Cache-aside: read the cache; on a miss, read the source and populate the cache. Write-through: the application or cache integration synchronously coordinates the source write and cache update. Write-behind: the source write occurs later and therefore requires durable delivery, retries and a recovery policy. A database update followed by SET is not an atomic transaction across both services. Caching strategies.

Even “commit, then delete the cache key” has a race: reader A fetches old database data; writer B commits and invalidates; A then caches its old result. Solutions depend on the freshness contract: versioned values/keys, coordinated invalidation, fresh source reads for sensitive operations, or accepting a measured stale interval. Two independent writes cannot establish “always current.” A TTL bounds cache residence, not time since the underlying business event.

Stacked TTLs can compound staleness. Under a simplified no-write/no-refresh-race model, Redis retains an old source result for 300 seconds and CloudFront fetches it just before expiry, then retains it another sixty seconds. The old representation may remain available for nearly 360 seconds, not just sixty. Add replication lag, read races and stale-serving policies to the actual budget.

A local dictionary without expiry or a size limit can grow and retain values indefinitely. If a local layer copies a Redis value, give it the upstream value's remaining freshness deadline; resetting a new full TTL can extend stale data. Account for every worker's independent cache and distinguish monotonic duration clocks from transferable wall-clock timestamps. Prefer omitting the local layer until measurements justify that complexity.

Warming also needs a budget. Load a bounded list of known popular keys or an appropriate index, handle pagination and partial results, and rate-limit the work. A filtered DynamoDB Scan still reads the examined data, and one request need not reach the table's end. Boto3 Decimal values require an explicit serialization policy rather than blindly calling json.dumps.

5. Application: a public catalog API with coordinated refill

Assume the representation is public, contains no personalized prices or secrets, and depends only on normalized category and page. The CloudFront cache policy must include these query strings, use minimum TTL zero and allow the shared sixty-second TTL. The origin returns application/json. Configure private routes separately. Product availability and final prices are revalidated at checkout.

Build an application key from the complete normalized query. A digest avoids delimiter collisions; a schema-version prefix invalidates incompatible formats. These paired keys share a hash tag so the Lua scripts can also operate within one cluster slot:

import hashlib
import json

def catalog_keys(category, page):
    if category not in {"all", "books", "games"}:
        raise ValueError("Unknown category")
    if type(page) is not int or not 1 <= page <= 100:
        raise ValueError("Page must be 1–100")
    identity = json.dumps([category, page], separators=(",", ":"))
    digest = hashlib.sha256(identity.encode()).hexdigest()
    base = "catalog-v1:{" + digest + "}"
    return base + ":data", base + ":lock"

A cache stampede occurs when many callers refill the same expired key at once. The following lease reduces duplicate work on one cache primary. Each owner has a random token. Release and publication check that token atomically, so an expired worker cannot delete a replacement lease or overwrite its result:

import secrets
import time

RELEASE = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0
"""
PUBLISH = """
if redis.call('GET', KEYS[2]) ~= ARGV[1] then return 0 end
redis.call('SET', KEYS[1], ARGV[2], 'EX', ARGV[3])
redis.call('DEL', KEYS[2])
return 1
"""

class CacheBusy(Exception):
    pass

def cached_catalog(cache, category, page, fetch):
    data_key, lock_key = catalog_keys(category, page)
    deadline = time.monotonic() + 0.2
    while True:
        cached = cache.get(data_key)
        if cached is not None:
            return json.loads(cached)
        token = secrets.token_hex(20)
        if cache.set(lock_key, token, nx=True, px=5000):
            try:
                # Another owner may have published between our first GET and SET.
                cached = cache.get(data_key)
                if cached is not None:
                    return json.loads(cached)
                data = fetch(category, page)
                payload = json.dumps(data, allow_nan=False, separators=(",", ":"))
                if len(payload.encode()) > 64 * 1024:
                    raise ValueError("Catalog page exceeds this example's 64-KiB cap")
                ttl = 270 + secrets.randbelow(61)
                if cache.eval(PUBLISH, 2, data_key, lock_key, token, payload, ttl) != 1:
                    raise CacheBusy("Lease expired before publication")
                return data
            finally:
                cache.eval(RELEASE, 1, lock_key, token)
        if time.monotonic() >= deadline:
            raise CacheBusy("Refill already in progress")
        time.sleep(0.02)

The 200-ms deadline bounds the contention loop between calls, not total wall-clock latency: network operations and fetch need their own timeouts. The database fetch must be a bounded, authorized read with deterministic ordering and at most one page of results. Amounts use integer cents or an explicit decimal encoding. The 64-KiB cap is this application's policy, not the engine's universal limit. Keys are owned exclusively by this component and have the expected string types. Redis lease ownership, Lua execution.

This is a cache-refill optimization, not a lock protecting payments or cross-system correctness. Lease expiry or asynchronous replica failover can permit overlapping fetches; token checking constrains publication on the current primary. A network timeout can also leave an uncertain command outcome. The helper propagates cache errors, malformed cache data and CacheBusy; the HTTP layer can return a bounded 503 with Cache-Control: no-store. CloudFront applies separate error-caching rules to an origin 503: no-store alone does not disable its Error Caching Minimum TTL, which defaults to ten seconds. It can also serve an expired cached representation after an origin 5xx. Configure error caching and stale-response behavior for the catalog's failure/freshness contract, then test failure and recovery. Error-cache duration, origin-error handling. Any database fallback needs its own concurrency/rate budget. Sending every failed-cache request straight to the database can cause a second outage.

A successful public response can use:

def catalog_response_headers():
    return {"Content-Type": "application/json",
            "Cache-Control": "public, max-age=0, s-maxage=60"}

TTL jitter here spreads expiry between 270 and 330 seconds; it does not by itself stop many callers competing for one key. Do not put cache-source diagnostics inside the response body, where CloudFront could replay them as if they described the current request. Observe hit/miss/error behavior through telemetry instead.

6. Application: expiring sessions without identity overwrite

A session token is a bearer credential. Create it after authentication, use a secure cookie with an appropriate HttpOnly, Secure and SameSite policy, and do not log it. Keep authenticated identity separate from user-editable profile fields. A cache outage must not cause the application to invent an authenticated session.

This component uses fixed one-hour expiration, not sliding activity renewal. It stores one JSON string with expiry in the same SET operation, avoiding the crash gap between HSET and EXPIRE:

import json
import re
import secrets

TOKEN = re.compile(r"[A-Za-z0-9_-]{43}\Z")

def new_session(cache, authenticated_user_id):
    if not isinstance(authenticated_user_id, str) or not authenticated_user_id:
        raise ValueError("Authenticated user ID is required")
    for _ in range(3):
        token = secrets.token_urlsafe(32)
        value = json.dumps({"user_id": authenticated_user_id})
        if cache.set("session:" + token, value, ex=3600, nx=True):
            return token
    raise RuntimeError("Could not allocate a session token")

def get_session(cache, token):
    if not isinstance(token, str) or TOKEN.fullmatch(token) is None:
        return None
    raw = cache.get("session:" + token)
    if raw is None:
        return None
    value = json.loads(raw)
    if (not isinstance(value, dict) or set(value) != {"user_id"}
            or not isinstance(value["user_id"], str) or not value["user_id"]):
        raise ValueError("Invalid stored session")
    return value

def revoke_session(cache, token):
    if isinstance(token, str) and TOKEN.fullmatch(token):
        cache.delete("session:" + token)

Authorization and session rotation are separate from this lookup. Recheck sensitive privileges against the authoritative policy; do not persist a client-supplied role=admin. The session store must be private and access-controlled. A token may disappear before its hour because of eviction or a failure; choose memory/eviction/recovery policy for the application's login availability needs. A request that read a session just before revocation may already be in flight, so deleting a key is not instantaneous cancellation of every operation. Redis SET options, OWASP session guidance.

Pipelines and transactions are not synonyms. In redis-py, pipeline() normally wraps queued commands in MULTI/EXEC; pipeline(transaction=False) batches without that isolation. Neither is a rollback-capable transaction with an external database. Runtime errors do not imply that every other command was undone. Use a single command where it expresses the required mutation cleanly. redis-py batches, Redis transactions.

7. Application: a leaderboard with explicit retry semantics

Sorted sets associate unique members with numeric scores. ZADD sets scores; ZINCRBY adds a delta atomically. Atomic increment is not retry-idempotent: sending the same award twice can count it twice. Persist/deduplicate score events in the authoritative system, then publish an absolute cumulative total.

For this monotonically increasing season score, GT rejects an older lower total while still allowing new members. This choice is unsuitable for a score that can decrease because of penalties. All writes must follow the same contract, and authoritative event processing must prevent lost increments upstream:

class Leaderboard:
    def __init__(self, cache, season):
        if not isinstance(season, str) or not season:
            raise ValueError("Season required")
        self.cache = cache
        self.key = "leaderboard:" + season

    def publish_total(self, player, score):
        if not isinstance(player, str) or not player:
            raise ValueError("Player required")
        if type(score) is not int or not 0 <= score <= 2**53 - 1:
            raise ValueError("Score outside exact integer range")
        self.cache.zadd(self.key, {player: score}, gt=True)

    def top(self, count=10):
        if type(count) is not int or not 1 <= count <= 100:
            raise ValueError("Count must be 1–100")
        return [(player, int(score)) for player, score in
                self.cache.zrevrange(self.key, 0, count - 1, withscores=True)]

    def rank(self, player):
        rank = self.cache.zrevrank(self.key, player)
        return None if rank is None else rank + 1

Scores use IEEE-754 doubles, so the example stays within the exact integer range. ZADD is O(log N) per member; retrieving M ranked entries is O(log N + M), not O(log N) regardless of output size. A count of zero would turn an unguarded end index into -1, retrieving the whole set; validate bounds. Equal scores use reverse lexicographical member ordering for these descending commands. rank gives an ordinal position, not a shared competition rank. ZADD, ZREVRANGE, ZREVRANK.

With Bob at 2,300, Carol at 1,800 and Alice at 1,700, the top three are Bob, Carol, Alice. Publishing Alice's 1,700 twice or receiving an older 1,500 total does not increase or reduce her stored score. Rebuild the leaderboard from durable authoritative data after loss; expire/archive season keys using a deliberate retention job. A single global sorted-set key can become a hot shard, so benchmark the actual workload instead of assuming automatic unlimited scaling.

8. Application: DAX-backed article reads

DAX has separate item and query caches. Both default TTLs are five minutes, but their behavior differs: cached Query/Scan results do not populate the item cache, and item writes do not invalidate existing query-cache results. A repeated query can remain stale until its own expiry even after a successful write through DAX. Direct writes that bypass DAX can also leave its item cache stale. DAX caches, consistency details.

Strongly consistent reads are supported through DAX: it forwards them to DynamoDB and does not cache the returned results. ConsistentRead=True does not inherently fail. Base-table/LSI support still differs from GSIs, which cannot serve strong reads. Transactional reads likewise pass through. BatchGetItem can benefit from the item cache; “batch operations are not useful with DAX” is too broad.

Use the DAX client, cluster discovery endpoint, VPC connectivity, IAM and the correct transport. For an encrypted cluster, use the reported daxs:// endpoint, normally port 9111, rather than the unencrypted dax:///8111 combination. DAX is not a replacement for table-management APIs. DAX encryption in transit, Python client.

from amazondax import AmazonDaxClient

def connect_dax(discovery_endpoint, region):
    return AmazonDaxClient(endpoint_url=discovery_endpoint, region_name=region)

class Articles:
    def __init__(self, client, table_name="Articles"):
        self.client = client
        self.table_name = table_name

    def get(self, article_id, require_fresh=False):
        # Articles has only articleId as its primary key.
        response = self.client.get_item(
            TableName=self.table_name,
            Key={"articleId": {"S": article_id}},
            ConsistentRead=require_fresh)
        return response.get("Item")

    def newest(self, category, limit=20, start_key=None):
        # GSI: category (string partition key), publishedAt (numeric sort key).
        if type(limit) is not int or not 1 <= limit <= 100:
            raise ValueError("Limit must be 1–100")
        params = {
            "TableName": self.table_name, "IndexName": "CategoryPublishedIndex",
            "KeyConditionExpression": "#category = :category",
            "ExpressionAttributeNames": {"#category": "category"},
            "ExpressionAttributeValues": {":category": {"S": category}},
            "ScanIndexForward": False, "Limit": limit
        }
        if start_key is not None:
            params["ExclusiveStartKey"] = start_key
        response = self.client.query(**params)
        return response.get("Items", []), response.get("LastEvaluatedKey")

Provision the table and GSI separately with projections for the fields the caller needs. This low-level API uses DynamoDB attribute-value maps, not ordinary Python item dictionaries. newest returns one page and its continuation key; the index orders by publication timestamp, not “trending popularity.” Equal timestamps need a tie-breaker design if deterministic cross-page ordering matters. Validate any exposed cursor against the authenticated query scope. A fresh article lookup does not refresh the cached category listing.

Benchmark cold and warm reads, representative keys, concurrent load, cache evictions, errors and freshness after writes. A loop reading one hot key a hundred times measures that scenario, not a universal 10–20× improvement. DAX adds cluster cost and a network hop on misses; use it when measured reuse and latency benefits justify the consistency and operational tradeoffs.

9. Measure usefulness, not just hit rate

For interval counters, hit rate is hits / (hits + misses). With 800 hits and 200 misses it is 80%. No requests means no measured hit rate, not 100%. Cumulative engine counters need deltas and reset handling; they may include operations other than the application requests you are measuring.

At 1,000 requests/second, an 80% hit rate leaves 200 misses/second. At 81%, it leaves 190: ten fewer misses, or a 5% reduction in misses, not a guaranteed 10–20% infrastructure saving. Under a simplified model with 1-ms hits and 11-ms misses, an 80% hit rate gives 0.8 × 1 + 0.2 × 11 = 3 ms mean latency. Tail latency, queueing and outage load require separate measurements.

Track hit/miss/error rates, p95/p99 latency, evictions, bytes, CPU, connections, source load and freshness violations. A cache can show a high hit rate while leaking tenant data or serving obsolete permissions. Test wrong-key isolation, expired values, owner replacement during refill, retry duplicates and a full cache outage before declaring the design successful. The right TTL and hit-rate target come from the application's workload and correctness budget.