Storage & Data Services

Storage & Data Services

A photo application needs several kinds of storage. Images are large blobs, user accounts need transactions, a shared working directory needs file operations, and a dashboard scans many historical records. Putting all of these into one service usually makes the application harder to operate.

This lesson teaches how to choose object, block, file, database, and transfer services, then combines them in four applications. Examples assume current boto3, suitable IAM permissions, configured credentials, and resources in the stated Region. Resource creation is asynchronous unless a waiter is shown. Names are illustrative; creation examples require names you own and compatible service configurations.

1. Choose an access model before choosing a product

An object store is not a mounted block device, and a cache does not automatically become the authoritative database. The right choice starts with operations and failure requirements. AWS storage, database services.

2. S3: objects, retention, and access

An S3 object has a key, bytes, and metadata; a versioned object also has a version ID. A slash in photos/u123/image.jpg is part of the key, not a real directory hierarchy. These examples use general purpose buckets, rather than directory or table buckets. In the traditional shared namespace, names are unique across accounts and Regions within an AWS partition. S3 also supports an account regional namespace with its own naming convention. Bucket naming.

S3 now documents objects up to 50 TB, increased from the old 5 TB limit; the precise multipart ceiling is about 53.7 decimal TB (48.8 TiB). Large objects require multipart upload, and individual operation limits still apply. Do not confuse the total object ceiling with a single PUT request. Object limits, upload methods.

S3 Standard is designed for 99.999999999% annual durability and stores data across at least three AZs. Durability describes preservation of data, not uptime, application correctness, or protection from authorized deletion. One Zone storage has a different failure-domain tradeoff. Keep backup/version retention appropriate to the recovery requirement. Storage-class comparison.

Lifecycle transition age is not minimum storage billing duration. You can transition directly from Standard to Deep Archive; no mandatory Standard-IA → Glacier chain exists. Since July 16, 2026, transitions to Standard-IA/One Zone-IA can be configured from day zero; their thirty-day minimum storage billing duration still applies. Deep Archive's 180-day minimum likewise concerns billing after entering that class. IA transition update. New/updated lifecycle configurations normally exclude objects smaller than 128 KiB from transitions unless explicitly configured otherwise. Transition constraints, S3 pricing.

All new S3 object uploads receive server-side encryption by default. An omitted encryption field does not mean plaintext storage. Choose SSE-KMS when its key controls/auditing are needed, and provide applicable KMS permissions. New general purpose buckets normally have Block Public Access enabled and ACLs disabled through Bucket owner enforced Object Ownership. Presigned URLs delegate the signer's permitted operation; they are bearer credentials with limited validity. Default encryption, Object Ownership, presigned access.

3. EBS and EFS: disks versus shared files

EBS volumes attach to EC2 instances in the same Availability Zone. A snapshot can create a new volume in another AZ; it does not move the original attachment. Snapshots are managed by EBS and are not files in a customer S3 bucket that you manage with S3 lifecycle or CRR. They are incremental at the storage level, while each retained snapshot is usable as a recovery point. Quiesce application writes or use an application-consistent backup procedure when required. EBS snapshots.

These are volume capabilities, not an unconditional application throughput promise. Instance EBS limits, I/O size, queue depth, and workload also constrain results. gp3 on Outposts has lower limits. Since April 30, 2025, all io2 volumes use Block Express; an old table separating a universal 64,000-IOPS “io2” tier from Block Express is misleading. gp3/gp2, io2, HDD volumes.

For example, gp3's 80,000-IOPS maximum requires at least 160 GiB: 500 × 160 = 80,000. Its 2,000 MiB/s maximum requires at least 8,000 provisioned IOPS and 16 GiB. Additional performance is charged separately. An ordinary 100-GiB example can start at the included baseline:

aws ec2 create-volume --volume-type gp3 --size 100 --availability-zone us-east-1a --iops 3000 --throughput 125 --encrypted
aws ec2 create-snapshot --volume-id vol-0123456789abcdef0 --description "Application-quiesced recovery point"

The second command uses an existing illustrative volume ID; use the actual returned ID after the volume and application are ready. Attaching a volume does not automatically partition, format, mount, or extend its filesystem. Elastic Volumes can change supported volume properties online, but filesystem and application work may remain. Supported io1/io2 Multi-Attach is a specialized exception to the usual single-instance pattern; it needs compatible instances in one AZ and software that coordinates shared writes. It does not turn an ordinary filesystem into EFS. Multi-Attach, volume modifications.

EFS provides shared NFS file access. Regional EFS stores data across multiple AZs; One Zone has a different resilience/cost tradeoff. Its storage offerings include Standard, Infrequent Access, and Archive, with availability and performance depending on filesystem configuration. A filesystem ID alone is not a mount: prepare mount targets, security-group access to NFS, client mount software, and filesystem/POSIX permissions. Use TLS for in-transit protection and an encrypted filesystem for at-rest protection. EFS storage classes, mount targets.

EC2 in AZ A ── attached block volume in AZ A

Linux client in AZ A ── NFS mount target ─┐
                                        ├── shared Regional EFS filesystem
Linux client in AZ B ── NFS mount target ─┘

4. Relational databases: distinguish availability from read scaling

RDS manages supported relational engines, including MySQL, PostgreSQL, MariaDB, Oracle, SQL Server, and Db2. Aurora is a related managed relational service compatible with MySQL or PostgreSQL. AWS operates infrastructure and supported backup/maintenance workflows; you still own schema design, access, queries, capacity choices, and recovery testing. Storage autoscaling is not automatic resizing of every DB instance's CPU and memory. RDS overview, storage autoscaling.

Use the writer when a workflow needs to read a just-committed change and its replica might lag. An Aurora reader endpoint is not the endpoint format for every ordinary RDS instance. Multi-AZ failover requires connection recovery and is not a fixed-duration guarantee. Marketing comparisons such as “5× MySQL” or “3× PostgreSQL” describe particular benchmarks, not every query. Aurora storage ceilings and supported features vary by engine/version. Aurora Global Database extends supported Aurora deployments across Regions: a primary Region accepts writes, while secondary Regions replicate its data and serve regional reads. Supported versions and Regions matter; measure replication lag and practice switchover/failover rather than assuming a fixed RPO or RTO. Aurora Global Database. Multi-AZ instance, Multi-AZ cluster, read replicas, Aurora architecture.

An RDS MySQL creation example can ask RDS to manage its master password in Secrets Manager rather than put a password in lesson code:

import boto3

rds = boto3.client("rds", region_name="us-east-1")

def create_mysql(subnet_group, security_group):
    return rds.create_db_instance(
        DBInstanceIdentifier="photo-metadata-db",
        DBInstanceClass="db.m7g.large", Engine="mysql",
        MasterUsername="dbadmin", ManageMasterUserPassword=True,
        AllocatedStorage=100, StorageType="gp3", StorageEncrypted=True,
        MultiAZ=True, BackupRetentionPeriod=7,
        PubliclyAccessible=False, DBSubnetGroupName=subnet_group,
        VpcSecurityGroupIds=[security_group])

The DB subnet group must cover appropriate AZs, and the engine/class combination must be offered in the Region. Use the DB availability waiter before connecting, retrieve credentials securely, and verify TLS certificates. This call does not create a read replica. Automated-backup retention can extend up to 35 days, but recovery is limited to the actually available restore window and creates a new instance. Managed passwords, automated backups.

5. DynamoDB: design the key and the recovery contract

DynamoDB stores items addressed by a partition key and an optional sort key. A GetItem requires the complete primary key. A Query selects one partition-key value and can constrain the sort key; a filter does not turn a scan into an efficient key lookup. Read every page when the result exceeds one response. Query.

On-demand capacity bills usage and adapts within its scaling rules and quotas. Provisioned capacity reserves throughput; autoscaling needs both a scalable target and a scaling policy. Neither model fixes a hot partition or an unsuitable key design. A strongly consistent base-table read reflects completed writes when the read occurs; it is not a lock preventing a later writer. Global secondary indexes use eventually consistent reads. Capacity modes, read consistency.

import boto3
from boto3.dynamodb.conditions import Key

db = boto3.resource("dynamodb", region_name="us-east-1")

def create_activity_table():
    table = db.create_table(
        TableName="UserActivity",
        KeySchema=[{"AttributeName": "userId", "KeyType": "HASH"},
                   {"AttributeName": "sequence", "KeyType": "RANGE"}],
        AttributeDefinitions=[{"AttributeName": "userId", "AttributeType": "S"},
                              {"AttributeName": "sequence", "AttributeType": "N"}],
        BillingMode="PAY_PER_REQUEST")
    table.wait_until_exists()
    return table

def read_activity(table, user_id):
    args = {"KeyConditionExpression": Key("userId").eq(user_id),
            "ConsistentRead": True}
    rows = []
    while True:
        page = table.query(**args)
        rows.extend(page.get("Items", []))
        if not page.get("LastEvaluatedKey"):
            return rows
        args["ExclusiveStartKey"] = page["LastEvaluatedKey"]

For this composite key, retrieving sequence 1 requires Key={"userId": "u123", "sequence": 1}. Numeric attributes in boto3's resource interface require integers or Decimal, not native Python floats. Individual items have a 400 KiB limit including attribute names and values. Large images belong in an object store, with references in DynamoDB. DynamoDB constraints, boto3 DynamoDB guide.

Enable point-in-time recovery deliberately. Its configurable recovery period is 1–35 days, and restoration is restricted to the reported earliest/latest restorable timestamps; enabling it today does not create 35 days of prior history. TTL deletion is asynchronous, typically within a few days, so expired items may still appear and need application-side filtering. Streams capture item changes for 24 hours. Global tables' current version supports multi-Region eventual consistency (MREC) and multi-Region strong consistency (MRSC) with different constraints; do not assume all replicas have the same freshness contract. PITR, TTL, global tables.

6. Analytics, caching, and hybrid access

Redshift is a managed analytical warehouse, with provisioned and serverless deployment choices. Its columnar storage and parallel execution suit scans and aggregations rather than replacing every transactional database. Spectrum can query supported data in S3. Data format, table layout, permissions, and load idempotency remain your responsibility. Redshift overview.

ElastiCache supports Valkey, Redis OSS, and Memcached. Features depend on engine, version, and node-based versus serverless deployment. Avoid a universal “Redis is single-threaded, Memcached has no replication, Redis always has AOF” comparison: managed-service capabilities and serverless architecture differ from a simple self-hosted engine comparison. Current node-based ElastiCache for supported Valkey configurations also offers optional durability backed by a Multi-AZ transactional log. A traditional disposable cache and that explicitly configured durable service have different contracts. ElastiCache architecture, durability.

For a cache-aside profile lookup: read cache; on a miss, read the authoritative database; cache the result for a bounded time. After a profile update, invalidate or refresh the cached value, recognizing that concurrent reads can still race with invalidation. A cache outage needs a controlled fallback so it does not overwhelm the database. Sessions that must survive failures need an explicitly chosen persistence/recovery design. TTL alone does not guarantee freshness. Caching considerations.

DataSync moves supported data between storage locations over a network. Storage Gateway exposes hybrid interfaces: S3 File Gateway provides NFS/SMB shares backed by S3, Volume Gateway provides iSCSI volumes with cached/stored modes, and Tape Gateway presents a virtual tape library. These do not expose arbitrary customer S3 buckets as RDS/EBS snapshots; their APIs and recovery procedures matter. DataSync, S3 File Gateway, Volume Gateway, Tape Gateway.

Old Snow Family shopping tables are not a current migration plan. Snowball Edge closed to new customers on November 7, 2025; existing-customer eligibility and device availability must be checked. Snowcone was discontinued in November 2024. Evaluate available network transfer, connectivity, and partner/offline options instead of prescribing a truck or an old 80-TB device to every new customer. Data size alone is not a service-selection rule. Snowball availability, Snow device updates.

7. Application: photo sharing with distinct data paths

Browser ── /assets/* ── CloudFront ── private S3 REST origin
   │
   └── API requests ── load balancer ── application
                                         ├── RDS: users, photo ownership
                                         ├── ElastiCache: disposable profile cache
                                         └── S3: original/processed image bytes

The static-content path does not flow through S3 and then into the load balancer. CloudFront can instead route separate cache behaviors to an S3 origin and an application origin. Origin Access Control (OAC) lets CloudFront access a private S3 REST origin; an S3 website endpoint is a different integration and cannot use OAC. Private S3 origins.

An authorized application can upload already-validated JPEG bytes to a versioning-enabled bucket, then return a short-lived download URL for that exact version:

from botocore.config import Config

s3 = boto3.client("s3", region_name="us-east-1",
                  config=Config(signature_version="s3v4"))

def store_photo(bucket, key, jpeg_bytes):
    result = s3.put_object(
        Bucket=bucket, Key=key, Body=jpeg_bytes,
        ContentType="image/jpeg", ServerSideEncryption="AES256")
    version = result["VersionId"]
    url = s3.generate_presigned_url(
        "get_object", Params={"Bucket": bucket, "Key": key, "VersionId": version},
        ExpiresIn=300)
    return {"bucket": bucket, "key": key, "versionId": version, "downloadUrl": url}

The caller derives the permitted key from the authenticated user and validates the image's type and size. The uploader needs s3:PutObject; the signing identity needs s3:GetObjectVersion. URL validity can end earlier if temporary credentials expire. Do not log/share these bearer URLs casually. A plain private-object URL would not authorize a browser to read it.

Store the returned bucket/key/version and ownership metadata in the database. Upload and metadata commit are separate operations: use an operation record and reconciliation to recover orphaned uploads or uncertain retries. Retrying this PUT can create another version; the function is not an exactly-once upload primitive. Lifecycle policies can transition older photos to GLACIER_IR when their access pattern justifies it, but measure request, retrieval, and minimum-duration costs instead of promising a fixed percentage saving.

8. Application: migrate 50 TB while the source remains active

Start with bandwidth arithmetic. At a sustained 1 Gbit/s, 50 decimal TB requires 50 × 10^12 × 8 / 10^9 = 400,000 seconds, about 4.63 days, before protocol overhead or competing traffic. At 100 Mbit/s it is about 46.30 days. Compare measured throughput, source change rate, downtime tolerance, and available transfer services.

An initial DataSync copy followed by incremental runs is viable when the connection can catch up with changes. Create supported source/destination locations and an activated agent where the source requires one. The following uses existing location ARNs, avoiding embedded SMB passwords and pretending a task creates its agent:

datasync = boto3.client("datasync", region_name="us-east-1")

def create_migration_task(source_arn, destination_arn):
    return datasync.create_task(
        Name="FilesToS3", TaskMode="BASIC",
        SourceLocationArn=source_arn, DestinationLocationArn=destination_arn,
        Options={"TransferMode": "CHANGED", "OverwriteMode": "ALWAYS",
                 "PreserveDeletedFiles": "PRESERVE",
                 "VerifyMode": "ONLY_FILES_TRANSFERRED"},
        Schedule={"ScheduleExpression": "rate(1 hour)"})["TaskArn"]

def start_migration(task_arn):
    return datasync.start_task_execution(TaskArn=task_arn)["TaskExecutionArn"]

This example assumes a supported file-source-to-S3 Standard destination and a role permitting that location's S3 access. CHANGED supports an initial transfer followed by differences. PRESERVE keeps destination files that no longer exist at the source, so this is not an exact deletion mirror. ONLY_FILES_TRANSFERRED verifies transferred data at completion; it is not a transactionally consistent snapshot of a running database. Task options.

Scheduled tasks can run no more frequently than hourly. Creating the task is not completing a transfer: track the returned execution ARN until success or error, inspect verification reports and transferred counts, then validate application reads. At cutover, pause or otherwise coordinate source writes, run and verify the final delta, switch clients, and retain a rollback plan. Use database-aware tooling such as DMS when database change capture is the requirement. Scheduling, DMS.

9. Application: transform CSV into warehouse-ready JSON

Versioned raw S3 bucket → upload notification → transform Lambda
                                                   ↓
                                    processed S3: one JSON object per line
                                                   ↓
                                    durable load coordinator → Redshift

The transform should handle every notification record, decode the key, read the named source version, and use a deterministic output identity. Separate input/output buckets prevent a transform's own writes from recursively triggering it. This example bounds input to 2 MiB and assumes the trusted CSV contract id,email,created,status; dates are ISO dates and status is active or inactive.

import csv
import hashlib
import io
import json
from datetime import date
from urllib.parse import unquote_plus

def csv_to_jsonl(raw):
    reader = csv.DictReader(io.StringIO(raw.decode("utf-8-sig")))
    if reader.fieldnames != ["id", "email", "created", "status"]:
        raise ValueError("Unexpected CSV header")
    output = []
    for row in reader:
        if None in row or any(v is None for v in row.values()):
            raise ValueError("Malformed CSV row")
        if not row["id"] or row["status"] not in ("active", "inactive"):
            raise ValueError("Invalid identity/status")
        transformed = {
            "user_id": row["id"], "email": row["email"].strip().lower(),
            "signup_date": date.fromisoformat(row["created"]).isoformat(),
            "is_active": row["status"] == "active"}
        output.append(json.dumps(transformed, separators=(",", ":")))
    return ("\n".join(output) + ("\n" if output else "")).encode("utf-8")

def transform_uploads(event, output_bucket):
    if event.get("Event") == "s3:TestEvent":
        return []
    outputs = []
    for record in event["Records"]:
        source = record["s3"]
        bucket = source["bucket"]["name"]
        key = unquote_plus(source["object"]["key"])
        version = source["object"].get("versionId")
        if bucket == output_bucket or not key.startswith("raw/") or not key.endswith(".csv"):
            raise ValueError("Unexpected source")
        if not version or version == "null":
            raise ValueError("A named source version is required")
        response = s3.get_object(Bucket=bucket, Key=key, VersionId=version)
        stream = response["Body"]
        try:
            raw = stream.read(2 * 1024 * 1024 + 1)
        finally:
            stream.close()
        if len(raw) > 2 * 1024 * 1024:
            raise ValueError("Use a larger-file processing path")
        identity = json.dumps([bucket, key, version], separators=(",", ":"))
        output_key = "processed/" + hashlib.sha256(identity.encode()).hexdigest() + ".jsonl"
        s3.put_object(Bucket=output_bucket, Key=output_key,
                      Body=csv_to_jsonl(raw), ContentType="application/x-ndjson")
        outputs.append(output_key)
    return outputs

The example application normalizes email case by its own contract; it is not a universal rule that all email local parts are case-insensitive. Bound record/field lengths and resource use for untrusted inputs. S3 notifications can duplicate or arrive out of order. Reading the notified version prevents a later overwrite from changing the transformation input. Retrying writes the same transformed bytes to the same output key; it can still emit another downstream notification. S3 event structure, notification behavior.

Use newline-separated root JSON objects for a conventional Redshift COPY ... JSON 'auto' into matching columns. An array wrapping all rows is one root structure, not automatically one row per member. A loader should use a manifest listing exact objects, a fixed/trusted destination, and a durable batch identity. Do not interpolate arbitrary event keys into SQL, and do not treat repeating COPY as idempotent. Stage, validate, and merge/record completion under a coordinated transaction strategy; Redshift primary-key declarations alone do not enforce uniqueness. JSON COPY, load manifests, constraints.

10. Application: prove a disaster-recovery target

An RPO of fifteen minutes limits acceptable data loss; an RTO of one hour limits acceptable recovery time. Neither is established merely by enabling a replication checkbox.

S3 source ── configured S3 CRR ── S3 destination
RDS source ── supported replica or snapshot/backup copy ── RDS recovery resources
DynamoDB ── chosen global-table replication mode ── regional replica
                  plus independent backups and restore procedures

RDS automated snapshots are not ordinary objects in your application bucket. S3 CRR does not replicate them. Copy RDS snapshots with RDS/AWS Backup mechanisms, or select supported cross-Region replication/backup features. Exporting an RDS snapshot to S3 produces analytical Parquet data; it is not a native restorable RDS snapshot. RDS snapshot copying, snapshot exports.

For a compatible existing MREC DynamoDB table using global tables version 2019.11.21, adding a replica uses UpdateTable, not the legacy CreateGlobalTable API:

ddb = boto3.client("dynamodb", region_name="us-east-1")

def add_west_replica(table_name):
    return ddb.update_table(
        TableName=table_name,
        ReplicaUpdates=[{"Create": {"RegionName": "us-west-2"}}])

Prepare permissions, quotas, encryption and supported table settings, and wait until the replica is active. MREC replication is asynchronous; MRSC has a different supported topology and consistency contract. Neither replication mode substitutes for recovery from an authorized but erroneous write. Current global-table setup.

S3 Replication Time Control is designed to replicate 99.99% of eligible new objects within fifteen minutes. Its SLA uses a 99.9% monthly fifteen-minute replication threshold per Region pair per account, subject to conditions and exclusions. RTC SLA. This is not “every object within fifteen minutes,” nor an application-wide RPO promise. Existing objects require a separate replication plan, such as Batch Replication. The next lesson provides the complete replication configuration and deletion caveats. S3 RTC.

A recovery exercise should measure source/destination lag, provision or activate compute, make keys/secrets/networking available, restore or promote data, switch endpoints, and run application checks. If detection, recovery, and validation/switching take 10, 35, and 20 minutes respectively, the total is 65 minutes, missing a one-hour RTO even if storage recovery alone took only 35 minutes. Record measured results and dependencies, not just service marketing times.

Practice: choose one data type in the photo application. Describe its authoritative location, access path, encryption/permissions, stale-read behavior, and tested recovery procedure. Then explain why another service in the table would make at least one of those requirements harder to satisfy.