S3 Mastery

S3 Mastery

S3's basic operations are simple: put bytes under a key and retrieve them later. The harder questions are whether a caller is authorized, which version they read, when archival data becomes usable, and what a retry does after part of a workflow succeeds.

This lesson develops those questions through private website delivery, a tiered data lake, temporary document sharing, and image processing. It focuses on general purpose buckets. Directory buckets, S3 Express One Zone, table buckets, and other specialized offerings have distinct constraints. Examples use current boto3; resources, credentials, IAM/KMS permissions, and the specified Region must be prepared separately unless shown.

1. Object identity, limits, and consistency

Bucket in an AWS Region
  key: documents/contracts/nda.pdf
    version A → bytes + metadata
    version B → different bytes + metadata

The full key identifies an object; slashes form useful prefixes but do not create ordinary filesystem directories. A key plus version ID identifies a particular stored version. Traditional shared-namespace bucket names are unique across accounts/Regions within a partition; S3 also supports an account regional namespace with its own naming scheme. Choose the namespace deliberately and use the actual bucket name, never a shell wildcard. Naming rules.

S3 now documents a 50 TB object limit, rather than the old 5 TB limit. Its exact multipart ceiling follows 10,000 parts of up to 5 GiB: about 53.7 decimal TB, or 48.8 TiB. A single PUT and single GET have separate limits; the largest objects need multipart upload and ranged GETs. A small PutObject example does not demonstrate uploading a 50-TB object. Object sizes, multipart limits, downloading objects.

S3 Standard's 99.999999999% annual durability design target is different from availability and an SLA. The arithmetic behind the familiar ten-million-object illustration is 10,000,000 × (1 − 0.99999999999) = 0.0001 expected lost objects per year under that simplified model. Its reciprocal is 10,000 years per expected loss; this is not a schedule or a promise for your bucket. Deletion, compromised permissions, software mistakes, and correlated failures require their own controls. S3 durability.

S3 provides strong read-after-write consistency for object PUT/DELETE and subsequent GET/LIST operations. After a successful overwrite, a later read reflects the new object unless another write intervenes. That does not mean atomic transactions across keys, instantly propagated bucket configuration, or cache invalidation at CloudFront. Concurrent writers still need conditional operations or application coordination. Consistency model.

2. Authorization is policy evaluation, not a serial firewall stack

S3 considers the request identity, resource, applicable policies, and conditions. An explicit applicable deny overrides an allow. Within one account, identity and resource policy allows generally combine; it is incorrect to say their “most restrictive permission always wins.” Cross-account access, permission boundaries, session policies, organization controls, endpoint policies, and KMS authorization add their own rules. IAM evaluation.

Here is a bucket-policy allow statement for one named role using a public corporate egress range. 203.0.113.0/24 is an example address range to replace, not a real office network:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowReportRoleFromOffice",
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::123456789012:role/ReportReader"},
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::example-private-reports/*",
    "Condition": {"IpAddress": {"aws:SourceIp": "203.0.113.0/24"}}
  }]
}

This grants access through this statement when its conditions match. It does not deny another authorization path outside that IP range. If the requirement is a mandatory perimeter, design and test appropriate deny conditions while preserving required AWS service access. aws:SourceIp is not the key to identify an arbitrary private subnet through an S3 VPC endpoint; use appropriate keys such as aws:SourceVpce or aws:VpcSourceIp, with the full endpoint/VPC context. S3 policy conditions, global condition keys.

For ordinary private buckets, keep Block Public Access enabled and use Bucket owner enforced Object Ownership, which disables ACLs. The latter is the default for new buckets. A public website endpoint is a deliberate alternative with different access constraints; CloudFront OAC with an S3 REST origin supports private delivery. Block Public Access, Object Ownership.

3. Encryption and the boundaries of Bucket Keys

New uploads are encrypted even when ServerSideEncryption is omitted. Bucket default encryption affects new writes; it does not retroactively re-encrypt all existing objects. The April 2026 rollout blocks SSE-C writes by default in new general purpose buckets, except in Middle East (Bahrain) and Middle East (UAE). Existing buckets were changed only in accounts with no SSE-C objects; accounts already using SSE-C retained their existing bucket configurations. Enable SSE-C explicitly where the bucket blocks it and the application requires it. Server-side encryption does not encrypt the network connection: use HTTPS as well. Default encryption, SSE-C defaults.

import boto3

s3 = boto3.client("s3", region_name="us-east-1")

def put_confidential(bucket, key, body, kms_key_arn):
    return s3.put_object(
        Bucket=bucket, Key=key, Body=body,
        ServerSideEncryption="aws:kms", SSEKMSKeyId=kms_key_arn,
        BucketKeyEnabled=True)

The KMS key must be suitable for the bucket's Region, and authorization must cover both S3 and the required KMS operations. Bucket Keys reduce the S3-to-KMS request traffic for supported SSE-KMS operations and can reduce KMS request costs by up to 99%. Savings are workload-dependent; this is not a guaranteed 99% reduction in your entire S3 bill. Do not model every conceivable S3 request as exactly one KMS call or memorize a single fixed KMS quota for all Regions. Bucket Keys, KMS quotas.

4. Storage classes and lifecycle: eligibility versus billing

One Zone-IA's single-AZ placement makes it suitable for re-creatable data, not the only copy of irreplaceable data. Intelligent-Tiering includes monitoring/automation costs for eligible objects. Glacier Flexible Retrieval offers several retrieval options; Deep Archive's standard retrieval is typically within twelve hours and bulk within forty-eight hours. Do not promise the same latency for all archives or compare only storage price. Storage classes, archive retrieval options.

You can upload directly to Deep Archive or transition there directly from Standard. The 180-day minimum is a billing commitment, not a requirement to keep an object in Standard for 180 days first. The former thirty-day age restriction for transitions to Standard-IA/One Zone-IA was removed on July 16, 2026: a rule can now specify day zero. Their thirty-day minimum storage billing duration still applies. IA transition update. A single lifecycle rule that moves between classes must also respect the prior class's minimum duration; separating rules can incur remaining-duration charges. Lifecycle constraints.

This complete configuration for an unversioned log bucket transitions eligible logs/ objects to Standard-IA at thirty days, to Glacier Flexible Retrieval at ninety days, and expires them at 365 days. Lifecycle actions are asynchronous after eligibility, not exact wall-clock timers:

{
  "Rules": [{
    "ID": "ArchiveLogs",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Transitions": [
      {"Days": 30, "StorageClass": "STANDARD_IA"},
      {"Days": 90, "StorageClass": "GLACIER"}
    ],
    "Expiration": {"Days": 365},
    "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
  }]
}

These classes remain long enough before the next step: sixty days in Standard-IA, then 275 days in Flexible Retrieval. In a versioning-enabled bucket, current-version expiration normally adds a delete marker and makes the data version noncurrent; it does not erase every retained version. Add a deliberate noncurrent-version retention rule where appropriate. Expiration behavior.

New or modified lifecycle configurations normally do not transition objects below 128 KiB. An explicit size filter can change that behavior, but transition requests and archive metadata overhead can make tiny-object transitions uneconomic. Minimum billable size and minimum storage duration are different dimensions. Small-object transitions.

For a transparent cost exercise, assume illustrative storage rates of $0.023 and $0.0125 per billed GB-month. For 1,000 billed GB held for a full month, the storage-only amounts are $23 and $12.50: a $10.50 or 45.65% reduction. Retrieval, transitions, requests, minimum charges, Region/tier, and transfer may change the total. “Lower per-GB storage rate” does not establish “cheapest workload.” S3 pricing.

5. Versioning and replication are separate controls

Enable versioning using the exact status value:

aws s3api put-bucket-versioning --bucket example-documents --versioning-configuration Status=Enabled
aws s3api list-object-versions --bucket example-documents --prefix contracts/nda.pdf

The list prefix can match more than one key; inspect exact keys and version IDs before restoring or deleting anything. Enabling versioning does not create historical versions for earlier overwrites. It can later be suspended, not returned to the original unversioned state. Existing non-null versions remain, while writes during suspension use the null-version behavior. Wait for initial versioning configuration propagation before relying on version IDs; AWS recommends fifteen minutes before issuing writes after first enablement. Versioning, enabling versioning.

A delete marker can hide an object without deleting its older versions. A version-specific DELETE can permanently remove a version. Retained versions consume storage according to their size and count; growth is not inherently exponential. Object Lock adds retention/legal-hold controls where needed, with governance/compliance semantics that require deliberate administration. Deleting versions, Object Lock.

Cross-Region Replication (CRR) copies eligible data to another Region; Same-Region Replication (SRR) uses the same Region. Both are asynchronous live-replication mechanisms requiring appropriate versioning and permissions. Existing objects are not automatically backfilled by a new live rule; plan S3 Batch Replication separately. Replication, what replicates.

The following is a complete replication-configuration object for eligible SSE-S3 objects, using a prepared replication role, two versioning-enabled buckets in different Regions, and RTC. It must be submitted as ReplicationConfiguration to PutBucketReplication on the source:

{
  "Role": "arn:aws:iam::123456789012:role/S3ReplicationRole",
  "Rules": [{
    "ID": "ReplicateDocuments",
    "Priority": 1,
    "Status": "Enabled",
    "Filter": {"Prefix": "documents/"},
    "DeleteMarkerReplication": {"Status": "Disabled"},
    "Destination": {
      "Bucket": "arn:aws:s3:::example-documents-west",
      "StorageClass": "STANDARD",
      "ReplicationTime": {"Status": "Enabled", "Time": {"Minutes": 15}},
      "Metrics": {"Status": "Enabled", "EventThreshold": {"Minutes": 15}}
    }
  }]
}

The role needs the documented trust and source/destination permissions. Cross-account destinations add bucket-policy requirements; KMS-encrypted replication needs additional opt-in/key settings and permissions. Including Filter requires Priority and DeleteMarkerReplication; RTC is paired with replication metrics. Configuration elements, replication permissions.

RTC is designed to replicate 99.99% of eligible new objects within fifteen minutes. Its SLA uses a 99.9% monthly fifteen-minute replication threshold per Region pair per account, with conditions and exclusions. Neither figure guarantees every object or an entire application's recovery deadline. RTC SLA. This example intentionally does not replicate delete markers. Even when enabled, delete-marker replication has exclusions, including lifecycle-created markers, and is outside RTC's fifteen-minute SLA. Version-specific deletions are not replicated. Recovery therefore needs a tested policy for retained destination data, not an assumption that both buckets are identical mirrors. RTC, delete markers.

6. Performance: measure the bottleneck

S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix. These are not strict ceilings or instructions to randomize every key. S3 scales gradually; watch 503 Slow Down, use SDK retries/backoff, and test sustained load. Multiple useful prefixes can support parallelism, but adding a hash is not a fixed multiplier that guarantees a rate. Sequential/date-based keys are valid. Performance guidance.

Multipart upload is worth considering around 100 MB and above. It supports parallel parts and retrying failed parts; durable resumption across process restarts needs the upload ID and part state. A high-level upload_file call does not automatically give your application a persistent restart protocol. Incomplete parts consume storage until completion/abort; use explicit cleanup and suitable lifecycle rules. Multipart overview.

from boto3.s3.transfer import TransferConfig

transfer_config = TransferConfig(
    multipart_threshold=25 * 1024 * 1024,
    multipart_chunksize=8 * 1024 * 1024,
    max_concurrency=10,
    use_threads=True,
    preferred_transfer_client="classic")

def upload_large_file(local_path, bucket, key):
    s3.upload_file(local_path, bucket, key, Config=transfer_config)

The threshold is 25 MiB, not 1024 × 25 bytes (25 KiB). The requested part size is 8 MiB and concurrency is bounded at ten for this classic transfer-manager example. The manager can increase part size to respect multipart limits for large files. Use a current SDK and validate its supported large-object path. boto3 transfer configuration.

Transfer Acceleration routes transfers through edge locations and the AWS network. It requires an eligible bucket and explicit client configuration; distance alone does not guarantee a fixed speed improvement. Benchmark the standard and accelerated paths and include the additional charge. For boto3, use Config(s3={"use_accelerate_endpoint": True}) instead of treating a bucket-specific endpoint string as a universal client setting. Acceleration.

S3 Select is no longer available to new customers. Existing customers can still use its supported single-object query feature, but a new learning exercise should not require that access. Athena and Redshift Spectrum are alternatives for querying suitably cataloged S3 data; they have their own formats, permissions, and costs. S3 Select availability, Athena.

7. Application: private static website delivery with CloudFront

Use a private S3 REST origin, OAC configured to sign requests, HTTPS for viewers, and an appropriate distribution cache behavior. Set a default root object such as index.html. Do not enable an S3 website endpoint and then expect OAC to secure it; website endpoints are custom origins and require a different access design. OAC setup.

For a distribution you own, a bucket policy can allow that distribution to read objects:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCloudFrontDistribution",
    "Effect": "Allow",
    "Principal": {"Service": "cloudfront.amazonaws.com"},
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::example-react-app/*",
    "Condition": {"StringEquals": {
      "AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/E123EXAMPLE"
    }}
  }]
}

Replace the identifiers and preserve any other required bucket policy statements. Enabling OAC in the distribution and allowing it in S3 are both necessary. For SSE-KMS objects, configure the key policy too. Origin access control does not authenticate website viewers; use viewer authorization if the content itself must be restricted.

For a build that places only content-hashed assets in build/assets/, upload assets first and the entry page last:

aws s3 sync build/assets/ s3://example-react-app/assets/ --cache-control "public,max-age=31536000,immutable"
aws s3 cp build/index.html s3://example-react-app/index.html --content-type text/html --cache-control "no-cache,max-age=0,must-revalidate"

The one-year cache lifetime is appropriate for content-hashed immutable names, not every arbitrary file. Keep old assets long enough for cached pages or open sessions that still reference them. Avoid immediately deleting old bundles during each deployment. Configure CloudFront's entry-page behavior with minimum TTL 0 so it can honor the revalidation headers; a positive minimum TTL can override origin no-cache/no-store. no-cache permits storage but requires revalidation—it does not mean “never stored.” Configure SPA deep-link handling deliberately; the default root object alone does not rewrite every nested route. CloudFront expiration, default root object.

8. Application: a data lake with intentional retrieval latency

Store analytics records under meaningful partition prefixes such as events/year=2026/month=09/day=15/. Use actual calendar dates, not a loop that creates January day 00 through day 99. Upload eligible objects with StorageClass="INTELLIGENT_TIERING"; a tiering configuration alone does not change every object's storage class.

Automatic Intelligent-Tiering tiers provide millisecond access: Frequent Access, Infrequent Access after thirty days without access, and Archive Instant Access after ninety days. Objects below 128 KiB remain in Frequent Access and are not monitored for automatic tiering. Optional Archive Access and Deep Archive Access need restoration and must be enabled deliberately. Tiering behavior.

For data that may wait hours, this configuration enables Archive Access after 180 days and Deep Archive Access after 365 days without access. It applies only to matching Intelligent-Tiering objects, and does not convert Standard objects:

def enable_optional_archives(bucket):
    config = {
        "Id": "AnalyticsArchives", "Status": "Enabled",
        "Filter": {"Prefix": "events/"},
        "Tierings": [{"Days": 180, "AccessTier": "ARCHIVE_ACCESS"},
                     {"Days": 365, "AccessTier": "DEEP_ARCHIVE_ACCESS"}]}
    s3.put_bucket_intelligent_tiering_configuration(
        Bucket=bucket, Id=config["Id"], IntelligentTieringConfiguration=config)

Ask the analytics owner whether a dashboard may wait for a restore. If not, leave optional archives disabled for its required data. Compact tiny files into suitably sized objects to reduce per-object overhead, but preserve partition pruning and useful parallelism. Track actual access and storage costs; a hundred tiny JSON objects do not demonstrate automatic archival savings. Tier configuration.

9. Application: temporary document sharing

A presigned URL includes authorization derived from the signer's credentials. It avoids giving the recipient the secret access key, but the URL itself is a bearer credential and can usually be reused until expiry. Its maximum validity is bounded by the requested time, underlying credential expiry, and applicable policy restrictions.

from botocore.config import Config

signer = boto3.client("s3", region_name="us-east-1",
                     config=Config(signature_version="s3v4"))

def document_upload_url(bucket, allocated_key):
    return signer.generate_presigned_url(
        "put_object", Params={"Bucket": bucket, "Key": allocated_key,
                              "ContentType": "application/pdf"},
        ExpiresIn=3600)

def document_download_url(bucket, key, version_id):
    return signer.generate_presigned_url(
        "get_object", Params={"Bucket": bucket, "Key": key,
                              "VersionId": version_id,
                              "ResponseContentDisposition": "attachment"},
        ExpiresIn=300)

Authenticate and authorize the user before allocating a scoped key or signing a download. The uploader must send the signed Content-Type: application/pdf header; that header does not prove the bytes are a safe PDF. Validate file contents before wider distribution. A PUT URL can overwrite its key or create another version, so use a unique allocated key and a completion/validation workflow. CORS may be needed for a browser, but CORS is not authorization. The downloader receives a specific named version rather than whichever bytes are current later. Presigned URLs, upload behavior, CORS.

10. Application: repeatable image processing

Use a versioning-enabled input bucket and a separate output bucket. The processor handles all records, decodes keys, reads the notified version, and produces a deterministic output key incorporating a transform version. Notifications can duplicate; they are not an exactly-once job system. Configure failure recovery and monitor it. S3 notifications, record format.

This Python handler requires Pillow in the deployment package and OUTPUT_BUCKET in the environment. It accepts bounded JPEG/PNG images, uses the first stored image for multi-image files (which can be a default poster in APNG), preserves aspect ratio within 200×200 pixels, and writes a fresh RGB PNG without copying image metadata. Transparent input is converted to RGB without a special background-compositing policy; add one if your product needs it.

import hashlib
import io
import json
import os
from urllib.parse import unquote_plus
from botocore.exceptions import ClientError
from PIL import Image, ImageOps

OUTPUT_BUCKET = os.environ["OUTPUT_BUCKET"]

def make_thumbnail(raw):
    with Image.open(io.BytesIO(raw)) as source:
        if source.format not in ("JPEG", "PNG"):
            raise ValueError("Unsupported image format")
        if source.width * source.height > 20_000_000:
            raise ValueError("Image pixel limit exceeded")
        source.seek(0)
        frame = ImageOps.exif_transpose(source).convert("RGB")
        frame.thumbnail((200, 200))
        clean = Image.new("RGB", frame.size)
        clean.paste(frame)
        output = io.BytesIO()
        clean.save(output, format="PNG")
        return output.getvalue()

def lambda_handler(event, context):
    if event.get("Event") == "s3:TestEvent":
        return {"processed": 0}
    count = 0
    for record in event["Records"]:
        source = record["s3"]
        bucket = source["bucket"]["name"]
        key = unquote_plus(source["object"]["key"])
        version = source["object"].get("versionId")
        if bucket == OUTPUT_BUCKET or not key.startswith("uploads/"):
            raise ValueError("Unexpected input location")
        if not version or version == "null":
            raise ValueError("Named source version required")
        response = s3.get_object(Bucket=bucket, Key=key, VersionId=version)
        stream = response["Body"]
        try:
            raw = stream.read(5 * 1024 * 1024 + 1)
        finally:
            stream.close()
        if len(raw) > 5 * 1024 * 1024:
            raise ValueError("Compressed image limit exceeded")
        identity = json.dumps([bucket, key, version, "thumbnail-v1"])
        output_key = "thumbnails/" + hashlib.sha256(identity.encode()).hexdigest() + ".png"
        try:
            s3.put_object(Bucket=OUTPUT_BUCKET, Key=output_key,
                          Body=make_thumbnail(raw), ContentType="image/png",
                          IfNoneMatch="*")
        except ClientError as error:
            if error.response["Error"]["Code"] != "PreconditionFailed":
                raise
        count += 1
    return {"processed": count}

For this example, only this processor writes the output prefix, existing outputs are valid for their identity, and outputs are not deleted during the retry window. A conditional-write PreconditionFailed then means the desired output already exists; a conflict such as ConditionalRequestConflict should propagate for retry. Change the transform version when output semantics change. Image processing and any later metadata write are separate effects: if you add DynamoDB, make its update idempotent and reconcile partial completion. Conditional writes, Pillow image operations, EXIF orientation.

Prepare an input-bucket notification filtered to uploads/ and grant that bucket permission to invoke the function. The bucket and function must be in the same Region. This deployment helper uses the existing clients and replaces the bucket's entire notification configuration; merge other notifications if the bucket is shared:

lambda_client = boto3.client("lambda", region_name="us-east-1")

def configure_image_trigger(input_bucket, account_id, function_arn):
    lambda_client.add_permission(
        FunctionName=function_arn, StatementId="AllowInputBucket",
        Action="lambda:InvokeFunction", Principal="s3.amazonaws.com",
        SourceArn="arn:aws:s3:::" + input_bucket, SourceAccount=account_id)
    s3.put_bucket_notification_configuration(
        Bucket=input_bucket, NotificationConfiguration={
            "LambdaFunctionConfigurations": [{
                "LambdaFunctionArn": function_arn,
                "Events": ["s3:ObjectCreated:*"],
                "Filter": {"Key": {"FilterRules": [
                    {"Name": "prefix", "Value": "uploads/"}]}}
            }]})

The execution role needs named-version input reads and output writes, plus applicable KMS permissions. Permission statement IDs must be managed on redeployment; blindly adding the same ID again can fail. Keep originals until the recovery/retention policy permits deletion. If processing tags drive lifecycle rules, apply them to the intended version and coordinate tag writers; replacing an entire tag set can remove unrelated tags. Notification permissions.

Practice: test an expired URL, a duplicate notification, a filename containing spaces, an overwritten source key, a lifecycle-archived input, and an output write whose response is lost. Explain the expected recovery at each boundary before measuring performance or savings.