Serverless & Lambda

Serverless & Lambda

A photo upload should create a thumbnail. An HTTP request should fetch a task. A daily schedule should summarize orders. AWS Lambda lets each event run a handler without your team provisioning the underlying servers. The engineering challenge is deciding what happens when an event is slow, duplicated, malformed, or retried after a partial failure.

This lesson covers handlers, execution environments, event sources, concurrency, cost, and four practical applications. It focuses on Lambda's default compute type. Lambda Managed Instances has a different execution and pricing model; do not apply every default-compute limit to it. Examples assume an AWS account, suitable IAM permissions, and resources in the intended Region. Resource names are examples, not resources created by the code here.

1. What serverless changes

AWS operates the default Lambda compute infrastructure. You still own application correctness, dependencies, permissions, data protection, networking choices, and monitoring. Automatic scaling is bounded by quotas, scaling rates, and downstream capacity. A database can be overwhelmed by a function that scales successfully.

Default on-demand Lambda generally charges for requests and allocated-memory duration. Duration is rounded up to 1 ms and includes applicable initialization and extension execution. Provisioned concurrency, extra ephemeral storage, and connected services can add charges. “No idle server bill” does not mean every Lambda configuration is free while idle. Lambda pricing, shared responsibility.

Stateless application design means correctness must not depend on reusing a particular execution environment. Module variables, connections, and /tmp files may survive a warm invocation, but they are neither durable nor shared across environments. Reuse clients and safe caches; store business state in a durable service.

For Python, save this as app.py and configure the handler as app.lambda_handler. Invoke it with a direct test event such as {"name":"Ada"}:

import json

def lambda_handler(event, context):
    name = event.get('name', 'World')
    return {
        'statusCode': 200,
        'headers': {'Content-Type': 'application/json'},
        'body': json.dumps({'message': f'Hello, {name}!'})
    }

The handler returns a Python dictionary; its body is a JSON string. The HTTP envelope is useful for a compatible proxy integration, but an API Gateway event does not normally put a submitted JSON field directly at event['name']. Parse its body according to the integration's payload format. A direct Lambda invocation also does not turn statusCode into an HTTP response to an end user by itself.

A CommonJS Node.js equivalent, saved as index.cjs with handler index.handler, is:

exports.handler = async (event) => ({
    statusCode: 200,
    headers: {'Content-Type': 'application/json'},
    body: JSON.stringify({message: `Hello, ${event.name ?? 'World'}!`})
});

Select a supported runtime and pin compatible dependencies in your deployment. Current Node.js runtimes include AWS SDK for JavaScript v3, not the old require('aws-sdk') v2 package. Go uses an OS-only/custom runtime rather than the retired managed go1.x runtime. Python handlers, Node.js runtimes, runtime support.

2. Cold starts, warm reuse, and concurrency

For default Lambda compute without SnapStart, a simplified lifecycle is:

New environment: provision environment → initialize runtime/extensions/code → invoke handler
Reused environment: reuse initialized environment → invoke handler
After an invocation: environment may be retained and frozen, reset, or shut down

A cold start can occur when traffic requires another environment, after replacement, or following certain failures. It is not restricted to the first-ever invocation. Neither cold nor warm invocations have a universal 100–1,000 ms or 1–10 ms total latency: your handler's work and dependencies dominate many workloads. Execution lifecycle.

Initialize reusable SDK clients outside the handler. Keep initialization lightweight, avoid per-user data in globals, and recover from stale connections. Smaller packages can help, but benchmark your actual runtime and workload. Compiled versus interpreted is not a reliable universal ranking of startup speed.

Provisioned concurrency must be allocated successfully, and requests must target its version/alias. It cannot be configured on $LATEST. Excess requests may use on-demand environments when limits permit. SnapStart is a separate startup optimization: initialization happens when publishing the version, and new environments restore its snapshot rather than rerunning that initialization from scratch. It is limited to supported runtimes/configurations and cannot be combined with provisioned concurrency on the same version. Reserved concurrency, provisioned concurrency, SnapStart.

For steady traffic, estimate average concurrent executions with:

requests_per_second = 1000
average_duration_seconds = 0.1
average_concurrency = requests_per_second * average_duration_seconds
assert average_concurrency == 100

This estimates an average, not safe peak capacity. Allow for bursts, long-tail latency, retries, source-specific limits, and downstream throughput. The published default Regional concurrency quota is 1,000, but new accounts can have reduced quotas; inspect your account. Default function scaling also has a rate limit. Concurrency and scaling.

3. Limits, timeouts, and cost arithmetic

For default Lambda compute, the configurable timeout is 1–900 seconds, with a 3-second default. Memory is normally 128–10,240 MB; AWS uses MB here for binary units. /tmp is configurable from 512 to 10,240 MB. New-account quota profiles can be lower. A durable workflow's lifetime is separate from one invocation's timeout. Lambda Managed Instances also has separately documented timeout capabilities. Function quotas, timeout configuration, Managed Instances timeout announcement.

The caller's deadline matters independently. For buffered API Gateway requests, HTTP APIs have a 30-second maximum integration timeout. REST APIs commonly use 29 seconds; Regional/private REST APIs can request increases. REST response streaming has different timeout behavior. A gateway timeout does not mean the Lambda function has necessarily stopped. For longer jobs, an API can acknowledge a job ID and let a worker store a result for later retrieval. HTTP API quotas, REST API quotas, response streaming.

Set timeouts from measured latency distributions and dependency deadlines. Give network calls bounded connect/read timeouts, leave time for cleanup, and test failure paths. Checking context.get_remaining_time_in_millis() can prevent starting work you cannot finish; it does not interrupt an already-blocked dependency.

For an illustrative US East (N. Virginia) first-tier x86 on-demand duration rate of $0.0000166667 per GB-second, before free allowances, discounts, request charges, or other services:

from decimal import Decimal
rate = Decimal('0.0000166667')

def duration_cost(memory_mb, billed_seconds):
    return Decimal(memory_mb) / Decimal(1024) * Decimal(billed_seconds) * rate

assert duration_cost(512, '5') == Decimal('0.00004166675')
assert duration_cost(1024, '2.5') == Decimal('0.00004166675')

Doubling memory and halving billed duration gives the same duration charge in that example. It does not establish that any particular function will run twice as fast.

The 128 MB option is cheapest for duration in this second set, while 1,024 MB is fastest. A latency objective may justify the higher cost. Rates vary with Region, architecture, tier, and compute mode. The pricing page lists the monthly 1 million requests and 400,000 GB-second free allowances with applicability conditions; connected-service bills and provisioned capacity still matter. Use the current price table and measured billed durations, not a promise of permanently free operation.

4. Event sources and retry ownership

An event source mapping is a polling integration, not a third InvocationType alongside RequestResponse and Event. SQS is a queue, not a stream. Its visibility timeout, redrive policy, and batch handling differ from stream checkpointing. Invocation methods, event source mappings, SQS integration.

For Lambda's asynchronous queue, function errors normally receive two additional attempts. Throttling/system errors have different retry behavior, normally up to six hours. Configure maximum event age, retry attempts, and a supported failure destination or DLQ. These settings do not replace SQS queue redrive controls. A handler returning {'statusCode': 500} normally succeeds as a Lambda invocation; raise an error when asynchronous processing should fail. Duplicate delivery remains possible even after success. Asynchronous error handling.

5. Example: an API Gateway task backend

Use a REST API Lambda proxy integration for the following event format (httpMethod, pathParameters, and string body). The sample table has one string partition key, taskId, and string task fields. Configure routes for GET /tasks/{taskId}, POST /tasks, and PUT /tasks/{taskId}. Authentication, authorization, throttling, and infrastructure creation are separate prerequisites; do not expose every user's records through a shared public endpoint.

import base64
import json
import os
import uuid
from datetime import datetime, timezone
import boto3
from botocore.exceptions import ClientError

table = boto3.resource('dynamodb').Table(os.environ['TASKS_TABLE'])

def reply(status, value):
    return {'statusCode': status,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps(value)}

def read_body(event):
    raw = event.get('body') or '{}'
    if event.get('isBase64Encoded'):
        raw = base64.b64decode(raw, validate=True).decode('utf-8')
    value = json.loads(raw)
    if not isinstance(value, dict):
        raise ValueError('Expected a JSON object')
    return value

def lambda_handler(event, context):
    method = event['httpMethod']
    task_id = (event.get('pathParameters') or {}).get('taskId')
    if method == 'GET' and task_id:
        item = table.get_item(Key={'taskId': task_id}).get('Item')
        return reply(200, item) if item else reply(404, {'error': 'Not found'})
    if method not in ('POST', 'PUT'):
        return reply(405, {'error': 'Unsupported route or method'})
    try:
        body = read_body(event)
    except (ValueError, UnicodeError):
        return reply(400, {'error': 'Invalid JSON body'})
    if method == 'POST':
        title = body.get('title')
        if not isinstance(title, str) or not 1 <= len(title.strip()) <= 200:
            return reply(400, {'error': 'title must contain 1–200 characters'})
        item = {'taskId': str(uuid.uuid4()), 'title': title.strip(),
                'status': 'pending',
                'createdAt': datetime.now(timezone.utc).isoformat()}
        table.put_item(Item=item, ConditionExpression='attribute_not_exists(taskId)')
        return reply(201, item)
    if not task_id or body.get('status') not in ('pending', 'done'):
        return reply(400, {'error': 'taskId and valid status are required'})
    try:
        table.update_item(
            Key={'taskId': task_id},
            UpdateExpression='SET #s = :s',
            ExpressionAttributeNames={'#s': 'status'},
            ExpressionAttributeValues={':s': body['status']},
            ConditionExpression='attribute_exists(taskId)')
    except ClientError as error:
        if error.response['Error']['Code'] == 'ConditionalCheckFailedException':
            return reply(404, {'error': 'Not found'})
        raise
    return reply(200, {'taskId': task_id, 'status': body['status']})

The update condition prevents an update from silently creating a missing task. A POST retried by a client can still create two tasks because each attempt generates a new ID. To make creation retry-safe, accept a scoped idempotency key and atomically associate it with the created task/result.

Listing is a separate design decision: one DynamoDB Scan or Query response is limited to a page of data. Follow LastEvaluatedKey with ExclusiveStartKey, or expose a validated cursor to clients. Do not call a single scan() “all tasks.” For a multi-user application, design a user-scoped key/query rather than scanning the entire table. DynamoDB numeric values returned by the Python resource interface use Decimal; define a JSON representation when you add numeric fields. Proxy event/response format, DynamoDB pagination, conditional writes.

6. Example: retry-safe thumbnail output

Configure an S3 ObjectCreated notification filtered to incoming/ on a versioned input bucket. Use a different output bucket with no trigger back into this function. Grant read access to input versions and write access to output objects; allow the input bucket to invoke Lambda. Package a compatible Pillow build with the function or a layer. A layer does not remove Lambda's combined uncompressed package limit.

Versioned input bucket /incoming/ → S3 notification → Lambda + Pillow
                                                        ↓
                                     Separate output bucket /thumbnails/

The sample reads the exact version named by each event, decodes the object key, and uses a deterministic output key. Repeated delivery does not produce a different logical thumbnail. It preserves aspect ratio inside a 200×200 box; it does not crop every image into a 200×200 square.

import hashlib
import json
import os
import warnings
from io import BytesIO
from urllib.parse import unquote_plus
import boto3
from botocore.exceptions import ClientError
from PIL import Image

s3 = boto3.client('s3')
output_bucket = os.environ['OUTPUT_BUCKET']
MAX_BYTES = 20 * 1024 * 1024
Image.MAX_IMAGE_PIXELS = 20_000_000
warnings.simplefilter('error', Image.DecompressionBombWarning)

def lambda_handler(event, context):
    if event.get('Event') == 's3:TestEvent':
        return {'processed': 0}
    processed = 0
    for record in event['Records']:
        info = record['s3']
        bucket = info['bucket']['name']
        key = unquote_plus(info['object']['key'])
        version = info['object'].get('versionId')
        if bucket == output_bucket:
            raise ValueError('Output bucket must differ from input')
        if not key.startswith('incoming/'):
            continue
        if not version or version == 'null':
            raise ValueError('This example requires versioned input objects')
        response = s3.get_object(Bucket=bucket, Key=key, VersionId=version)
        try:
            data = response['Body'].read(MAX_BYTES + 1)
        finally:
            response['Body'].close()
        if len(data) > MAX_BYTES:
            raise ValueError('Image exceeds the sample size limit')
        with Image.open(BytesIO(data)) as image:
            if image.width * image.height > 20_000_000:
                raise ValueError('Image exceeds the sample pixel limit')
            image.seek(0)
            image = image.convert('RGBA')
            image.thumbnail((200, 200), Image.Resampling.LANCZOS)
            buffer = BytesIO()
            image.save(buffer, format='PNG')
        identity = json.dumps([bucket, key, version], separators=(',', ':'))
        output_key = 'thumbnails/' + hashlib.sha256(identity.encode()).hexdigest() + '.png'
        try:
            s3.put_object(Bucket=output_bucket, Key=output_key,
                          Body=buffer.getvalue(), ContentType='image/png',
                          IfNoneMatch='*')
        except ClientError as error:
            if error.response['Error']['Code'] != 'PreconditionFailed':
                raise
        processed += 1
    return {'processed': processed}

IfNoneMatch='*' prevents overwriting an existing current output object; retain outputs for the replay window. A simultaneous-write conflict can still require a retry. The function deliberately fails on malformed/unsupported input rather than silently declaring it processed: configure a failure destination and operational handling for those events. It writes the image at Pillow index zero as PNG. For an APNG with a separate default/poster image, that is the poster rather than the first animation frame. PNG reencoding may retain metadata such as an ICC profile; explicitly remove unwanted metadata if the application requires it. Pillow APNG and PNG behavior. Benchmark memory and timeout against representative images instead of assuming a universal 512–1,024 MB requirement. S3 event encoding and versions, conditional writes, Pillow thumbnail behavior, package/layer limits.

7. Example: a daily order report

EventBridge Scheduler can invoke a report function during the 06:00 UTC minute. Cron schedules have six required fields: minute, hour, day-of-month, month, day-of-week, year. In cron(0 6 * * ? *), the ? leaves day-of-week unspecified because day-of-month already selects the days. Scheduler supports named time zones; this example explicitly selects UTC and disables its flexible window. It still has 60-second precision. Schedule syntax.

The Scheduler execution role must trust scheduler.amazonaws.com and permit invocation of the target function. The function must already exist. This is a real boto3 request shape:

import boto3

scheduler = boto3.client('scheduler', region_name='us-east-1')
scheduler.create_schedule(
    Name='daily-orders',
    ScheduleExpression='cron(0 6 * * ? *)',
    ScheduleExpressionTimezone='UTC',
    FlexibleTimeWindow={'Mode': 'OFF'},
    Target={
        'Arn': 'arn:aws:lambda:us-east-1:123456789012:function:DailyReport',
        'RoleArn': 'arn:aws:iam::123456789012:role/SchedulerInvokeDailyReport',
        'Input': '{"scheduledTime":"<aws.scheduler.scheduled-time>"}'
    })

Use the supplied scheduled time, not the retry's current clock, to determine the reporting day. The example below assumes an Orders table with DateIndex partition key orderDate (YYYY-MM-DD UTC), projection including amountCents, and nonnegative integer cent amounts. Deploy the three imported AWS SDK v3 packages with the Node.js function. Save it as index.mjs, handler index.handler. Configure environment variables for the table and verified SES sender/recipient; SES sandbox restrictions and sending permissions apply.

import {DynamoDBClient} from '@aws-sdk/client-dynamodb';
import {DynamoDBDocumentClient, QueryCommand} from '@aws-sdk/lib-dynamodb';
import {SESClient, SendEmailCommand} from '@aws-sdk/client-ses';

const db = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const ses = new SESClient({});

export const handler = async (event) => {
    const scheduled = new Date(event.scheduledTime);
    if (!Number.isFinite(scheduled.getTime())) throw new Error('Invalid scheduledTime');
    scheduled.setUTCDate(scheduled.getUTCDate() - 1);
    const day = scheduled.toISOString().slice(0, 10);
    let cursor;
    let count = 0;
    let cents = 0;
    do {
        const page = await db.send(new QueryCommand({
            TableName: process.env.ORDERS_TABLE,
            IndexName: 'DateIndex',
            KeyConditionExpression: 'orderDate = :day',
            ExpressionAttributeValues: {':day': day},
            ...(cursor ? {ExclusiveStartKey: cursor} : {})
        }));
        for (const order of page.Items ?? []) {
            if (!Number.isSafeInteger(order.amountCents) || order.amountCents < 0)
                throw new Error('Expected nonnegative integer cents');
            cents += order.amountCents;
            if (!Number.isSafeInteger(cents)) throw new Error('Report total too large');
            count += 1;
        }
        cursor = page.LastEvaluatedKey;
    } while (cursor && Object.keys(cursor).length > 0);
    const dollars = `${Math.floor(cents / 100)}.${String(cents % 100).padStart(2, '0')}`;
    await ses.send(new SendEmailCommand({
        Source: process.env.REPORT_FROM,
        Destination: {ToAddresses: [process.env.REPORT_TO]},
        Message: {
            Subject: {Data: `Daily report: ${day}`},
            Body: {Text: {Data: `Orders: ${count}\nRevenue: $${dollars}`}}
        }
    }));
    return {day, count, amountCents: cents};
};

Pagination prevents a silently truncated report, but a GSI query is eventually consistent and pagination is not a point-in-time snapshot. Define when a reporting day is closed and how late orders are reconciled. This example is appropriate for bounded datasets that fit the invocation timeout; large reports need partitioned work or another compute design. Email sending is not exactly once: if SES accepts the email but the response is lost, a retry can send another. Use a durable report/outbox workflow and an explicit duplicate-delivery policy for production. Scheduler context values, GSI reads, SDK v3 examples, SES sandbox.

8. Example: stream processing without double-counting

DynamoDB ADD score :points is atomic, but not idempotent. The uppercase action spelling is a convention here; DynamoDB also accepts lowercase action keywords. AWS update-expression examples. Starting at zero, applying ADD 10 twice produces 20. Atomic means concurrent updates are protected from lost-update races; it does not mean a duplicate event is ignored.

Suppose an append-only activity table emits stream records with NEW_IMAGE. Award points once per INSERT record; MODIFY and REMOVE do not award or reverse points in this model. Use a separate ledger table (eventKey string key) and score table (userId string key), in the same Region/account. A transaction atomically writes a deduplication marker and adds the points:

import hashlib
import json
import os
import boto3
from botocore.exceptions import ClientError

db = boto3.client('dynamodb')
ledger = os.environ['LEDGER_TABLE']
scores = os.environ['SCORES_TABLE']
POINTS = {'post_created': 10, 'comment_added': 5, 'like_given': 1}

def apply_record(record):
    if record['eventName'] != 'INSERT':
        return
    item = record['dynamodb']['NewImage']
    user_id = item['userId']['S']
    points = POINTS.get(item['action']['S'])
    if points is None:
        raise ValueError('Unknown activity action')
    identity = json.dumps([record['eventSourceARN'], record['eventID']])
    event_key = hashlib.sha256(identity.encode()).hexdigest()
    try:
        db.transact_write_items(TransactItems=[
            {'Put': {'TableName': ledger,
                     'Item': {'eventKey': {'S': event_key}},
                     'ConditionExpression': 'attribute_not_exists(eventKey)'}},
            {'Update': {'TableName': scores,
                        'Key': {'userId': {'S': user_id}},
                        'UpdateExpression': 'ADD score :points',
                        'ExpressionAttributeValues': {':points': {'N': str(points)}}}}
        ])
    except ClientError as error:
        if error.response['Error']['Code'] != 'TransactionCanceledException':
            raise
        prior = db.get_item(TableName=ledger,
                            Key={'eventKey': {'S': event_key}},
                            ConsistentRead=True)
        if not prior.get('Item'):
            raise
        # A committed marker proves this record's score update committed too.

def lambda_handler(event, context):
    for record in event['Records']:
        apply_record(record)
    return {'processed': len(event['Records'])}

Only this transaction should create ledger markers, and markers must be retained for the full intended replay window. If markers expire or are deleted, old replays can count again. Duplicate business actions written as different source items also require a business-level idempotency key; this example deduplicates delivery of the same stream record.

The sample raises on unhandled errors, so the batch can be retried. Committed earlier records are safe to replay while their markers remain. Configure bounded record age/retries and a failure destination; a poison record otherwise delays progress. For selective retry, enable ReportBatchItemFailures on the event source mapping and return the prescribed sequence-number response, using a tested batch processor. Batch size and the 6 MB invocation payload both constrain batches. Parallelization can preserve order per item without giving a global order across the table. DynamoDB stream behavior, transaction semantics, partial batch responses.

9. Deployment and operational checklist

Keep ZIP packages within the documented limits: 50 MB for direct upload, larger compressed archives via S3, and 250 MB uncompressed including layers/custom runtimes. Container images have a different 10 GB uncompressed limit. Layers share dependencies; they do not bypass the combined ZIP limit. Package compatible dependencies and test them on the target architecture/runtime. Deployment packages.

Scope IAM permissions to the actual operations and resources. A thumbnail reader of named versions needs the corresponding version-read permission; a transaction writer needs permissions for both participating tables. An execution role does not by itself authorize every trigger to invoke the function. VPC attachment gives network access to chosen VPC resources; it does not automatically provide internet access from a private subnet. Permissions, VPC internet access.

Monitor error rate, throttles, concurrency, duration percentiles, and source-specific backlog/age. Inspect billed duration and initialization data before tuning memory or provisioned concurrency. Test duplicate events, multi-record batches, dependency failures, malformed input, and timeout recovery. A local dictionary of processed IDs is an optimization at most: it cannot provide cross-environment or durable deduplication.

Practice: explain what happens if the thumbnail PUT succeeds and the function times out immediately afterward. Then explain the equivalent failure window between “increment score” and “record processed” in two separate DynamoDB writes. The first example has a deterministic conditional output write; the second requires the transaction above to close that gap.