Secrets Manager & Parameter Store

Parameter Store Essentials: Hierarchies, Types, and Reading Values

Picture a shop app whose database host sits in config.py, whose feature flag lives in a .env file on one server, and whose payment API key was pasted into a CI variable two years ago. Changing any of them means hunting through three places, and nobody can say who can read what. AWS Systems Manager Parameter Store (a managed key-value store for configuration and secrets, part of the Systems Manager service, called SSM in the CLI) gives each value one named home with versioning and access control. This section covers how to name values, pick a parameter type, understand versions and labels, and read values back with the CLI.

Predict first

A teammate stored the database password as a SecureString (a parameter whose value is encrypted with a KMS key, AWS's managed encryption-key service). Then they ran:

aws ssm get-parameter --name /shop/prod/db/password

Task: predict what the Value field contains. Then write the corrected call.

aws ssm get-parameter --name /shop/prod/db/password --with-decryption \
  --query 'Parameter.Value' --output text

Hierarchical names as a design tool

A parameter name can be a path such as /shop/prod/db/password. Names are case-sensitive, and a hierarchy can go up to 15 levels deep. A consistent layout is /app/env/service/key, and the path becomes the unit you work with:

• Bulk reads: one call can fetch everything under /shop/prod/payments.
• IAM scoping: a policy can grant access to parameter/shop/prod/payments/* and nothing else (written out in the next section).
• Environment separation: dev and prod differ by one path segment, so code takes the environment as a variable and builds the prefix.

Worked layout for the shop app:

/shop
  /dev
    /db/host                 String
    /flags/new-checkout      String
  /prod
    /db/host                 String
    /db/password             SecureString
    /flags/new-checkout      String
    /web/allowed-origins     StringList
    /payments/api-key        SecureString

Put the most stable, widest-scoped segments first (app, then environment) and the most specific last. Whatever you would want to grant or read as a group has to share a prefix, because you cannot easily regroup later without renaming.

The three types

Rule of thumb: ask whether the value would hurt if it appeared in a screenshot, a log line or a support ticket. If yes, use SecureString. StringList is only a convention, because the stored value is a.com,b.com and your code splits it. Use it for a small, flat list and move anything structured elsewhere.

Standard vs advanced tier

A parameter policy attaches rules to a parameter, such as an expiration time or a notification before it expires. Standard is enough when values are small, you hold fewer than 10,000 parameters and you do not need policies, which fits most apps. Choose advanced for a value over 4 KB, a need for expiry policies, or sharing a parameter with another account. An advanced parameter is billed per month and also per API interaction (on 2026-10-05: $0.05 per advanced parameter per month and $0.05 per 10,000 interactions). A standard parameter can be changed to advanced but not back (to stop the charge you delete it and recreate it as standard). Limits and prices change, so check the current Parameter Store pricing page before budgeting.

Versions and labels

Every overwrite of a parameter creates a new version, a numbered snapshot of the value. A label is a movable pointer, a name you attach to one version.

# Overwrite creates version 2 (you must pass --overwrite)
aws ssm put-parameter --name /shop/prod/db/host --value db2.internal \
  --type String --overwrite

# Point the label prod-current at version 2
aws ssm label-parameter-version --name /shop/prod/db/host \
  --parameter-version 2 --labels prod-current

# Read a specific version or a label with a selector
aws ssm get-parameter --name /shop/prod/db/host:1
aws ssm get-parameter --name /shop/prod/db/host:prod-current

A label sits on only one version of a parameter at a time, so attaching prod-current to version 3 later moves it from version 2. Readers using name:prod-current then see the new value without changing their code. A plain read with no selector always returns the latest version, whichever label it has.

Reading patterns

There are three read commands, each fitting a different need:

# 1. One known value
aws ssm get-parameter --name /shop/prod/db/host

# 2. A known set (up to 10 names per call); missing names are reported
#    in an InvalidParameters list instead of failing the call
aws ssm get-parameters --names /shop/prod/db/host /shop/prod/flags/new-checkout

# 3. Everything under a path
aws ssm get-parameters-by-path --path /shop/prod --recursive --with-decryption

Without --recursive, get-parameters-by-path returns only parameters directly under the path, so it would find nothing at /shop/prod in our layout, since every parameter sits two levels deeper. The API returns results in pages (chunks), and a response that has more results includes a NextToken to request the next chunk. The AWS CLI v2 follows these tokens for you by default, but SDK code must loop until no token comes back.

Guided attempt

Task: load every /shop/prod value, readable, as a table of name, type and value.

aws ssm get-parameters-by-path --path /shop/prod --recursive --with-decryption \
  --query 'Parameters[].[Name,Type,Value]' --output table

Expected shape of the result for our layout, with five rows (values are made-up examples):

/shop/prod/db/host          String        db2.internal
/shop/prod/db/password      SecureString  s3cr3t-example
/shop/prod/flags/new-checkout String      true
/shop/prod/web/allowed-origins StringList a.com,b.com
/shop/prod/payments/api-key SecureString  sk-example

Notice three things. The dev parameters do not appear, because the path is the filter. SecureStrings appear in plaintext only because of --with-decryption. The StringList arrives as one comma-joined string. Now try this: how would you read only the payments values, and what would you drop if you wanted no plaintext secrets in your terminal history? (Hint: change the path, and think about the flag.)

Pitfall: a secret in a plain String

A String parameter stores its value unencrypted, and anyone with ssm:GetParameter on it sees plaintext with no KMS check at all. Here is a hierarchy review, listing each parameter with its type:

/shop/prod/db/host              String
/shop/prod/flags/new-checkout   String
/shop/prod/payments/api-key     String
/shop/prod/web/session-signing-key  String

Task: flag the entries that are wrong and say what you would do.

Who Can Read It: IAM and KMS Permissions for Parameters

A payments service starts in production and fails with AccessDeniedException. Its role clearly allows ssm:GetParameter, yet the secret will not come back. The cause is almost always the same: reading a SecureString passes through two separate authorization checks, and the role cleared only one. This section teaches you to write both checks with least privilege (granting only what a workload needs) and to read the error that tells you which one failed.

Two locks on one door

IAM (AWS Identity and Access Management) decides what a role may do. KMS (Key Management Service) holds the encryption keys that protect SecureString values. A get-parameter --with-decryption call on a SecureString clears two layers in order:

aws ssm get-parameter --name /shop/prod/payments/api-key --with-decryption
        ↓
Layer 1: SSM authorization
   ssm:GetParameter on the parameter ARN
   fails → AccessDenied naming ssm:GetParameter
        ↓
Layer 2: KMS authorization
   kms:Decrypt on the encrypting key (IAM policy and/or key policy)
   fails → AccessDenied naming kms:Decrypt
        ↓
Plaintext returned

The actions are ssm:GetParameter (one value), ssm:GetParameters (a known list) and ssm:GetParametersByPath (everything under a path). The resource is the parameter ARN (Amazon Resource Name, the unique identifier of an AWS object): arn:aws:ssm:<region>:<account>:parameter followed by the name. The name's leading slash is not doubled, so /shop/prod/db/password becomes parameter/shop/prod/db/password.

⚠️ A call without --with-decryption never asks KMS to decrypt, so it succeeds with ciphertext. A test that skips the flag can pass while the real application, which does decrypt, is denied.

Worked policy: the payments role

The tempting policy uses Resource: parameter/*, which lets the role that holds it read every parameter in that account and Region. Scope it to the service's own subtree instead:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadPaymentsParams",
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParameters", "ssm:GetParametersByPath"],
      "Resource": [
        "arn:aws:ssm:us-east-1:111122223333:parameter/shop/prod/payments",
        "arn:aws:ssm:us-east-1:111122223333:parameter/shop/prod/payments/*"
      ]
    },
    {
      "Sid": "DecryptWithProdKey",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/1111aaaa-22bb-33cc-44dd-5555eeee6666"
    }
  ]
}

The bare path ARN is there because GetParametersByPath is authorized against the path you pass in, not against each parameter it returns. So a by-path call on an allowed path returns everything beneath it, even a parameter that another statement denies for ssm:GetParameter. Here is what the role can and cannot read:

The trailing slash matters. Writing parameter/shop/prod/payments* would match payments-legacy too. This is the payoff of the path hierarchy from the first section: the path is your unit of permission.

Which key encrypts it?

A SecureString is encrypted with either the AWS managed key aws/ssm or a customer managed key (CMK, a key you create and whose policy you control). A key policy is the resource policy attached to a KMS key, and every key has one.

The aws/ssm key policy already trusts principals in your account who call through SSM, so a role often needs no kms:Decrypt in its own IAM policy. That is convenient, but it means you cannot say "team A may decrypt, team B may not" at the key. A CMK lets you do that. It is also required for letting another account read a SecureString: sharing goes through AWS RAM (Resource Access Manager), works only for advanced-tier parameters, and the key has to be shared separately. The other account reads a shared parameter by its full ARN with get-parameter or get-parameters (not by path), and ECS secrets injection does not support shared parameters. Each decrypt is a KMS request, so heavy reading adds cost and uses KMS request quota (check current pricing).

With a CMK, same-account access needs one of two things. Either the key policy names the role directly, or the key policy enables IAM (it contains a statement for the account root) and the role's IAM policy allows kms:Decrypt. Cross-account access needs both sides to say yes.

Guided diagnosis: ssm is allowed, read still fails

The role from above lost its second statement during a refactor. The SecureString is encrypted with the CMK, and the call returns:

An error occurred (AccessDeniedException) when calling the GetParameter operation:
User: arn:aws:sts::111122223333:assumed-role/payments-prod/i-0abc is not authorized
to perform: kms:Decrypt on the resource associated with this ciphertext ...

Read it in three steps. First, the operation is GetParameter, and it was allowed far enough to reach KMS. Second, the text after perform: is kms:Decrypt, not an ssm: action, so layer 2 failed. Third, the principal is an assumed role, so the fix goes on that role (or on the key policy). The exact wording varies, but the action name after perform: is always the clue. The minimal fix is the one missing statement:

{
  "Effect": "Allow",
  "Action": "kms:Decrypt",
  "Resource": "arn:aws:kms:us-east-1:111122223333:key/1111aaaa-22bb-33cc-44dd-5555eeee6666",
  "Condition": {"StringEquals": {"kms:ViaService": "ssm.us-east-1.amazonaws.com"}}
}

The kms:ViaService condition is optional but useful. It means the role can use this key only through SSM, not to decrypt arbitrary ciphertext directly. If the denial persists after adding the statement, the key policy does not enable IAM, so add the role to the key policy.

Two failing calls, one difference

Task: Both calls below fail, and neither error says access denied. The parameter exists as /shop/prod/payments/api-key in eu-west-1, and your CLI profile's default Region is us-east-1. Find the cause in each.

# Call A
aws ssm get-parameter --name shop/prod/payments/api-key --with-decryption --region eu-west-1
# Call B
aws ssm get-parameter --name /shop/prod/payments/api-key --with-decryption

Separating write from read

Runtime roles should only read. Give the CI/CD role (the pipeline identity that deploys changes) ssm:PutParameter on the paths it manages, plus kms:Encrypt on the key so it can write SecureStrings (kms:GenerateDataKey instead for advanced-tier parameters). Runtime roles get the Get* actions and kms:Decrypt, and nothing that writes. Then a compromised application cannot overwrite its own configuration.

To confirm who actually read what, query CloudTrail, the AWS service that logs API calls. The parameter name appears in the event's request details, and the secret value is not recorded:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetParameter \
  --max-results 5 --region us-east-1

The event shows the calling role and its session. The matching KMS Decrypt call is a separate event in the same history (look it up with AttributeValue=Decrypt). Event history covers the past 90 days of management events, so a trail is needed for longer retention.

Independent task: staging-only reader

Context: Account 111122223333, Region us-east-1. Role shop-staging-app must read everything under /shop/staging/ and nothing under /shop/prod/. Staging values use a CMK with ID 2222bbbb-33cc-44dd-55ee-6666ffff7777; prod uses a different CMK.

Write: (1) an IAM policy with the SSM and KMS statements, and (2) the statement to add to the staging key's policy so the role may decrypt. Then check yourself against the rubric.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParameters", "ssm:GetParametersByPath"],
      "Resource": [
        "arn:aws:ssm:us-east-1:111122223333:parameter/shop/staging",
        "arn:aws:ssm:us-east-1:111122223333:parameter/shop/staging/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/2222bbbb-33cc-44dd-55ee-6666ffff7777"
    }
  ]
}

{
  "Sid": "AllowStagingRoleDecrypt",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111122223333:role/shop-staging-app"},
  "Action": "kms:Decrypt",
  "Resource": "*"
}

Getting Values into Lambda and ECS: Start-up Injection vs Run-time Reads

Your payments service uses a third-party API key stored as a SecureString. Someone rotates the key at 10:00 and edits the parameter. When does the running service start using it: immediately, in five minutes, or never until someone redeploys? The answer depends on how the value was delivered. There are two models. Start-up injection means the platform fetches the value once when the workload launches and hands it over as an environment variable. A run-time read means your code calls the SDK (the AWS client library) while it runs. Each model fails in its own way.

ECS: Injection with the secrets Block

An ECS task definition is the JSON blueprint for a container group. Its secrets array holds entries with a name, which is the environment variable the container sees, and a valueFrom, which is the parameter ARN. In the same Region you can use just the parameter name.

{
  "family": "payments",
  "executionRoleArn": "arn:aws:iam::111122223333:role/payments-exec-role",
  "taskRoleArn": "arn:aws:iam::111122223333:role/payments-task-role",
  "containerDefinitions": [
    {
      "name": "payments",
      "image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/payments:1.4",
      "secrets": [
        {
          "name": "PAYMENTS_API_KEY",
          "valueFrom": "arn:aws:ssm:eu-west-1:111122223333:parameter/shop/prod/payments/api-key"
        }
      ]
    }
  ]
}

What happens at task launch:

ECS starts the task
  ↓ assumes the EXECUTION role
Calls ssm:GetParameters for each valueFrom (decrypting with KMS)
  ↓
Puts plaintext into env var PAYMENTS_API_KEY
  ↓
Container starts; app reads os.environ["PAYMENTS_API_KEY"]

The fetch happens once, before your code runs. The task definition holds only the pointer, never the value.

Execution Role vs Task Role

The task execution role is used by ECS itself to pull the image and resolve secrets before your container exists. The task role is the identity your application code gets for SDK calls while running. Injection needs the permissions on the execution role: ssm:GetParameters on the parameter ARN, plus kms:Decrypt if a customer managed key encrypts it (see "Who Can Read It" for writing both layers).

Try it. A new task stops immediately with: ResourceInitializationError: unable to pull secrets or registry auth ... AccessDeniedException: payments-exec-role is not authorized to perform: ssm:GetParameters. The task role already allows ssm:GetParameter on /shop/prod/payments/*. What do you change?

When Does a Changed Value Arrive?

Because injection happens at launch, a running ECS task never sees later edits. Predict the outcome of each change for tasks already running:

1. You overwrite /shop/prod/payments/api-key (this creates a new parameter version).
2. You do (1), then run aws ecs update-service --force-new-deployment.
3. You register task definition revision 8 pointing at /shop/prod/payments/api-key-v2, but do not update the service.

Lambda: Read It Yourself in Init Code

Lambda has no valueFrom equivalent. Your code fetches the value, and the right place is init code, the module-level code that runs once when a new execution environment (the sandbox that runs your function) starts. Handler invocations reuse the same environment while it stays warm, so a module-level cache means one API call per environment instead of one per invocation. Add a TTL (time to live, how long a cached value counts as fresh) so rotations eventually arrive.

import time
import boto3
from botocore.config import Config

# Client built once per execution environment; "standard" retry mode
# retries throttling errors with exponential backoff and jitter.
ssm = boto3.client("ssm", config=Config(retries={"mode": "standard", "max_attempts": 8}))
TTL_SECONDS = 300
_cache = {}  # name -> (value, fetched_at)

def get_param(name):
    entry = _cache.get(name)
    now = time.time()
    if entry and now - entry[1] < TTL_SECONDS:
        return entry[0]
    resp = ssm.get_parameter(Name=name, WithDecryption=True)
    _cache[name] = (resp["Parameter"]["Value"], now)
    return resp["Parameter"]["Value"]

def handler(event, context):
    api_key = get_param("/shop/prod/payments/api-key")
    ...

Trace one environment: cold start at t=0 runs init with an empty cache. The first invocation misses and makes API call 1. Invocations at t=10s and t=299s hit the cache. The invocation at t=301s finds the entry older than 300s, so it makes call 2. A rotation at t=50s therefore reaches this environment at t=301s, so worst-case staleness equals the TTL. Different environments have different fetch times, so they switch over at different moments.

The Parameters and Secrets Lambda Extension

An AWS-provided Lambda extension (a helper process added as a layer that runs beside your function) can do the caching for you. Your code makes a local HTTP request and the extension serves it from a cache, calling Parameter Store only on a miss. Its default TTL is 300 seconds, which is also the maximum. The environment variable SSM_PARAMETER_STORE_TTL (0 to 300 seconds, where 0 bypasses the cache) changes it, and SECRETS_MANAGER_TTL does the same for secrets.

import json, os, urllib.parse, urllib.request

def get_param(name):
    qs = urllib.parse.urlencode({"name": name, "withDecryption": "true"})
    req = urllib.request.Request(
        "http://localhost:2773/systemsmanager/parameters/get?" + qs,
        headers={"X-Aws-Parameters-Secrets-Token": os.environ["AWS_SESSION_TOKEN"]},
    )
    with urllib.request.urlopen(req) as r:
        return json.load(r)["Parameter"]["Value"]

The staleness rule is unchanged: the cache is per execution environment, so the worst case is still the TTL. What changes is that you write no cache code. The function role still needs the SSM and KMS permissions, because the extension calls AWS with your function's credentials. One caution about the snippet: Lambda does not set AWS_SESSION_TOKEN in every initialization mode (a SnapStart function is one example), and AWS recommends reading the token from the SDK's credential provider inside the handler in that case.

Throttling and Cost

Calling get-parameter on every invocation looks harmless in testing. At 100 invocations per second that is 100 API calls per second, and SSM throttles with ThrottlingException once you exceed its per-account, per-Region limits (by default 40 requests per second, shared by GetParameter, GetParameters and GetParametersByPath; a paid higher-throughput setting raises it). Cached with a 300-second TTL, 100 concurrent environments make about 100/300 ≈ 0.33 calls per second. Three levers combine:

• Cache with a TTL, as above.
• Batch: get_parameters_by_path(Path="/shop/prod/payments/", Recursive=True, WithDecryption=True) loads a whole branch with one request per page of at most 10 parameters, instead of one request per key.
• Retry with backoff: SDK retry settings, as in the snippet, absorb occasional throttling.

Fewer calls also mean fewer KMS decrypt requests, which are billed.

⚠️ Pitfall: Decrypting at Deploy Time

Resolving a value during deployment and writing it into the function's environment variables copies the plaintext into the function configuration. A CloudFormation {{resolve:ssm:...}} reference or a template parameter of type AWS::SSM::Parameter::Value<String> does this for String and StringList parameters only (CloudFormation does not resolve a SecureString that way). A {{resolve:secretsmanager:...}} reference, or a deploy script that reads the secret and sets the variable, does it for a real secret. Anyone who can view the function configuration in the console or API sees the secret. A template output that echoes it is visible to anyone who can view the stack. The value is also frozen until the next deploy. Pass the parameter name in the environment variable instead (API_KEY_PARAM=/shop/prod/payments/api-key) and fetch the value at run time.

Guided Task: The Five-Minute Rotation

A service must use a rotated API key within 5 minutes. Decide step by step:

1. Injection or run-time read? Injection delivers new values only when tasks are replaced, so you would need an automated forced deployment after every rotation. A run-time read with a TTL gives you a bound you control.
2. TTL? Worst-case staleness equals the TTL, so it must be below 300 seconds, with margin for retries. Choosing 60 seconds costs about one call per minute per environment.
3. Justify. The old key must stay valid through the overlap window, or requests fail during the switch.

Independent Variation

Decide for two services. (a) A Lambda handling 500 invocations per second across about 100 environments must see a new feature flag within 30 seconds. (b) An ECS worker uses a database host that changes a few times a year, always through a pipeline deployment. Choose a delivery model and a TTL if any, and give one reason each.

Secrets Manager: Rotation, Staging Labels, Resource Policies, and Cost

Your database password must change every 30 days. With Parameter Store you would write a script that generates a password, changes it in the database, overwrites the parameter, and hopes every app picks it up in the right order. AWS Secrets Manager is the service that runs that choreography for you. This section traces what it does, how running clients behave while it happens, and what it costs.

Core model: secrets, versions, and staging labels

A secret holds a text value or, more commonly, a JSON document such as {"username":"app","password":"...","host":"db.internal","port":5432}. Every change creates a new version. A version has no number. You address it by its version ID (a UUID) or, far more often, by staging labels, which are movable tags attached to versions:

• AWSCURRENT: the version clients should use now.
• AWSPREVIOUS: the version that was current before the last change.
• AWSPENDING: a candidate version being prepared during rotation.

A version can carry several labels. A version with no label is treated as deprecated and can be deleted by the service.

# Default read: returns the AWSCURRENT version
aws secretsmanager get-secret-value --secret-id prod/shop/db

# Read a specific stage
aws secretsmanager get-secret-value --secret-id prod/shop/db --version-stage AWSPREVIOUS

The JSON arrives as a string in the SecretString field, so your code must parse it. Unlike Parameter Store, there is no --with-decryption flag, because decryption is always on. The caller needs secretsmanager:GetSecretValue, plus kms:Decrypt on the key when a customer managed key encrypts the secret instead of the AWS managed key aws/secretsmanager.

Tracing a rotation

Rotation means automatically replacing a secret's value on a schedule. AWS invokes a rotation Lambda function (a function holding the logic for your secret type; AWS supplies ready-made ones for RDS databases) four times, once per step. Assume the secret starts with version A labeled AWSCURRENT:

Start            A = AWSCURRENT
  ↓
createSecret     generate new password, store as version B = AWSPENDING
  ↓
setSecret        change the password in the database to B's value
  ↓
testSecret       log in to the database using the AWSPENDING value
  ↓
finishSecret     B gets AWSCURRENT, A gets AWSPREVIOUS, AWSPENDING removed

The labels only move in the last step, so a failure in steps one to three leaves AWSCURRENT on A. That is why AWSPREVIOUS is useful afterward: the old value stays retrievable, so you can inspect it or move the label back.

⚠️ Simplified picture: whether the old password still works in the database depends on the rotation strategy. In single-user rotation, setSecret changes the password of the same database user, so version A stops working at that moment. In alternating-users rotation, the Lambda switches between two database users, so the previous credential stays valid for a full cycle. The second strategy is the safer one for clients that cache.

Check your trace. testSecret fails with a login error. Which version holds AWSCURRENT, and which clients are affected?

Client behavior: retry, then refresh

A client that reads the secret once and caches it for an hour keeps using version A after the label moves. With single-user rotation it gets authentication errors for up to an hour. A long TTL (time to live, how long a cached value is trusted) is the wrong tool. Instead, treat an authentication failure as the signal that the secret changed: refresh once, retry once, then give up.

import json
import boto3
import psycopg2

sm = boto3.client("secretsmanager")
_creds = None  # module-level cache, reused across calls

def _load():
    global _creds
    resp = sm.get_secret_value(SecretId="prod/shop/db")  # AWSCURRENT
    _creds = json.loads(resp["SecretString"])
    return _creds

def _open(c):
    return psycopg2.connect(host=c["host"], port=c["port"],
                            user=c["username"], password=c["password"],
                            dbname="shop", connect_timeout=5)

def connect():
    try:
        return _open(_creds or _load())
    except psycopg2.OperationalError:
        # Login failed: the secret may have rotated. Refresh once, retry once.
        return _open(_load())

The retry is bounded to one refresh, so a real database outage does not turn into a loop of Secrets Manager calls. OperationalError also covers network failures, so the extra read is wasted there but harmless. Existing open connections usually survive a password change; only new logins fail.

Resource policies and cross-account access

A resource policy is a permission document attached to the secret itself, naming who may use it. For cross-account access, both sides must allow it: the secret's policy and the caller's IAM policy. Take a shared-services account 111111111111 that owns prod/shop/db, and a workload account 222222222222 whose ECS task execution role needs it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::222222222222:role/shop-task-execution-role"},
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "*"
  }]
}

Here "Resource": "*" means "this secret". Three more pieces are needed:

1. The execution role in account 222222222222 needs an IAM statement allowing secretsmanager:GetSecretValue on the secret's ARN.
2. The secret must be encrypted with a customer managed KMS key, because the AWS managed key aws/secretsmanager has a fixed key policy that cannot grant another account access.
3. That key's policy must allow the workload role kms:Decrypt, and the role's IAM policy must allow it too.

Cross-account callers must use the secret's full ARN, not its short name. Missing the key grant produces AccessDeniedException mentioning KMS even though the secret policy is perfect.

ECS integration

In an ECS task definition, valueFrom accepts a Secrets Manager ARN, and the task's execution role fetches it when the task starts (see "Getting Values into Lambda and ECS" for the execution-role vs task-role split). To extract one JSON key, append :jsonkey:: to the ARN. The general form is secret-arn:json-key:version-stage:version-id, and leaving stage and ID empty means AWSCURRENT.

Guided attempt. Write the secrets block that injects DB_USERNAME and DB_PASSWORD from the username and password keys of arn:aws:secretsmanager:us-east-1:111111111111:secret:prod/shop/db-AbCdEf.

"secrets": [
  {
    "name": "DB_USERNAME",
    "valueFrom": "arn:aws:secretsmanager:us-east-1:111111111111:secret:prod/shop/db-AbCdEf:username::"
  },
  {
    "name": "DB_PASSWORD",
    "valueFrom": "arn:aws:secretsmanager:us-east-1:111111111111:secret:prod/shop/db-AbCdEf:password::"
  }
]

⚠️ Rotation pitfall. That start-time injection is exactly the trap. A task launched on day 1 holds the day-1 password in its environment forever. If rotation is enabled with single-user strategy, the next rotation window breaks every new connection from that task, and nothing restarts it. Before enabling rotation, list every consumer and confirm each either re-reads the secret on authentication failure, as in the pattern above, or is redeployed after rotation, or the secret uses alternating users.

Cost analysis

Secrets Manager charges a monthly fee per secret plus a fee per 10,000 API calls. Parameter Store standard parameters carry no per-parameter charge. The figures below are illustrative (about $0.40 per secret per month and $0.05 per 10,000 calls), so confirm them on the current pricing page before deciding. KMS charges for a customer managed key are separate.

• 300 secrets, light reads: 300 × $0.40 = $120/month in storage fees. The same 300 as standard parameters would cost nothing per parameter, which is why bulk configuration does not belong here.
• 20 secrets, 100 containers, read once at start-up: storage is 20 × $0.40 = $8. Reads are 2,000 calls, about $0.01. Total is roughly $8.
• Same, but every container polls every secret every 5 minutes: 288 polls/day × 30 days = 8,640 calls per container per secret. Multiplied by 2,000 container-secret pairs, that is 17,280,000 calls, or 1,728 blocks of 10,000 × $0.05 = $86.40. Total is roughly $94.

The lesson from the third case is that call volume, not secret count, can dominate the bill. Cache in-process and refresh on failure instead of polling.

Choosing, Combining, and Diagnosing: A Decision and Failure Clinic

A teammate asks, "Should this go in Parameter Store or Secrets Manager?" and the usual answers are slogans: "secrets go in Secrets Manager" or "Parameter Store is free." Both slogans break on real workloads. This section replaces them with a short list of cues, each with the condition that triggers it, and then trains you to diagnose the four failures you will meet once the values are deployed.

Decision cues, with their conditions

A cue is a fact about the workload that pushes the choice one way, but only when its condition holds. Quoted sizes and prices are illustrative, so verify current limits and pricing before you commit.

No single cue decides alone. One rotating password beats a hundred free config values, because rotation is the one capability you cannot cheaply rebuild yourself.

Try it: score three cases

For each case, name the cues that fire and pick a store.

• A: A log level and 40 feature flags read by 30 services, edited weekly, no secrecy.
• B: An RDS password that must rotate every 30 days and is read by ECS services in two accounts.
• C: A 12 KB signing-certificate bundle, replaced by hand once a year, used in one account.

The hybrid pattern

Most real systems use both stores. Put flags, hostnames and tuning values in Parameter Store, and put database and third-party credentials in Secrets Manager. Parameter Store can also act as a doorway to a secret. Reading a name that starts with /aws/reference/secretsmanager/ returns the Secrets Manager value:

# Reads secret "shop/prod/payments/db" through the SSM API
aws ssm get-parameter \
  --name /aws/reference/secretsmanager/shop/prod/payments/db \
  --with-decryption --region eu-west-1

This lets code that only knows the SSM API reach both stores. The limits matter, though:

• The path is a read-only lookup, so you cannot put-parameter into it.
• It is meant for get-parameter and get-parameters, so do not expect it to behave like a normal hierarchy for get-parameters-by-path or listing.
• It does not remove the need for Secrets Manager permissions. The caller still needs secretsmanager:GetSecretValue (plus kms:Decrypt when a customer managed key encrypts the secret), and the secret is still billed as a secret.

The failure clinic

Every failure starts with the same move: read the exact error text, because it names the action or resource that was denied or missing.

Worked diagnosis: Lambda fine in dev, failing in prod

The function reads the SecureString /shop/${STAGE}/db/password. The failures appear one after another as each layer is fixed, so check the layers in order.

import os
import boto3

# Bug: Region is hardcoded, so every call goes to us-east-1
ssm = boto3.client("ssm", region_name="us-east-1")
STAGE = os.environ["STAGE"]

def handler(event, context):
    name = f"/shop/{STAGE}/db/password"
    return ssm.get_parameter(Name=name, WithDecryption=True)["Parameter"]["Value"]

1. Region. Dev lives in us-east-1, so it works. Prod runs in eu-west-1 and gets ParameterNotFound. Fix: boto3.client("ssm"), which uses the function's own Region.
2. Resource ARN. Now the error is AccessDeniedException on ssm:GetParameter. Compare the denied ARN with the role policy. The prod role's statement was written for arn:aws:ssm:us-east-1:...:parameter/shop/prod/*, so it does not match the eu-west-1 ARN. Fix the Region in the resource ARN.
3. Role policy for KMS. Prod encrypts with its own customer managed key, so the next error names kms:Decrypt. Add it for the prod key's ARN.
4. Key policy. If it still fails, the key policy does not trust this role, and the IAM policy alone is not enough. Add the role to the prod key's policy, or delegate to IAM through the account root statement.

Each fix removed one denial and exposed the next layer, so one error does not mean one problem.

Harder variation: cross-account rotating secret

An ECS service in a workload account must read a rotating database secret owned by a shared-services account. The secret is encrypted with a customer managed key (CMK) and injected with valueFrom.

Task: decide (1) the secret policy, (2) the key arrangement, (3) the execution role permission, (4) the refresh behavior.

Independent transfer task

Setup: an app has three services across dev and prod. The web service runs on ECS and needs a database host, a feature-flag set and a session-signing key. The payments service runs on ECS and needs a rotating Postgres password and a payment-provider API key. The worker runs on Lambda and needs a queue URL and the same Postgres credentials as payments, using its own database user.

Design the layout:

• names and types
• which store each value uses
• IAM and KMS scoping
• the delivery method for each service
• a monthly cost estimate, using illustrative prices of about $0.40 per secret per month, $0.05 per 10,000 Secrets Manager API calls, $1 per CMK per month, and free standard parameters (verify current pricing)

Then check yourself against the checklist below.

Checklist:

• Each name is unique per environment and service.
• No credential sits in a plain String parameter.
• Every role names its prefix or secret ARNs, never *.
• Dev and prod use different keys.
• Each rotating secret has a consumer that refreshes it.
• Every store choice cites a cue.

Outcome check

• Can you name and read the right parameter, with the correct Region, leading slash and --with-decryption?
• Can you write a two-layer permission (SSM or Secrets Manager action plus KMS decrypt)?
• Can you predict when a change is picked up: at task start, at cache expiry, or at the next deployment?
• Can you trace a rotation and say which consumers see it?
• Can you choose a store and defend it with named cues?

If any answer is shaky, redo the matching exercise above without opening the hidden answers.