Security, Governance & Observability

Security, Governance & Observability

An application starts returning errors after a permissions change. An administrator sees a red compliance finding and clicks “remediate.” Before either action helps, the team needs to know who changed what, which requests are failing, whether the finding is current, and what the repair will affect.

This lesson connects prevention, evidence and recovery through four applications: a scoped reader role, organization audit logging, controlled S3 remediation and cross-account monitoring. The examples assume prepared accounts and reviewed IAM roles. They are components to test in a learning environment before adopting them in production.

1. Assign each service a question it can answer

• IAM: may this principal perform this action on this resource under these conditions? Prefer federation and temporary workload roles; IAM groups contain IAM users, not roles.
• Organizations: which central restrictions apply to member-account principals? An account boundary helps separate ownership and recovery responsibilities; an OU groups accounts for policy management.
• CloudTrail: which supported events were recorded, with what identity, request and outcome? It is evidence of activity, not a record of every application operation.
• CloudWatch: what do the selected metrics, logs and traces reveal about system behavior? EventBridge handles event routing; it is a distinct service built on the former CloudWatch Events model.
• AWS Config: what supported resource configuration was recorded, and how did a rule evaluate it? Recording scope, frequency and rule triggers determine coverage and freshness.
• GuardDuty: which observed activity matches threat-detection logic? Inspector examines supported workloads for vulnerabilities, while Macie helps discover sensitive S3 data.
• Security Hub CSPM: what posture findings and control evaluations need attention? Security Hub combines findings and exposure information to help prioritize response. These are complementary services; neither certifies an organization as compliant.

Establish an owner and a response procedure for each signal. A dashboard with no responsible team is a display, not an operational control. Current service distinctions are documented in Security Hub and CSPM, CloudWatch and Config.

2. Understand permission and governance boundaries

An applicable explicit deny overrides an allow. The absence of a deny does not establish access: the request must also satisfy its applicable permission paths and limits. Identity policies, resource policies, session policies, permissions boundaries, SCPs and, where supported, resource control policies have different roles. A same-account resource policy grant can behave differently depending on whether it names a user, role or role session. Do not replace the full evaluation model with “one allow anywhere means success.” Policy evaluation.

request: principal + action + resource + context
                  |
       applicable explicit deny? ---- yes ---> deny
                  | no
       required grants and limits satisfied?
                  | yes                 | no
                allow                  deny

SCPs limit member-account principals, including member-account root users; they do not grant permissions. They do not constrain management-account principals or service-linked roles. An external principal receiving access through a member's resource policy is not thereby governed by that member's SCP. Retain the required allow policies along the organization hierarchy when using a deny-list design. Test changes in a limited OU with a reviewed recovery path. SCP scope and exceptions.

For example, this complete SCP denies one organization-management action to affected principals. It does not lock an account into the organization against every action the management account can perform:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyMemberLeavingOrganization",
    "Effect": "Deny",
    "Action": "organizations:LeaveOrganization",
    "Resource": "*"
  }]
}

aws:RequestedRegion identifies the endpoint Region being requested; it is not a complete data-residency control. Global services and cross-Region effects need deliberate treatment. A blanket deny outside two Regions can disrupt global services. Prefer a reviewed service-specific restriction or maintain appropriate exceptions; test actual API paths. RequestedRegion semantics.

Use the root identity only for tasks that actually require it, with appropriate MFA and recovery protection. Prefer federated workforce access through IAM Identity Center and short-lived workload credentials. Avoid turning a periodic key-rotation target into a reason to retain unnecessary long-lived keys. Enforce MFA at the relevant identity provider; not every federated session exposes aws:MultiFactorAuthPresent. IAM security practices.

3. Application: give a reader access to one development prefix

Suppose a role should list reports/ and read current object versions in one development bucket. The following is an identity policy attached to that role. Replace the illustrative bucket name with the exact bucket you own. Assume SSE-S3 objects; reading SSE-KMS objects can additionally require appropriate KMS authorization.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListReportKeys",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-development-reports",
      "Condition": {"StringLike": {"s3:prefix": ["reports/", "reports/*"]}}
    },
    {
      "Sid": "ReadReportObjects",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::example-development-reports/reports/*"
    }
  ]
}

Bucket listing uses the bucket ARN; object reads use object ARNs. The list condition limits the requested prefix, not the result of every other S3 operation. This policy contains no write/delete/version-read grant. It also does not cancel grants attached elsewhere: review the role, bucket policy and organization limits together. An independent production account and narrowly defined trust policy provide a clearer boundary than relying only on names beginning with dev-. S3 action/resource mapping, prefix restrictions.

Test the contract with the intended assumed role: list reports/, fetch a known permitted object, attempt to list the bucket root, read a different prefix and write an object. Expect the two intended operations to succeed and the others to fail under the complete policy set. Use IAM Access Analyzer's policy validation and relevant analyzer findings, then test in the target environment. A policy simulator does not reproduce every real-world policy context. Access Analyzer validation, simulator limits.

Practice: a bucket policy later grants this role s3:GetObject on private/*. Explain why the small identity policy alone is no longer evidence that the role is restricted to reports/.

4. Application: build organization audit coverage and protect the evidence

CloudTrail event history provides the past 90 days of management events in each Region. Trails and event data stores add configurable collection and retention. Data events, such as selected S3 object operations, must be explicitly covered; ordinary event history does not supply them. Event categories are not different kinds of trail. An organization trail and a multi-Region trail describe scope, while management/data events describe activity. Event history, logging data events.

Use a dedicated log-archive account and a reviewed organization trail. Before creation, confirm Organizations all-features mode, CloudTrail trusted access, authorized management/delegated administration, and the enabled Regions to cover. The example below creates the trail from the management account in us-east-1. Member accounts inherit organization-trail coverage; do not describe CloudTrail as “disabled” in those accounts. Organization prerequisites.

Prepare a new, uniquely named general-purpose S3 bucket in the log-archive account. Enable versioning, all four Block Public Access settings and a deliberately chosen retention policy. S3 encrypts new uploads with SSE-S3 by default; this example uses SSE-S3. A customer-managed KMS key requires a compatible key policy and trail configuration, not just a label in a diagram. Default encryption.

Generate this policy for the dedicated logging bucket. trail_arn is the exact ARN the intended management-account trail will have. Use validated account IDs and organization ID from your account inventory; the management-account prefix and organization prefix are both deliberate.

def trail_bucket_policy(bucket, trail_arn, organization_id, management_account_id):
    bucket_arn = f"arn:aws:s3:::{bucket}"
    return {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "TrailAclCheck",
                "Effect": "Allow",
                "Principal": {"Service": "cloudtrail.amazonaws.com"},
                "Action": "s3:GetBucketAcl",
                "Resource": bucket_arn,
                "Condition": {"StringEquals": {"aws:SourceArn": trail_arn}}
            },
            {
                "Sid": "TrailDelivery",
                "Effect": "Allow",
                "Principal": {"Service": "cloudtrail.amazonaws.com"},
                "Action": "s3:PutObject",
                "Resource": [
                    f"{bucket_arn}/AWSLogs/{management_account_id}/*",
                    f"{bucket_arn}/AWSLogs/{organization_id}/*"
                ],
                "Condition": {"StringEquals": {
                    "aws:SourceArn": trail_arn,
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }}
            }
        ]
    }

Install the generated JSON as the policy on this dedicated bucket using log-archive credentials. Replacing a policy on a shared bucket could remove unrelated permissions; use a reviewed merge there. CloudTrail delivery policy.

With that policy installed, use a CloudTrail client authenticated to the management account and the chosen home Region. This creates a new named trail, configures selected S3 data events and then starts collection:

def create_audit_trail(cloudtrail, name, log_bucket, sensitive_bucket):
    response = cloudtrail.create_trail(
        Name=name, S3BucketName=log_bucket,
        IsMultiRegionTrail=True, IsOrganizationTrail=True,
        IncludeGlobalServiceEvents=True, EnableLogFileValidation=True
    )
    trail_arn = response["TrailARN"]
    cloudtrail.put_event_selectors(
        TrailName=trail_arn,
        EventSelectors=[{
            "ReadWriteType": "All",
            "IncludeManagementEvents": True,
            "DataResources": [{
                "Type": "AWS::S3::Object",
                "Values": [f"arn:aws:s3:::{sensitive_bucket}/"]
            }]
        }]
    )
    cloudtrail.start_logging(Name=trail_arn)
    return cloudtrail.get_trail_status(Name=trail_arn)

put_event_selectors replaces the trail's selectors; merge deliberately when modifying an existing trail. The data-resource value is an ARN prefix ending in /, not an IAM wildcard expression. This configuration covers the selected bucket's supported object events; it does not select every data event in the organization. The bucket policy must match the returned trail ARN. A failed intermediate step can leave a created trail without the desired collection settings, so inspect its current configuration before retrying. Selectors.

Verify logging status and delivery errors, then generate a harmless management event and a read of a test object in the selected bucket. Confirm both records arrive under the expected account/Region paths. Delivery is asynchronous; IsLogging=true is not proof that every expected event has arrived.

Log-file validation creates signed digest information so a subsequent validation operation can detect changes or missing files. It does not prevent deletion or run verification automatically. Object Lock protects retained object versions: governance mode can be bypassed by appropriately authorized requests, while compliance-mode retention cannot be shortened even by root. Neither recovers events never collected. Choose and test retention from actual requirements; “seven years” is not a universal compliance rule. Integrity validation, Object Lock modes.

Practice: management events arrive but test-object reads do not. Inspect selector scope before concluding that the object was never accessed.

5. Application: connect a compliance finding to a controlled repair

Config records supported resources and evaluates rules; the rule itself does not perform the repair. A rule with Owner: AWS is a managed rule even when you choose its name or parameters. An aggregator provides a consolidated view of recorded data; it does not enable recording or remediation everywhere. Config offers conformance packs with mappings for multiple frameworks, including HIPAA Security. Such mappings help evaluate technical controls; they do not establish complete organizational compliance. Managed rules, HIPAA Security mapping.

This CloudFormation example connects an S3 public-read check to a real Automation document. It starts with manual remediation. Prerequisites: a running Config recorder covering S3 buckets in this account/Region, its delivery setup, and an existing Automation role trusted by Systems Manager. That role needs scoped S3 tag/public-access-block read/write permissions. The deploying operator also needs the relevant Config/SSM permissions and role-passing authorization.

Only buckets tagged SecurityRemediation=BlockPublicAccess may be changed. Control who can set that tag: it is an authorization input, not an informal label. Expected owner checking prevents acting on a same-named bucket in the wrong account. The template is for an initial deployment; pin the reviewed document version again when changing the runbook.

AWSTemplateFormatVersion: '2010-09-09'
Description: Manually approved Config remediation for opted-in S3 buckets
Parameters:
  AutomationRoleArn: {Type: String}
Resources:
  PublicReadRule:
    Type: AWS::Config::ConfigRule
    Properties:
      Source: {Owner: AWS, SourceIdentifier: S3_BUCKET_PUBLIC_READ_PROHIBITED}
      Scope: {ComplianceResourceTypes: ['AWS::S3::Bucket']}
  BlockPublicDocument:
    Type: AWS::SSM::Document
    Properties:
      DocumentType: Automation
      Content:
        schemaVersion: '0.3'
        description: Block public access only on explicitly opted-in buckets
        assumeRole: '{{ AutomationAssumeRole }}'
        parameters:
          AutomationAssumeRole: {type: String}
          BucketName: {type: String}
          ExpectedOwner: {type: String}
        mainSteps:
          - name: reconcile
            action: aws:executeScript
            timeoutSeconds: 120
            isEnd: true
            inputs:
              Runtime: python3.11
              Handler: remediate
              InputPayload:
                bucket: '{{ BucketName }}'
                owner: '{{ ExpectedOwner }}'
              Script: |
                import boto3
                from botocore.exceptions import ClientError
                def remediate(events, context):
                    s3 = boto3.client("s3")
                    args = {"Bucket": events["bucket"], "ExpectedBucketOwner": events["owner"]}
                    tags = s3.get_bucket_tagging(**args)["TagSet"]
                    approved = any(t["Key"] == "SecurityRemediation" and
                                   t["Value"] == "BlockPublicAccess" for t in tags)
                    if not approved:
                        raise ValueError("Bucket is not approved for this remediation")
                    desired = dict.fromkeys([
                        "BlockPublicAcls", "IgnorePublicAcls",
                        "BlockPublicPolicy", "RestrictPublicBuckets"
                    ], True)
                    try:
                        current = s3.get_public_access_block(**args)["PublicAccessBlockConfiguration"]
                    except ClientError as exc:
                        if exc.response["Error"]["Code"] != "NoSuchPublicAccessBlockConfiguration":
                            raise
                        current = {}
                    if all(current.get(k) is True for k in desired):
                        return {"result": "already-configured"}
                    s3.put_public_access_block(**args, PublicAccessBlockConfiguration=desired)
                    actual = s3.get_public_access_block(**args)["PublicAccessBlockConfiguration"]
                    if not all(actual.get(k) is True for k in desired):
                        raise RuntimeError("Public access block read-back differs")
                    return {"result": "configured"}
  Remediation:
    Type: AWS::Config::RemediationConfiguration
    Properties:
      ConfigRuleName: !Ref PublicReadRule
      ResourceType: AWS::S3::Bucket
      TargetType: SSM_DOCUMENT
      TargetId: !Ref BlockPublicDocument
      TargetVersion: '1'
      Automatic: false
      Parameters:
        AutomationAssumeRole:
          StaticValue: {Values: [!Ref AutomationRoleArn]}
        ExpectedOwner:
          StaticValue: {Values: [!Ref 'AWS::AccountId']}
        BucketName:
          ResourceValue: {Value: RESOURCE_ID}

Choose an evaluated noncompliant test bucket, approve the change and start remediation through Config. Check the execution status, fresh bucket settings and subsequent rule evaluation. A public website can break when public access is removed; exclusions and application ownership must be settled before enrollment. The action strengthens all four bucket-level flags without rewriting existing ACLs or policies. When a bucket policy is classified as public, RestrictPublicBuckets can also block explicit grants to other accounts. Review those private cross-account consumers before remediation. Block Public Access effects.

After testing failure paths and approved scope, automatic remediation can be configured separately with bounded attempts and monitored exceptions. Config may invoke remediation from stale compliance data, including after a resource was fixed elsewhere; the script therefore rereads settings and becomes a no-op if they already match. Tag/read/write operations are not one atomic transaction, so serialize conflicting administrative changes. RetryAttemptSeconds is a retry/exception window, not a promise to remediate within that many seconds. Sending an SNS notification is not a substitute for executing this repair. Remediation behavior, remediation configuration.

Encryption is a separate control. Default S3 encryption affects new writes; changing a default does not rewrite older objects. An S3 Bucket Key is a feature for appropriate SSE-KMS configurations, not an SSE-S3 option. Existing unencrypted EBS volumes likewise cannot be encrypted in place: plan a supported snapshot/copy/replacement and application cutover. S3 Bucket Keys, EBS encryption.

6. Application: monitor two source accounts from one monitoring account

CloudWatch metrics are identified by namespace, name and their full dimension set. A resource tag is not automatically a metric dimension. CPU metrics are available from EC2; guest memory/filesystem usage generally needs an agent or application instrumentation. Confirm the published metric name, dimensions, unit and sampling interval before creating an alarm. Metrics and dimensions, agent metrics.

OAM uses a sink in the monitoring account and links in source accounts, within a Region. The sink policy authorizes sharing; it does not grant every monitoring-account user permission to read all shared data. The following functions use separately authenticated OAM clients in us-east-1. Account lists are supplied from reviewed inventory, and the sources deliberately opt in to sharing their metrics. Logging/tracing requires additional resource types and access design.

import json

def create_metric_sink(oam_monitor, source_accounts):
    sink = oam_monitor.create_sink(
        Name="reviewed-metric-sink", Tags={"Environment": "Learning"}
    )["Arn"]
    policy = {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"AWS": [f"arn:aws:iam::{a}:root" for a in source_accounts]},
            "Action": ["oam:CreateLink", "oam:UpdateLink"],
            "Resource": "*",
            "Condition": {"ForAllValues:StringEquals": {
                "oam:ResourceTypes": ["AWS::CloudWatch::Metric"]
            }}
        }]
    }
    oam_monitor.put_sink_policy(SinkIdentifier=sink, Policy=json.dumps(policy))
    return sink

def link_metrics(oam_source, sink_arn):
    return oam_source.create_link(
        SinkIdentifier=sink_arn, LabelTemplate="$AccountName",
        ResourceTypes=["AWS::CloudWatch::Metric"]
    )

def put_cpu_dashboard(cloudwatch_monitor, account_a, instance_a, account_b, instance_b):
    body = {"widgets": [{
        "type": "metric", "width": 12, "height": 6,
        "properties": {
            "title": "Two reviewed EC2 instances", "region": "us-east-1",
            "view": "timeSeries", "stat": "Average", "period": 300,
            "metrics": [
                ["AWS/EC2", "CPUUtilization", "InstanceId", instance_a, {"accountId": account_a}],
                ["AWS/EC2", "CPUUtilization", "InstanceId", instance_b, {"accountId": account_b}]
            ]
        }
    }]}
    response = cloudwatch_monitor.put_dashboard(
        DashboardName="reviewed-cross-account-cpu", DashboardBody=json.dumps(body)
    )
    if response.get("DashboardValidationMessages"):
        raise RuntimeError("Inspect dashboard validation messages before relying on it")
    return response

Create the sink once, install its policy, link each source account and then publish the dashboard with real instance IDs. These are creation functions, not an idempotent organization-enrollment service. Inspect existing sinks/links before retries. Verify both sources publish fresh datapoints and the intended operator can view them; an empty graph can indicate missing metrics, incorrect dimensions or incomplete sharing. Repeat the setup deliberately in other Regions. Cross-account observability, CreateLink.

An alarm is a separate control. For a single instance, run this in its source account with an existing authorized SNS topic and confirmed subscription:

def cpu_alarm(cloudwatch, instance_id, topic_arn):
    return cloudwatch.put_metric_alarm(
        AlarmName=f"cpu-high-{instance_id}",
        Namespace="AWS/EC2", MetricName="CPUUtilization",
        Dimensions=[{"Name": "InstanceId", "Value": instance_id}],
        Statistic="Average", Period=300,
        EvaluationPeriods=2, DatapointsToAlarm=2,
        Threshold=80, ComparisonOperator="GreaterThanThreshold",
        TreatMissingData="missing", ActionsEnabled=True,
        AlarmActions=[topic_arn], InsufficientDataActions=[topic_arn]
    )

With complete datapoints, two consecutive five-minute averages above 80% satisfy this 2-of-2 alarm. They represent ten minutes of period data, not a guaranteed ten-minute delay after every incident. Here insufficient data is separately notified; another metric may need another missing-data policy. Test notification delivery and recovery. A zero value, no datapoint and no alarm action are three different situations. Alarm evaluation.

For application errors, this Logs Insights QL query retains a named aggregation timestamp before sorting:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errors by bin(5m) as interval
| sort interval desc
| limit 20

It returns up to 20 most recent populated five-minute bins within the selected query time range. It counts matching log events, not necessarily unique failures or an error percentage, and empty bins need not appear. The raw @timestamp is not the grouped output field to sort after aggregation. Logs Insights supports several query languages; the syntax above is QL, not SQL. Aggregation rules.

7. Keep operational controls usable

Systems Manager requires appropriately configured managed nodes, permissions and service connectivity. Run Command dispatch does not prove a patch succeeded; verify invocation results and application health. Use maintenance windows, rollout limits and rollback plans before restarting production services. State Manager applies associations on its configured schedule rather than guaranteeing software is present every instant. Systems Manager.

Parameter Store supports ordinary configuration and KMS-encrypted SecureString; decrypting requires authorization and returns a secret to the caller. Avoid shell-history exposure and secret logging. Secrets Manager supports configured rotation, including managed rotation for supported integrations; storing a secret does not automatically rotate every credential. Choose based on lifecycle, integration, size and current pricing rather than a universal free-versus-paid rule. Parameter types, rotation.

Set retention and access separately for operational logs and audit evidence. Enable the appropriate GuardDuty coverage in the Regions/accounts you operate and verify its configuration instead of repeatedly creating detectors blindly. Trusted Advisor currently organizes checks into six pillars, including operational excellence; available checks depend on support-plan entitlements and service eligibility. Its recommendations still require workload context. Trusted Advisor pillars.

For a final rehearsal, trace one access failure through identity/context, recorded API activity, resource configuration and application symptoms. Propose the smallest repair, identify its owner, describe how to validate it and state how you would recover if the repair failed. That sequence makes the service collection useful to the team operating it.