AWS CloudTrail - Dev Weekends

Module Overview

Estimated Time: 2-3 hours | Difficulty: Intermediate | Prerequisites: IAM, S3, basic logging concepts

AWS CloudTrail is a service that enables governance, compliance, and operational and risk auditing of your AWS account. It records API calls made to AWS services and delivers log files to an S3 bucket. Think of CloudTrail as the security camera system for your AWS account — it records who did what, when, and from where. When an S3 bucket gets deleted at 2 AM and nobody claims responsibility, CloudTrail is how you find out which access key made the call and from which IP address. It is non-negotiable for any production AWS environment. What You’ll Learn:

CloudTrail event types and structure
Trail configuration and multi-region setup
CloudTrail Lake for advanced querying
CloudTrail Insights for anomaly detection
Security investigation workflows

How CloudTrail Works

┌─────────────────────────────────────────────────────────────────────────┐
│                      CloudTrail Architecture                             │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   API Activity                   CloudTrail               Destinations  │
│   ────────────                   ──────────               ────────────  │
│                                                                          │
│   ┌──────────────┐          ┌─────────────────┐     ┌────────────────┐  │
│   │  AWS Console │─────────▶│                 │────▶│ S3 Bucket      │  │
│   └──────────────┘          │                 │     │ (encrypted)    │  │
│                              │   CloudTrail    │     └────────────────┘  │
│   ┌──────────────┐          │     Trail       │                         │
│   │   AWS CLI    │─────────▶│                 │     ┌────────────────┐  │
│   └──────────────┘          │  ─ Management   │────▶│ CloudWatch Logs│  │
│                              │    Events       │     └────────────────┘  │
│   ┌──────────────┐          │  ─ Data Events  │                         │
│   │   AWS SDKs   │─────────▶│  ─ Insights     │     ┌────────────────┐  │
│   └──────────────┘          │    Events       │────▶│ CloudTrail Lake│  │
│                              │                 │     └────────────────┘  │
│   ┌──────────────┐          └─────────────────┘                         │
│   │ AWS Services │─────────▶                        ┌────────────────┐  │
│   └──────────────┘                                  │ EventBridge    │  │
│                                                      └────────────────┘  │
│                                                                          │
│   Event Flow:                                                            │
│   ───────────                                                            │
│   1. User/Service makes API call                                        │
│   2. CloudTrail captures the event                                      │
│   3. Event processed and delivered (within ~15 minutes)                 │
│   4. Stored in S3, queryable in Lake                                    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Event Types

Management Events

Operations on AWS resources (control plane):

# Management events = "control plane" operations.
# Think of these as changes to the infrastructure itself, not the data inside it.
# Analogy: management events are like building security footage (who entered 
# which room), while data events are like watching what they did at their desk.
Examples:
  - CreateBucket, DeleteBucket
  - CreateUser, DeleteRole
  - StartInstances, TerminateInstances
  - CreateVpc, AttachInternetGateway
  - PutBucketPolicy, PutRolePolicy

# Recorded by default in all trails -- no extra configuration or cost needed.
# This is one of the first things auditors will check in a compliance review.
Recorded by default in all trails

Data Events

Operations on data within resources:

Examples:
  S3 Object-level:
    - GetObject
    - PutObject
    - DeleteObject
  
  Lambda Invocations:
    - Invoke function calls
  
  DynamoDB:
    - GetItem, PutItem, DeleteItem
  
  EBS Direct APIs:
    - Snapshot operations

# NOT recorded by default -- you must explicitly enable data events.
# Why not default? Volume. A single busy S3 bucket can generate millions
# of GetObject events per day. Cost tip: enable data events only for
# sensitive buckets (PII, financial data), not for log or asset buckets.
# Pricing: roughly $0.10 per 100,000 data events.
Not recorded by default (additional cost)

Insights Events

Unusual API activity detection:

Detection:
  - API call rate anomalies
  - Error rate spikes
  - Unusual patterns

Example:
  - Sudden spike in DeleteObject calls
  - Unusual number of failed AuthorizeSecurityGroupIngress
  - High rate of DescribeInstances

Event Structure

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "AIDAJ45Q7YFFAREXAMPLE",
    "arn": "arn:aws:iam::123456789012:user/alice",
    "accountId": "123456789012",
    "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
    "userName": "alice",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2024-01-15T10:00:00Z"
      }
    }
  },
  "eventTime": "2024-01-15T10:30:00Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "DeleteBucket",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "203.0.113.50",
  "userAgent": "aws-cli/2.0.0 Python/3.8.0 Linux/5.4.0",
  "requestParameters": {
    "bucketName": "my-important-bucket"
  },
  "responseElements": null,
  "requestID": "EXAMPLE123456789",
  "eventID": "EXAMPLE-abcd-1234-5678-EXAMPLE12345",
  "readOnly": false,
  "resources": [
    {
      "accountId": "123456789012",
      "type": "AWS::S3::Bucket",
      "ARN": "arn:aws:s3:::my-important-bucket"
    }
  ],
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "123456789012",
  "eventCategory": "Management"
}

Creating a Trail

Multi-Region Trail

# Create trail with all features
aws cloudtrail create-trail \
  --name organization-trail \
  --s3-bucket-name my-cloudtrail-bucket \
  --is-multi-region-trail \           # Captures events in ALL regions -- critical
                                      # because attackers deliberately target regions
                                      # you are not actively monitoring
  --enable-log-file-validation \      # Creates digest files with SHA-256 hashes so
                                      # you can prove logs have not been tampered with
  --include-global-service-events \   # IAM, STS, CloudFront are global services that
                                      # only log to us-east-1 -- this flag includes them
  --kms-key-id alias/cloudtrail-key   # Encrypt logs at rest with your own key --
                                      # required for most compliance frameworks

# Start logging
aws cloudtrail start-logging --name organization-trail

# Enable data events for all S3 buckets
aws cloudtrail put-event-selectors \
  --trail-name organization-trail \
  --event-selectors '[{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [{
      "Type": "AWS::S3::Object",
      "Values": ["arn:aws:s3"]
    }]
  }]'

CloudFormation

AWSTemplateFormatVersion: '2010-09-09'
Description: Enterprise CloudTrail Configuration

Resources:
  CloudTrailBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${AWS::AccountId}-cloudtrail-logs
      VersioningConfiguration:
        Status: Enabled
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !Ref CloudTrailKey
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      # Lifecycle rules keep costs under control while meeting retention requirements.
      # Common pattern: hot (S3 Standard) for 90 days for active investigations,
      # cold (Glacier) for archival, then expire after 365 days.
      # Cost tip: Glacier storage is ~$0.004/GB/month vs $0.023/GB for Standard --
      # an 83% savings for logs older than 90 days.
      # Adjust ExpirationInDays based on your compliance requirements (PCI requires
      # 1 year, HIPAA requires 6 years, some financial regulations require 7 years).
      LifecycleConfiguration:
        Rules:
          - Id: ArchiveOldLogs
            Status: Enabled
            Transitions:
              - StorageClass: GLACIER
                TransitionInDays: 90
            ExpirationInDays: 365

  CloudTrailBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref CloudTrailBucket
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AWSCloudTrailAclCheck
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:GetBucketAcl
            Resource: !GetAtt CloudTrailBucket.Arn
          - Sid: AWSCloudTrailWrite
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:PutObject
            Resource: !Sub ${CloudTrailBucket.Arn}/*
            Condition:
              StringEquals:
                s3:x-amz-acl: bucket-owner-full-control

  CloudTrailKey:
    Type: AWS::KMS::Key
    Properties:
      Description: CloudTrail encryption key
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable Root Account
            Effect: Allow
            Principal:
              AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
            Action: kms:*
            Resource: '*'
          - Sid: Allow CloudTrail
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action:
              - kms:GenerateDataKey*
              - kms:DescribeKey
            Resource: '*'

  OrganizationTrail:
    Type: AWS::CloudTrail::Trail
    DependsOn: CloudTrailBucketPolicy
    Properties:
      TrailName: organization-trail
      S3BucketName: !Ref CloudTrailBucket
      IsMultiRegionTrail: true
      EnableLogFileValidation: true
      IncludeGlobalServiceEvents: true
      IsOrganizationTrail: true
      KMSKeyId: !Ref CloudTrailKey
      CloudWatchLogsLogGroupArn: !GetAtt CloudTrailLogGroup.Arn
      CloudWatchLogsRoleArn: !GetAtt CloudTrailRole.Arn
      EventSelectors:
        - ReadWriteType: All
          IncludeManagementEvents: true
          DataResources:
            - Type: AWS::S3::Object
              Values:
                - arn:aws:s3
            - Type: AWS::Lambda::Function
              Values:
                - arn:aws:lambda
      InsightSelectors:
        - InsightType: ApiCallRateInsight
        - InsightType: ApiErrorRateInsight

  CloudTrailLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      RetentionInDays: 90

  CloudTrailRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: CloudTrailLogsPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: !GetAtt CloudTrailLogGroup.Arn

CloudTrail Lake

Create Event Data Store

CloudTrail Lake is a managed data lake purpose-built for CloudTrail events. Instead of the old workflow (S3 -> Athena -> query), Lake lets you run SQL queries directly against your events. The trade-off: Lake costs more per GB than S3+Athena for large volumes, but eliminates the operational overhead of maintaining Glue tables, partitions, and Athena workgroups. For teams without a dedicated data platform, Lake is usually the right choice.

# Create event data store for 1 year retention.
# Cost tip: Lake charges $2.50/GB ingested and $0.005/GB scanned.
# For a 50-account org generating ~10 GB/month, that is ~$25/month for ingestion.
aws cloudtrail create-event-data-store \
  --name security-events \
  --retention-period 365 \
  --multi-region-enabled \
  --organization-enabled

# Get event data store ID
aws cloudtrail list-event-data-stores

Query with SQL

-- Find all DeleteBucket events
SELECT 
    eventTime,
    userIdentity.userName,
    sourceIPAddress,
    requestParameters.bucketName
FROM security-events
WHERE eventName = 'DeleteBucket'
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC
LIMIT 100;

-- Find failed console logins
SELECT 
    eventTime,
    userIdentity.userName,
    sourceIPAddress,
    errorCode,
    errorMessage
FROM security-events
WHERE eventName = 'ConsoleLogin'
AND errorCode IS NOT NULL
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;

-- Find root account activity
SELECT 
    eventTime,
    eventName,
    sourceIPAddress,
    userAgent
FROM security-events
WHERE userIdentity.type = 'Root'
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;

-- Find security group modifications
SELECT 
    eventTime,
    userIdentity.userName,
    eventName,
    requestParameters
FROM security-events
WHERE eventName LIKE '%SecurityGroup%'
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;

-- Find IAM policy changes
SELECT 
    eventTime,
    userIdentity.userName,
    eventName,
    requestParameters.policyArn
FROM security-events
WHERE eventName LIKE '%Policy%'
AND eventSource = 'iam.amazonaws.com'
AND eventTime > '2024-01-01 00:00:00';

CloudTrail Insights

Enable Insights

# Enable insights on existing trail
aws cloudtrail put-insight-selectors \
  --trail-name organization-trail \
  --insight-selectors '[
    {"InsightType": "ApiCallRateInsight"},
    {"InsightType": "ApiErrorRateInsight"}
  ]'

Insights Event Example

{
  "eventVersion": "1.08",
  "eventTime": "2024-01-15T10:30:00Z",
  "eventSource": "cloudtrail.amazonaws.com",
  "eventName": "DeleteObject",
  "eventCategory": "Insight",
  "insightDetails": {
    "state": "Start",
    "eventSource": "s3.amazonaws.com",
    "eventName": "DeleteObject",
    "insightType": "ApiCallRateInsight",
    "insightContext": {
      "statistics": {
        "baseline": {
          "average": 10.5
        },
        "insight": {
          "average": 500.2
        }
      }
    }
  }
}

Security Monitoring

CloudWatch Alarms

# Metric filter for root account usage
Resources:
  RootAccountUsageFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ $.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent" }'
      MetricTransformations:
        - MetricName: RootAccountUsage
          MetricNamespace: CloudTrail
          MetricValue: '1'

  RootAccountUsageAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: RootAccountUsage
      AlarmDescription: Root account was used
      MetricName: RootAccountUsage
      Namespace: CloudTrail
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - !Ref SecurityAlertsTopic

  # Unauthorized API calls
  UnauthorizedAPICallsFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ ($.errorCode = "*UnauthorizedAccess*") || ($.errorCode = "AccessDenied*") }'
      MetricTransformations:
        - MetricName: UnauthorizedAPICalls
          MetricNamespace: CloudTrail
          MetricValue: '1'

  # Console login without MFA
  NoMFALoginFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ ($.eventName = "ConsoleLogin") && ($.additionalEventData.MFAUsed != "Yes") }'
      MetricTransformations:
        - MetricName: ConsoleLoginWithoutMFA
          MetricNamespace: CloudTrail
          MetricValue: '1'

  # IAM policy changes
  IAMPolicyChangesFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ ($.eventName=DeleteGroupPolicy) || ($.eventName=DeleteRolePolicy) || ($.eventName=DeleteUserPolicy) || ($.eventName=PutGroupPolicy) || ($.eventName=PutRolePolicy) || ($.eventName=PutUserPolicy) || ($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=CreatePolicyVersion) || ($.eventName=DeletePolicyVersion) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=DetachUserPolicy) || ($.eventName=AttachGroupPolicy) || ($.eventName=DetachGroupPolicy) }'
      MetricTransformations:
        - MetricName: IAMPolicyChanges
          MetricNamespace: CloudTrail
          MetricValue: '1'

EventBridge Rules

Resources:
  # Alert on security group changes
  SecurityGroupChangeRule:
    Type: AWS::Events::Rule
    Properties:
      Name: security-group-changes
      EventPattern:
        source:
          - aws.ec2
        detail-type:
          - AWS API Call via CloudTrail
        detail:
          eventSource:
            - ec2.amazonaws.com
          eventName:
            - AuthorizeSecurityGroupIngress
            - AuthorizeSecurityGroupEgress
            - RevokeSecurityGroupIngress
            - RevokeSecurityGroupEgress
            - CreateSecurityGroup
            - DeleteSecurityGroup
      Targets:
        - Arn: !Ref SecurityAlertsTopic
          Id: SNSTarget

Investigation Workflow

┌─────────────────────────────────────────────────────────────────────────┐
│                    Security Investigation Workflow                       │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   1. Alert Received                                                     │
│   ─────────────────                                                     │
│   ─ GuardDuty finding                                                   │
│   ─ Security Hub alert                                                  │
│   ─ CloudWatch alarm                                                    │
│                                                                          │
│   2. Initial Triage                                                     │
│   ─────────────────                                                     │
│   Query CloudTrail Lake:                                                │
│   ─ Who: userIdentity (user/role/service)                              │
│   ─ What: eventName, requestParameters                                  │
│   ─ When: eventTime                                                     │
│   ─ Where: sourceIPAddress, awsRegion                                  │
│                                                                          │
│   3. Timeline Analysis                                                  │
│   ──────────────────                                                    │
│   ─ Events before/after incident                                       │
│   ─ Related API calls from same identity                               │
│   ─ Same source IP activity                                            │
│                                                                          │
│   4. Scope Assessment                                                   │
│   ──────────────────                                                    │
│   ─ Which resources were accessed?                                     │
│   ─ What data was potentially exposed?                                 │
│   ─ Are other accounts affected?                                       │
│                                                                          │
│   5. Containment                                                        │
│   ─────────────                                                         │
│   ─ Disable compromised credentials                                    │
│   ─ Revoke sessions                                                    │
│   ─ Isolate affected resources                                         │
│                                                                          │
│   6. Document & Report                                                  │
│   ───────────────────                                                   │
│   ─ Export relevant CloudTrail events                                  │
│   ─ Create incident report                                             │
│   ─ Update runbooks                                                    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Best Practices

Enable Multi-Region

Attackers may target regions you don’t actively use

Log Validation

Enable log file integrity validation for forensics

Use Lake

CloudTrail Lake enables SQL queries without S3/Athena setup

Enable Insights

Detect unusual API activity patterns automatically

Security Checklist

Logging Configuration:
  ✓ Multi-region trail enabled
  ✓ Global service events included
  ✓ Log file validation enabled
  ✓ S3 bucket encrypted with KMS
  ✓ Data events for sensitive S3 buckets
  ✓ Lambda invocation logging
  ✓ CloudTrail Insights enabled

Monitoring:
  ✓ CloudWatch alarms for critical events
  ✓ EventBridge rules for real-time alerts
  ✓ Root account usage monitoring
  ✓ Failed authentication tracking

Retention:
  ✓ S3 lifecycle policies configured
  ✓ CloudTrail Lake for long-term analysis
  ✓ Compliance requirements met

🎯 Interview Questions

Q1: What's the difference between management events and data events?

Management Events (Control Plane):

Operations on AWS resources
CreateBucket, DeleteUser, StartInstances
Logged by default
Lower volume

Data Events (Data Plane):

Operations on data within resources
GetObject, PutObject, Invoke
Not logged by default (extra cost)
High volume

Q2: How would you investigate a potential security breach using CloudTrail?

Identify compromised credentials: Find the access key or role
Timeline analysis: Query all events from that identity
Source IP analysis: Check for unusual IPs or geolocations
Resource access: Identify what was accessed/modified
Lateral movement: Check for AssumeRole or CreateAccessKey
Scope assessment: Determine if other accounts affected

Use CloudTrail Lake SQL queries for efficient analysis.

Q3: How do you ensure CloudTrail log integrity?

Enable log file validation: Creates digest files with hashes
KMS encryption: Encrypt logs at rest
S3 bucket policy: Prevent deletion/modification
Object Lock: Enable S3 Object Lock for immutability
Cross-account: Store logs in separate security account
MFA Delete: Require MFA for log deletion

Hands-On Lab

Create Organization Trail

Set up multi-region trail with encryption and validation

Enable Data Events

Configure S3 and Lambda data event logging

Set Up CloudTrail Lake

Create event data store and run sample queries

Create Alarms

Set up CloudWatch alarms for root account usage

Investigate

Use CloudTrail Lake to investigate a simulated incident

Next Module

AWS Config

Configuration compliance and change management

​Module Overview

​How CloudTrail Works

​Event Types

​Management Events

​Data Events

​Insights Events

​Event Structure

​Creating a Trail

​Multi-Region Trail

​CloudFormation

​CloudTrail Lake

​Create Event Data Store

​Query with SQL

​CloudTrail Insights

​Enable Insights

​Insights Event Example

​Security Monitoring

​CloudWatch Alarms

​EventBridge Rules

​Investigation Workflow

​Best Practices

Enable Multi-Region

Log Validation

Use Lake

Enable Insights

​Security Checklist

​🎯 Interview Questions

​Hands-On Lab

​Next Module

AWS Config

Module Overview

How CloudTrail Works

Event Types

Management Events

Data Events

Insights Events

Event Structure

Creating a Trail

Multi-Region Trail

CloudFormation

CloudTrail Lake

Create Event Data Store

Query with SQL

CloudTrail Insights

Enable Insights

Insights Event Example

Security Monitoring

CloudWatch Alarms

EventBridge Rules

Investigation Workflow

Best Practices

Security Checklist

🎯 Interview Questions

Hands-On Lab

Next Module