Skip to main content
AWS CloudTrail Architecture

​
Module Overview

Estimated Time: 2-3 hours | Difficulty: Intermediate | Prerequisites: IAM, S3, basic logging concepts
AWS CloudTrail is a service that enables governance, compliance, and operational and risk auditing of your AWS account. It records API calls made to AWS services and delivers log files to an S3 bucket. What You’ll Learn:
  • CloudTrail event types and structure
  • Trail configuration and multi-region setup
  • CloudTrail Lake for advanced querying
  • CloudTrail Insights for anomaly detection
  • Security investigation workflows

​
How CloudTrail Works

┌─────────────────────────────────────────────────────────────────────────┐
│                      CloudTrail Architecture                             │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   API Activity                   CloudTrail               Destinations  │
│   ────────────                   ──────────               ────────────  │
│                                                                          │
│   ┌──────────────┐          ┌─────────────────┐     ┌────────────────┐  │
│   │  AWS Console │─────────▶│                 │────▶│ S3 Bucket      │  │
│   └──────────────┘          │                 │     │ (encrypted)    │  │
│                              │   CloudTrail    │     └────────────────┘  │
│   ┌──────────────┐          │     Trail       │                         │
│   │   AWS CLI    │─────────▶│                 │     ┌────────────────┐  │
│   └──────────────┘          │  ─ Management   │────▶│ CloudWatch Logs│  │
│                              │    Events       │     └────────────────┘  │
│   ┌──────────────┐          │  ─ Data Events  │                         │
│   │   AWS SDKs   │─────────▶│  ─ Insights     │     ┌────────────────┐  │
│   └──────────────┘          │    Events       │────▶│ CloudTrail Lake│  │
│                              │                 │     └────────────────┘  │
│   ┌──────────────┐          └─────────────────┘                         │
│   │ AWS Services │─────────▶                        ┌────────────────┐  │
│   └──────────────┘                                  │ EventBridge    │  │
│                                                      └────────────────┘  │
│                                                                          │
│   Event Flow:                                                            │
│   ───────────                                                            │
│   1. User/Service makes API call                                        │
│   2. CloudTrail captures the event                                      │
│   3. Event processed and delivered (within ~15 minutes)                 │
│   4. Stored in S3, queryable in Lake                                    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

​
Event Types

​
Management Events

Operations on AWS resources (control plane): 
Examples:
  - CreateBucket, DeleteBucket
  - CreateUser, DeleteRole
  - StartInstances, TerminateInstances
  - CreateVpc, AttachInternetGateway
  - PutBucketPolicy, PutRolePolicy

Recorded by default in all trails

​
Data Events

Operations on data within resources: 
Examples:
  S3 Object-level:
    - GetObject
    - PutObject
    - DeleteObject
  
  Lambda Invocations:
    - Invoke function calls
  
  DynamoDB:
    - GetItem, PutItem, DeleteItem
  
  EBS Direct APIs:
    - Snapshot operations

Not recorded by default (additional cost)

​
Insights Events

Unusual API activity detection: 
Detection:
  - API call rate anomalies
  - Error rate spikes
  - Unusual patterns

Example:
  - Sudden spike in DeleteObject calls
  - Unusual number of failed AuthorizeSecurityGroupIngress
  - High rate of DescribeInstances

​
Event Structure

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "AIDAJ45Q7YFFAREXAMPLE",
    "arn": "arn:aws:iam::123456789012:user/alice",
    "accountId": "123456789012",
    "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
    "userName": "alice",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2024-01-15T10:00:00Z"
      }
    }
  },
  "eventTime": "2024-01-15T10:30:00Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "DeleteBucket",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "203.0.113.50",
  "userAgent": "aws-cli/2.0.0 Python/3.8.0 Linux/5.4.0",
  "requestParameters": {
    "bucketName": "my-important-bucket"
  },
  "responseElements": null,
  "requestID": "EXAMPLE123456789",
  "eventID": "EXAMPLE-abcd-1234-5678-EXAMPLE12345",
  "readOnly": false,
  "resources": [
    {
      "accountId": "123456789012",
      "type": "AWS::S3::Bucket",
      "ARN": "arn:aws:s3:::my-important-bucket"
    }
  ],
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "123456789012",
  "eventCategory": "Management"
}

​
Creating a Trail

​
Multi-Region Trail

# Create trail with all features
aws cloudtrail create-trail \
  --name organization-trail \
  --s3-bucket-name my-cloudtrail-bucket \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --include-global-service-events \
  --kms-key-id alias/cloudtrail-key

# Start logging
aws cloudtrail start-logging --name organization-trail

# Enable data events for all S3 buckets
aws cloudtrail put-event-selectors \
  --trail-name organization-trail \
  --event-selectors '[{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [{
      "Type": "AWS::S3::Object",
      "Values": ["arn:aws:s3"]
    }]
  }]'

​
CloudFormation

AWSTemplateFormatVersion: '2010-09-09'
Description: Enterprise CloudTrail Configuration

Resources:
  CloudTrailBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${AWS::AccountId}-cloudtrail-logs
      VersioningConfiguration:
        Status: Enabled
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !Ref CloudTrailKey
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      LifecycleConfiguration:
        Rules:
          - Id: ArchiveOldLogs
            Status: Enabled
            Transitions:
              - StorageClass: GLACIER
                TransitionInDays: 90
            ExpirationInDays: 365

  CloudTrailBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref CloudTrailBucket
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AWSCloudTrailAclCheck
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:GetBucketAcl
            Resource: !GetAtt CloudTrailBucket.Arn
          - Sid: AWSCloudTrailWrite
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:PutObject
            Resource: !Sub ${CloudTrailBucket.Arn}/*
            Condition:
              StringEquals:
                s3:x-amz-acl: bucket-owner-full-control

  CloudTrailKey:
    Type: AWS::KMS::Key
    Properties:
      Description: CloudTrail encryption key
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable Root Account
            Effect: Allow
            Principal:
              AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
            Action: kms:*
            Resource: '*'
          - Sid: Allow CloudTrail
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action:
              - kms:GenerateDataKey*
              - kms:DescribeKey
            Resource: '*'

  OrganizationTrail:
    Type: AWS::CloudTrail::Trail
    DependsOn: CloudTrailBucketPolicy
    Properties:
      TrailName: organization-trail
      S3BucketName: !Ref CloudTrailBucket
      IsMultiRegionTrail: true
      EnableLogFileValidation: true
      IncludeGlobalServiceEvents: true
      IsOrganizationTrail: true
      KMSKeyId: !Ref CloudTrailKey
      CloudWatchLogsLogGroupArn: !GetAtt CloudTrailLogGroup.Arn
      CloudWatchLogsRoleArn: !GetAtt CloudTrailRole.Arn
      EventSelectors:
        - ReadWriteType: All
          IncludeManagementEvents: true
          DataResources:
            - Type: AWS::S3::Object
              Values:
                - arn:aws:s3
            - Type: AWS::Lambda::Function
              Values:
                - arn:aws:lambda
      InsightSelectors:
        - InsightType: ApiCallRateInsight
        - InsightType: ApiErrorRateInsight

  CloudTrailLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      RetentionInDays: 90

  CloudTrailRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: CloudTrailLogsPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: !GetAtt CloudTrailLogGroup.Arn

​
CloudTrail Lake

​
Create Event Data Store

# Create event data store for 1 year retention
aws cloudtrail create-event-data-store \
  --name security-events \
  --retention-period 365 \
  --multi-region-enabled \
  --organization-enabled

# Get event data store ID
aws cloudtrail list-event-data-stores

​
Query with SQL

-- Find all DeleteBucket events
SELECT 
    eventTime,
    userIdentity.userName,
    sourceIPAddress,
    requestParameters.bucketName
FROM security-events
WHERE eventName = 'DeleteBucket'
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC
LIMIT 100;

-- Find failed console logins
SELECT 
    eventTime,
    userIdentity.userName,
    sourceIPAddress,
    errorCode,
    errorMessage
FROM security-events
WHERE eventName = 'ConsoleLogin'
AND errorCode IS NOT NULL
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;

-- Find root account activity
SELECT 
    eventTime,
    eventName,
    sourceIPAddress,
    userAgent
FROM security-events
WHERE userIdentity.type = 'Root'
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;

-- Find security group modifications
SELECT 
    eventTime,
    userIdentity.userName,
    eventName,
    requestParameters
FROM security-events
WHERE eventName LIKE '%SecurityGroup%'
AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;

-- Find IAM policy changes
SELECT 
    eventTime,
    userIdentity.userName,
    eventName,
    requestParameters.policyArn
FROM security-events
WHERE eventName LIKE '%Policy%'
AND eventSource = 'iam.amazonaws.com'
AND eventTime > '2024-01-01 00:00:00';

​
CloudTrail Insights

​
Enable Insights

# Enable insights on existing trail
aws cloudtrail put-insight-selectors \
  --trail-name organization-trail \
  --insight-selectors '[
    {"InsightType": "ApiCallRateInsight"},
    {"InsightType": "ApiErrorRateInsight"}
  ]'

​
Insights Event Example

{
  "eventVersion": "1.08",
  "eventTime": "2024-01-15T10:30:00Z",
  "eventSource": "cloudtrail.amazonaws.com",
  "eventName": "DeleteObject",
  "eventCategory": "Insight",
  "insightDetails": {
    "state": "Start",
    "eventSource": "s3.amazonaws.com",
    "eventName": "DeleteObject",
    "insightType": "ApiCallRateInsight",
    "insightContext": {
      "statistics": {
        "baseline": {
          "average": 10.5
        },
        "insight": {
          "average": 500.2
        }
      }
    }
  }
}

​
Security Monitoring

​
CloudWatch Alarms

# Metric filter for root account usage
Resources:
  RootAccountUsageFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ $.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent" }'
      MetricTransformations:
        - MetricName: RootAccountUsage
          MetricNamespace: CloudTrail
          MetricValue: '1'

  RootAccountUsageAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: RootAccountUsage
      AlarmDescription: Root account was used
      MetricName: RootAccountUsage
      Namespace: CloudTrail
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - !Ref SecurityAlertsTopic

  # Unauthorized API calls
  UnauthorizedAPICallsFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ ($.errorCode = "*UnauthorizedAccess*") || ($.errorCode = "AccessDenied*") }'
      MetricTransformations:
        - MetricName: UnauthorizedAPICalls
          MetricNamespace: CloudTrail
          MetricValue: '1'

  # Console login without MFA
  NoMFALoginFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ ($.eventName = "ConsoleLogin") && ($.additionalEventData.MFAUsed != "Yes") }'
      MetricTransformations:
        - MetricName: ConsoleLoginWithoutMFA
          MetricNamespace: CloudTrail
          MetricValue: '1'

  # IAM policy changes
  IAMPolicyChangesFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: /aws/cloudtrail/organization
      FilterPattern: '{ ($.eventName=DeleteGroupPolicy) || ($.eventName=DeleteRolePolicy) || ($.eventName=DeleteUserPolicy) || ($.eventName=PutGroupPolicy) || ($.eventName=PutRolePolicy) || ($.eventName=PutUserPolicy) || ($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=CreatePolicyVersion) || ($.eventName=DeletePolicyVersion) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=DetachUserPolicy) || ($.eventName=AttachGroupPolicy) || ($.eventName=DetachGroupPolicy) }'
      MetricTransformations:
        - MetricName: IAMPolicyChanges
          MetricNamespace: CloudTrail
          MetricValue: '1'

​
EventBridge Rules

Resources:
  # Alert on security group changes
  SecurityGroupChangeRule:
    Type: AWS::Events::Rule
    Properties:
      Name: security-group-changes
      EventPattern:
        source:
          - aws.ec2
        detail-type:
          - AWS API Call via CloudTrail
        detail:
          eventSource:
            - ec2.amazonaws.com
          eventName:
            - AuthorizeSecurityGroupIngress
            - AuthorizeSecurityGroupEgress
            - RevokeSecurityGroupIngress
            - RevokeSecurityGroupEgress
            - CreateSecurityGroup
            - DeleteSecurityGroup
      Targets:
        - Arn: !Ref SecurityAlertsTopic
          Id: SNSTarget

​
Investigation Workflow

┌─────────────────────────────────────────────────────────────────────────┐
│                    Security Investigation Workflow                       │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   1. Alert Received                                                     │
│   ─────────────────                                                     │
│   ─ GuardDuty finding                                                   │
│   ─ Security Hub alert                                                  │
│   ─ CloudWatch alarm                                                    │
│                                                                          │
│   2. Initial Triage                                                     │
│   ─────────────────                                                     │
│   Query CloudTrail Lake:                                                │
│   ─ Who: userIdentity (user/role/service)                              │
│   ─ What: eventName, requestParameters                                  │
│   ─ When: eventTime                                                     │
│   ─ Where: sourceIPAddress, awsRegion                                  │
│                                                                          │
│   3. Timeline Analysis                                                  │
│   ──────────────────                                                    │
│   ─ Events before/after incident                                       │
│   ─ Related API calls from same identity                               │
│   ─ Same source IP activity                                            │
│                                                                          │
│   4. Scope Assessment                                                   │
│   ──────────────────                                                    │
│   ─ Which resources were accessed?                                     │
│   ─ What data was potentially exposed?                                 │
│   ─ Are other accounts affected?                                       │
│                                                                          │
│   5. Containment                                                        │
│   ─────────────                                                         │
│   ─ Disable compromised credentials                                    │
│   ─ Revoke sessions                                                    │
│   ─ Isolate affected resources                                         │
│                                                                          │
│   6. Document & Report                                                  │
│   ───────────────────                                                   │
│   ─ Export relevant CloudTrail events                                  │
│   ─ Create incident report                                             │
│   ─ Update runbooks                                                    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

​
Best Practices

Enable Multi-Region

Attackers may target regions you don’t actively use

Log Validation

Enable log file integrity validation for forensics

Use Lake

CloudTrail Lake enables SQL queries without S3/Athena setup

Enable Insights

Detect unusual API activity patterns automatically

​
Security Checklist

Logging Configuration:
  ✓ Multi-region trail enabled
  ✓ Global service events included
  ✓ Log file validation enabled
  ✓ S3 bucket encrypted with KMS
  ✓ Data events for sensitive S3 buckets
  ✓ Lambda invocation logging
  ✓ CloudTrail Insights enabled

Monitoring:
  ✓ CloudWatch alarms for critical events
  ✓ EventBridge rules for real-time alerts
  ✓ Root account usage monitoring
  ✓ Failed authentication tracking

Retention:
  ✓ S3 lifecycle policies configured
  ✓ CloudTrail Lake for long-term analysis
  ✓ Compliance requirements met

​
🎯 Interview Questions

Management Events (Control Plane):
  • Operations on AWS resources
  • CreateBucket, DeleteUser, StartInstances
  • Logged by default
  • Lower volume
Data Events (Data Plane):
  • Operations on data within resources
  • GetObject, PutObject, Invoke
  • Not logged by default (extra cost)
  • High volume
  1. Identify compromised credentials: Find the access key or role
  2. Timeline analysis: Query all events from that identity
  3. Source IP analysis: Check for unusual IPs or geolocations
  4. Resource access: Identify what was accessed/modified
  5. Lateral movement: Check for AssumeRole or CreateAccessKey
  6. Scope assessment: Determine if other accounts affected
Use CloudTrail Lake SQL queries for efficient analysis.
  1. Enable log file validation: Creates digest files with hashes
  2. KMS encryption: Encrypt logs at rest
  3. S3 bucket policy: Prevent deletion/modification
  4. Object Lock: Enable S3 Object Lock for immutability
  5. Cross-account: Store logs in separate security account
  6. MFA Delete: Require MFA for log deletion

​
Hands-On Lab

1

Create Organization Trail

Set up multi-region trail with encryption and validation
2

Enable Data Events

Configure S3 and Lambda data event logging
3

Set Up CloudTrail Lake

Create event data store and run sample queries
4

Create Alarms

Set up CloudWatch alarms for root account usage
5

Investigate

Use CloudTrail Lake to investigate a simulated incident

​
Next Module

AWS Config

Configuration compliance and change management