Documentation Index Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
Use this file to discover all available pages before exploring further.
Module Overview Estimated Time : 2-3 hours | Difficulty : Intermediate | Prerequisites : IAM, S3, basic logging concepts
AWS CloudTrail is a service that enables governance, compliance, and operational and risk auditing of your AWS account. It records API calls made to AWS services and delivers log files to an S3 bucket. Think of CloudTrail as the security camera system for your AWS account — it records who did what, when, and from where. When an S3 bucket gets deleted at 2 AM and nobody claims responsibility, CloudTrail is how you find out which access key made the call and from which IP address. It is non-negotiable for any production AWS environment.
What You’ll Learn:
CloudTrail event types and structure
Trail configuration and multi-region setup
CloudTrail Lake for advanced querying
CloudTrail Insights for anomaly detection
Security investigation workflows
How CloudTrail Works ┌─────────────────────────────────────────────────────────────────────────┐ │ CloudTrail Architecture │ ├─────────────────────────────────────────────────────────────────────────┤ │ │ │ API Activity CloudTrail Destinations │ │ ──────────── ────────── ──────────── │ │ │ │ ┌──────────────┐ ┌─────────────────┐ ┌────────────────┐ │ │ │ AWS Console │─────────▶│ │────▶│ S3 Bucket │ │ │ └──────────────┘ │ │ │ (encrypted) │ │ │ │ CloudTrail │ └────────────────┘ │ │ ┌──────────────┐ │ Trail │ │ │ │ AWS CLI │─────────▶│ │ ┌────────────────┐ │ │ └──────────────┘ │ ─ Management │────▶│ CloudWatch Logs│ │ │ │ Events │ └────────────────┘ │ │ ┌──────────────┐ │ ─ Data Events │ │ │ │ AWS SDKs │─────────▶│ ─ Insights │ ┌────────────────┐ │ │ └──────────────┘ │ Events │────▶│ CloudTrail Lake│ │ │ │ │ └────────────────┘ │ │ ┌──────────────┐ └─────────────────┘ │ │ │ AWS Services │─────────▶ ┌────────────────┐ │ │ └──────────────┘ │ EventBridge │ │ │ └────────────────┘ │ │ │ │ Event Flow: │ │ ─────────── │ │ 1. User/Service makes API call │ │ 2. CloudTrail captures the event │ │ 3. Event processed and delivered (within ~15 minutes) │ │ 4. Stored in S3, queryable in Lake │ │ │ └─────────────────────────────────────────────────────────────────────────┘
Event Types Management Events Operations on AWS resources (control plane):
# Management events = "control plane" operations. # Think of these as changes to the infrastructure itself, not the data inside it. # Analogy: management events are like building security footage (who entered # which room), while data events are like watching what they did at their desk. Examples : - CreateBucket, DeleteBucket - CreateUser, DeleteRole - StartInstances, TerminateInstances - CreateVpc, AttachInternetGateway - PutBucketPolicy, PutRolePolicy # Recorded by default in all trails -- no extra configuration or cost needed. # This is one of the first things auditors will check in a compliance review. Recorded by default in all trails
Data Events Operations on data within resources:
Examples : S3 Object-level : - GetObject - PutObject - DeleteObject Lambda Invocations : - Invoke function calls DynamoDB : - GetItem, PutItem, DeleteItem EBS Direct APIs : - Snapshot operations # NOT recorded by default -- you must explicitly enable data events. # Why not default? Volume. A single busy S3 bucket can generate millions # of GetObject events per day. Cost tip: enable data events only for # sensitive buckets (PII, financial data), not for log or asset buckets. # Pricing: roughly $0.10 per 100,000 data events. Not recorded by default (additional cost)
Insights Events Unusual API activity detection:
Detection : - API call rate anomalies - Error rate spikes - Unusual patterns Example : - Sudden spike in DeleteObject calls - Unusual number of failed AuthorizeSecurityGroupIngress - High rate of DescribeInstances
Event Structure { "eventVersion" : "1.08" , "userIdentity" : { "type" : "IAMUser" , "principalId" : "AIDAJ45Q7YFFAREXAMPLE" , "arn" : "arn:aws:iam::123456789012:user/alice" , "accountId" : "123456789012" , "accessKeyId" : "AKIAIOSFODNN7EXAMPLE" , "userName" : "alice" , "sessionContext" : { "sessionIssuer" : {}, "webIdFederationData" : {}, "attributes" : { "mfaAuthenticated" : "true" , "creationDate" : "2024-01-15T10:00:00Z" } } }, "eventTime" : "2024-01-15T10:30:00Z" , "eventSource" : "s3.amazonaws.com" , "eventName" : "DeleteBucket" , "awsRegion" : "us-east-1" , "sourceIPAddress" : "203.0.113.50" , "userAgent" : "aws-cli/2.0.0 Python/3.8.0 Linux/5.4.0" , "requestParameters" : { "bucketName" : "my-important-bucket" }, "responseElements" : null , "requestID" : "EXAMPLE123456789" , "eventID" : "EXAMPLE-abcd-1234-5678-EXAMPLE12345" , "readOnly" : false , "resources" : [ { "accountId" : "123456789012" , "type" : "AWS::S3::Bucket" , "ARN" : "arn:aws:s3:::my-important-bucket" } ], "eventType" : "AwsApiCall" , "managementEvent" : true , "recipientAccountId" : "123456789012" , "eventCategory" : "Management" }
Creating a Trail Multi-Region Trail # Create trail with all features aws cloudtrail create-trail \ --name organization-trail \ --s3-bucket-name my-cloudtrail-bucket \ --is-multi-region-trail \ # Captures events in ALL regions -- critical # because attackers deliberately target regions # you are not actively monitoring --enable-log-file-validation \ # Creates digest files with SHA-256 hashes so # you can prove logs have not been tampered with --include-global-service-events \ # IAM, STS, CloudFront are global services that # only log to us-east-1 -- this flag includes them --kms-key-id alias/cloudtrail-key # Encrypt logs at rest with your own key -- # required for most compliance frameworks # Start logging aws cloudtrail start-logging --name organization-trail # Enable data events for all S3 buckets aws cloudtrail put-event-selectors \ --trail-name organization-trail \ --event-selectors '[{ "ReadWriteType": "All", "IncludeManagementEvents": true, "DataResources": [{ "Type": "AWS::S3::Object", "Values": ["arn:aws:s3"] }] }]'
AWSTemplateFormatVersion : '2010-09-09' Description : Enterprise CloudTrail Configuration Resources : CloudTrailBucket : Type : AWS::S3::Bucket Properties : BucketName : !Sub ${AWS::AccountId}-cloudtrail-logs VersioningConfiguration : Status : Enabled BucketEncryption : ServerSideEncryptionConfiguration : - ServerSideEncryptionByDefault : SSEAlgorithm : aws:kms KMSMasterKeyID : !Ref CloudTrailKey PublicAccessBlockConfiguration : BlockPublicAcls : true BlockPublicPolicy : true IgnorePublicAcls : true RestrictPublicBuckets : true # Lifecycle rules keep costs under control while meeting retention requirements. # Common pattern: hot (S3 Standard) for 90 days for active investigations, # cold (Glacier) for archival, then expire after 365 days. # Cost tip: Glacier storage is ~$0.004/GB/month vs $0.023/GB for Standard -- # an 83% savings for logs older than 90 days. # Adjust ExpirationInDays based on your compliance requirements (PCI requires # 1 year, HIPAA requires 6 years, some financial regulations require 7 years). LifecycleConfiguration : Rules : - Id : ArchiveOldLogs Status : Enabled Transitions : - StorageClass : GLACIER TransitionInDays : 90 ExpirationInDays : 365 CloudTrailBucketPolicy : Type : AWS::S3::BucketPolicy Properties : Bucket : !Ref CloudTrailBucket PolicyDocument : Version : '2012-10-17' Statement : - Sid : AWSCloudTrailAclCheck Effect : Allow Principal : Service : cloudtrail.amazonaws.com Action : s3:GetBucketAcl Resource : !GetAtt CloudTrailBucket.Arn - Sid : AWSCloudTrailWrite Effect : Allow Principal : Service : cloudtrail.amazonaws.com Action : s3:PutObject Resource : !Sub ${CloudTrailBucket.Arn}/* Condition : StringEquals : s3:x-amz-acl : bucket-owner-full-control CloudTrailKey : Type : AWS::KMS::Key Properties : Description : CloudTrail encryption key EnableKeyRotation : true KeyPolicy : Version : '2012-10-17' Statement : - Sid : Enable Root Account Effect : Allow Principal : AWS : !Sub arn:aws:iam::${AWS::AccountId}:root Action : kms:* Resource : '*' - Sid : Allow CloudTrail Effect : Allow Principal : Service : cloudtrail.amazonaws.com Action : - kms:GenerateDataKey* - kms:DescribeKey Resource : '*' OrganizationTrail : Type : AWS::CloudTrail::Trail DependsOn : CloudTrailBucketPolicy Properties : TrailName : organization-trail S3BucketName : !Ref CloudTrailBucket IsMultiRegionTrail : true EnableLogFileValidation : true IncludeGlobalServiceEvents : true IsOrganizationTrail : true KMSKeyId : !Ref CloudTrailKey CloudWatchLogsLogGroupArn : !GetAtt CloudTrailLogGroup.Arn CloudWatchLogsRoleArn : !GetAtt CloudTrailRole.Arn EventSelectors : - ReadWriteType : All IncludeManagementEvents : true DataResources : - Type : AWS::S3::Object Values : - arn:aws:s3 - Type : AWS::Lambda::Function Values : - arn:aws:lambda InsightSelectors : - InsightType : ApiCallRateInsight - InsightType : ApiErrorRateInsight CloudTrailLogGroup : Type : AWS::Logs::LogGroup Properties : LogGroupName : /aws/cloudtrail/organization RetentionInDays : 90 CloudTrailRole : Type : AWS::IAM::Role Properties : AssumeRolePolicyDocument : Version : '2012-10-17' Statement : - Effect : Allow Principal : Service : cloudtrail.amazonaws.com Action : sts:AssumeRole Policies : - PolicyName : CloudTrailLogsPolicy PolicyDocument : Version : '2012-10-17' Statement : - Effect : Allow Action : - logs:CreateLogStream - logs:PutLogEvents Resource : !GetAtt CloudTrailLogGroup.Arn
CloudTrail Lake Create Event Data Store CloudTrail Lake is a managed data lake purpose-built for CloudTrail events. Instead of the old workflow (S3 -> Athena -> query), Lake lets you run SQL queries directly against your events. The trade-off: Lake costs more per GB than S3+Athena for large volumes, but eliminates the operational overhead of maintaining Glue tables, partitions, and Athena workgroups. For teams without a dedicated data platform, Lake is usually the right choice.
# Create event data store for 1 year retention. # Cost tip: Lake charges $2.50/GB ingested and $0.005/GB scanned. # For a 50-account org generating ~10 GB/month, that is ~$25/month for ingestion. aws cloudtrail create-event-data-store \ --name security-events \ --retention-period 365 \ --multi-region-enabled \ --organization-enabled # Get event data store ID aws cloudtrail list-event-data-stores
Query with SQL -- Find all DeleteBucket events SELECT eventTime, userIdentity . userName , sourceIPAddress, requestParameters . bucketName FROM security - events WHERE eventName = 'DeleteBucket' AND eventTime > '2024-01-01 00:00:00' ORDER BY eventTime DESC LIMIT 100 ; -- Find failed console logins SELECT eventTime, userIdentity . userName , sourceIPAddress, errorCode, errorMessage FROM security - events WHERE eventName = 'ConsoleLogin' AND errorCode IS NOT NULL AND eventTime > '2024-01-01 00:00:00' ORDER BY eventTime DESC ; -- Find root account activity SELECT eventTime, eventName, sourceIPAddress, userAgent FROM security - events WHERE userIdentity . type = 'Root' AND eventTime > '2024-01-01 00:00:00' ORDER BY eventTime DESC ; -- Find security group modifications SELECT eventTime, userIdentity . userName , eventName, requestParameters FROM security - events WHERE eventName LIKE '%SecurityGroup%' AND eventTime > '2024-01-01 00:00:00' ORDER BY eventTime DESC ; -- Find IAM policy changes SELECT eventTime, userIdentity . userName , eventName, requestParameters . policyArn FROM security - events WHERE eventName LIKE '%Policy%' AND eventSource = 'iam.amazonaws.com' AND eventTime > '2024-01-01 00:00:00' ;
CloudTrail Insights Enable Insights # Enable insights on existing trail aws cloudtrail put-insight-selectors \ --trail-name organization-trail \ --insight-selectors '[ {"InsightType": "ApiCallRateInsight"}, {"InsightType": "ApiErrorRateInsight"} ]'
Insights Event Example { "eventVersion" : "1.08" , "eventTime" : "2024-01-15T10:30:00Z" , "eventSource" : "cloudtrail.amazonaws.com" , "eventName" : "DeleteObject" , "eventCategory" : "Insight" , "insightDetails" : { "state" : "Start" , "eventSource" : "s3.amazonaws.com" , "eventName" : "DeleteObject" , "insightType" : "ApiCallRateInsight" , "insightContext" : { "statistics" : { "baseline" : { "average" : 10.5 }, "insight" : { "average" : 500.2 } } } } }
Security Monitoring CloudWatch Alarms # Metric filter for root account usage Resources : RootAccountUsageFilter : Type : AWS::Logs::MetricFilter Properties : LogGroupName : /aws/cloudtrail/organization FilterPattern : '{ $.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent" }' MetricTransformations : - MetricName : RootAccountUsage MetricNamespace : CloudTrail MetricValue : '1' RootAccountUsageAlarm : Type : AWS::CloudWatch::Alarm Properties : AlarmName : RootAccountUsage AlarmDescription : Root account was used MetricName : RootAccountUsage Namespace : CloudTrail Statistic : Sum Period : 300 EvaluationPeriods : 1 Threshold : 1 ComparisonOperator : GreaterThanOrEqualToThreshold AlarmActions : - !Ref SecurityAlertsTopic # Unauthorized API calls UnauthorizedAPICallsFilter : Type : AWS::Logs::MetricFilter Properties : LogGroupName : /aws/cloudtrail/organization FilterPattern : '{ ($.errorCode = "*UnauthorizedAccess*") || ($.errorCode = "AccessDenied*") }' MetricTransformations : - MetricName : UnauthorizedAPICalls MetricNamespace : CloudTrail MetricValue : '1' # Console login without MFA NoMFALoginFilter : Type : AWS::Logs::MetricFilter Properties : LogGroupName : /aws/cloudtrail/organization FilterPattern : '{ ($.eventName = "ConsoleLogin") && ($.additionalEventData.MFAUsed != "Yes") }' MetricTransformations : - MetricName : ConsoleLoginWithoutMFA MetricNamespace : CloudTrail MetricValue : '1' # IAM policy changes IAMPolicyChangesFilter : Type : AWS::Logs::MetricFilter Properties : LogGroupName : /aws/cloudtrail/organization FilterPattern : '{ ($.eventName=DeleteGroupPolicy) || ($.eventName=DeleteRolePolicy) || ($.eventName=DeleteUserPolicy) || ($.eventName=PutGroupPolicy) || ($.eventName=PutRolePolicy) || ($.eventName=PutUserPolicy) || ($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=CreatePolicyVersion) || ($.eventName=DeletePolicyVersion) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=DetachUserPolicy) || ($.eventName=AttachGroupPolicy) || ($.eventName=DetachGroupPolicy) }' MetricTransformations : - MetricName : IAMPolicyChanges MetricNamespace : CloudTrail MetricValue : '1'
EventBridge Rules Resources : # Alert on security group changes SecurityGroupChangeRule : Type : AWS::Events::Rule Properties : Name : security-group-changes EventPattern : source : - aws.ec2 detail-type : - AWS API Call via CloudTrail detail : eventSource : - ec2.amazonaws.com eventName : - AuthorizeSecurityGroupIngress - AuthorizeSecurityGroupEgress - RevokeSecurityGroupIngress - RevokeSecurityGroupEgress - CreateSecurityGroup - DeleteSecurityGroup Targets : - Arn : !Ref SecurityAlertsTopic Id : SNSTarget
Investigation Workflow ┌─────────────────────────────────────────────────────────────────────────┐ │ Security Investigation Workflow │ ├─────────────────────────────────────────────────────────────────────────┤ │ │ │ 1. Alert Received │ │ ───────────────── │ │ ─ GuardDuty finding │ │ ─ Security Hub alert │ │ ─ CloudWatch alarm │ │ │ │ 2. Initial Triage │ │ ───────────────── │ │ Query CloudTrail Lake: │ │ ─ Who: userIdentity (user/role/service) │ │ ─ What: eventName, requestParameters │ │ ─ When: eventTime │ │ ─ Where: sourceIPAddress, awsRegion │ │ │ │ 3. Timeline Analysis │ │ ────────────────── │ │ ─ Events before/after incident │ │ ─ Related API calls from same identity │ │ ─ Same source IP activity │ │ │ │ 4. Scope Assessment │ │ ────────────────── │ │ ─ Which resources were accessed? │ │ ─ What data was potentially exposed? │ │ ─ Are other accounts affected? │ │ │ │ 5. Containment │ │ ───────────── │ │ ─ Disable compromised credentials │ │ ─ Revoke sessions │ │ ─ Isolate affected resources │ │ │ │ 6. Document & Report │ │ ─────────────────── │ │ ─ Export relevant CloudTrail events │ │ ─ Create incident report │ │ ─ Update runbooks │ │ │ └─────────────────────────────────────────────────────────────────────────┘
Best Practices
Enable Multi-Region Attackers may target regions you don’t actively use
Log Validation Enable log file integrity validation for forensics
Use Lake CloudTrail Lake enables SQL queries without S3/Athena setup
Enable Insights Detect unusual API activity patterns automatically
Security Checklist Logging Configuration : ✓ Multi-region trail enabled ✓ Global service events included ✓ Log file validation enabled ✓ S3 bucket encrypted with KMS ✓ Data events for sensitive S3 buckets ✓ Lambda invocation logging ✓ CloudTrail Insights enabled Monitoring : ✓ CloudWatch alarms for critical events ✓ EventBridge rules for real-time alerts ✓ Root account usage monitoring ✓ Failed authentication tracking Retention : ✓ S3 lifecycle policies configured ✓ CloudTrail Lake for long-term analysis ✓ Compliance requirements met
🎯 Interview Questions
Q1: What's the difference between management events and data events?
Management Events (Control Plane):
Operations on AWS resources
CreateBucket, DeleteUser, StartInstances
Logged by default
Lower volume
Data Events (Data Plane):
Operations on data within resources
GetObject, PutObject, Invoke
Not logged by default (extra cost)
High volume
Q2: How would you investigate a potential security breach using CloudTrail?
Identify compromised credentials : Find the access key or roleTimeline analysis : Query all events from that identitySource IP analysis : Check for unusual IPs or geolocationsResource access : Identify what was accessed/modifiedLateral movement : Check for AssumeRole or CreateAccessKeyScope assessment : Determine if other accounts affected Use CloudTrail Lake SQL queries for efficient analysis.
Q3: How do you ensure CloudTrail log integrity?
Enable log file validation : Creates digest files with hashesKMS encryption : Encrypt logs at restS3 bucket policy : Prevent deletion/modificationObject Lock : Enable S3 Object Lock for immutabilityCross-account : Store logs in separate security accountMFA Delete : Require MFA for log deletion Hands-On Lab
Create Organization Trail
Set up multi-region trail with encryption and validation
Enable Data Events
Configure S3 and Lambda data event logging
Set Up CloudTrail Lake
Create event data store and run sample queries
Create Alarms
Set up CloudWatch alarms for root account usage
Investigate
Use CloudTrail Lake to investigate a simulated incident
Next Module
AWS Config Configuration compliance and change management
