CloudTrail Investigation: AWS Security Incident Response Guide

Investigating an AWS Security Incident With CloudTrail

When an AWS security incident occurs, time is the most scarce resource. Security engineers and SREs often face a deluge of unstructured log data, making it difficult to distinguish between benign configuration drift and active exploitation during a cloudtrail investigation. The core mechanism of AWS audit logging, CloudTrail, records every API call made within an account. However, viewing these logs sequentially in the CloudTrail console is inefficient for forensic analysis. Instead, the most effective investigation strategy treats CloudTrail data as a structured dataset, accessible via Amazon Athena, allowing for precise SQL-based querying of the attack chain.

The Mechanism of Trust and Event Generation

CloudTrail operates by intercepting requests sent to AWS services and writing them to a designated S3 bucket. Every event contains a userIdentity object, which is the critical artifact for tracking who performed an action. This object includes the type (e.g., IAMUser, AssumedRole, Root), the arn (Amazon Resource Name), and the accessKeyId [1]. When an attacker compromises an IAM user’s access key, they generate events that appear identical to legitimate traffic in terms of structure, but differ in context—such as unusual geographic locations, unexpected times, or access to sensitive services like iam:CreateAccessKey or s3:GetObject on restricted buckets.

Management Events, which cover operations related to the management of AWS resources (like creating an EC2 instance or modifying an IAM policy), are enabled by default in all regions [2]. Data Events, which record resource-level operations (like reading an S3 object or invoking a Lambda function), must be explicitly enabled [3]. For incident response, Management Events are the starting point because they reveal the attacker’s intent to move laterally or escalate privileges.

[1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html [2] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html [3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html

Querying the Noise with Athena

The CloudTrail console filters logs based on predefined templates, which often miss subtle indicators of compromise. By syncing CloudTrail logs to an Athena table, investigators can write custom queries to isolate suspicious activity. The first step in any investigation is to identify anomalies in the userIdentity field. For example, to find all API calls made by a specific compromised user, you would query the eventSource and eventName fields [4].

Consider a scenario where an attacker uses a stolen access key to create new credentials. A basic Athena query to detect this looks like:

SELECT 
    eventTime,
    eventName,
    sourceIPAddress,
    userIdentity.arn,
    requestParameters
FROM cloudtrail_logs
WHERE eventName = 'CreateAccessKey'
    AND userIdentity.type = 'IAMUser'
    AND userIdentity.arn LIKE '%compromised-user%'
ORDER BY eventTime DESC;

This query returns the exact moment the attacker attempted to establish persistence. The sourceIPAddress field is particularly useful for identifying geographic anomalies. If a user typically logs in from us-east-1 but suddenly generates events from ap-northeast-1, this is a strong indicator of compromise.

[4] https://docs.aws.amazon.com/athena/latest/ug/querying-cloudtrail-data.html

Tracing Lateral Movement

Once an attacker has initial access, they often attempt to escalate privileges by assuming roles with broader permissions. The sts:AssumeRole API call is a common mechanism for this [5]. By analyzing CloudTrail logs for AssumeRole events, investigators can map the attacker’s path through the IAM hierarchy. A critical aspect of this analysis is examining the requestParameters.roleArn field to see which roles were targeted.

For instance, an attacker might start with a low-privilege IAM user and then assume an administrative role. The following Athena query helps identify such transitions:

SELECT 
    eventTime,
    userIdentity.arn AS initiator,
    requestParameters.roleArn AS targetRole,
    sourceIPAddress
FROM cloudtrail_logs
WHERE eventName = 'AssumeRole'
    AND userIdentity.type = 'IAMUser'
ORDER BY eventTime DESC;

This query reveals the relationship between the initial compromised identity and the elevated roles accessed. If the targetRole is an administrative role that was not previously assumed by the initiator, this indicates a privilege escalation attempt. Additionally, checking for GetCallerIdentity calls can help confirm whether the attacker successfully adopted the new identity, as this API call returns the current caller’s ARN [6].

[5] https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html [6] https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html

Containment and Runbooks

Detection is only half the battle; containment is equally critical. Once suspicious activity is identified, immediate action must be taken to prevent further damage. The primary containment step is to revoke the compromised credentials. For IAM users, this involves disabling the access key or deleting the user. For temporary credentials obtained via AssumeRole, the session cannot be directly revoked, but the underlying IAM role can have its trust policy modified to prevent further assumption, or the IAM user who assumed the role can be disabled.

Automating this process is ideal. Using AWS Lambda triggered by Amazon EventBridge, you can create a runbook that automatically disables an IAM user when a high-confidence alert is generated. For example, if an Athena query detects a CreateAccessKey event from an unknown IP address, an EventBridge rule can trigger a Lambda function that calls iam:UpdateAccessKey [7]. This reduces the Mean Time to Respond (MTTR) from hours to seconds.

[7] https://docs.aws.amazon.com/IAM/latest/APIReference/API_UpdateAccessKey.html

Conclusion

Investigating AWS security incidents requires a shift from passive log viewing to active data analysis. By leveraging CloudTrail’s structured data and Athena’s query capabilities, security engineers can rapidly identify credential compromises, trace lateral movement, and execute containment runbooks. This approach not only improves detection accuracy but also ensures that response actions are timely and effective, minimizing the impact of security breaches.

Common Pitfalls

1. Ignoring Data Events: Relying solely on Management Events misses activity within specific resources like S3 buckets or DynamoDB tables, where attackers often exfiltrate data. Always enable Data Events for critical storage and compute resources.
2. Not Filtering by Region: Attackers may operate across multiple regions. Failing to query across all enabled CloudTrail trails can leave gaps in the timeline, allowing attackers to hide their tracks in less-monitored regions.
3. Assuming IAM User Keys Are the Only Vector: Attackers frequently use temporary credentials via AssumeRole or federated identities. Focusing only on long-term IAM user access keys may lead investigators to miss the initial foothold established through role assumption.

Practical Takeaways

1. Assume All Logs Are Potentially Compromised: Never trust the integrity of logs without verifying the source. Ensure CloudTrail logs are delivered to an immutable S3 bucket with strict access controls to prevent tampering.
2. Query by Identity First, Then Resource: Start investigations by isolating the userIdentity.arn to understand the scope of the compromised identity before drilling down into specific resource actions.
3. Automate Containment for Known Patterns: For high-confidence indicators of compromise, such as CreateAccessKey from unknown IPs, automate containment via Lambda and EventBridge to reduce response time from minutes to seconds.