Day 22: Cloud Incident Response - Investigating Compromised AWS Workloads

Why Cloud IR Is Different

Traditional incident response assumed you could physically touch the affected machine - pull the hard drive, image it, run forensics. In the cloud:

• Instances can be terminated and their storage wiped in seconds
• Evidence is spread across CloudTrail, VPC Flow Logs, GuardDuty, application logs - in different services
• Actions happen via API calls, not keyboard input
• The “attacker” may be an IAM key operating autonomously long after the initial breach
• Blast radius can span multiple accounts and regions within minutes

Cloud IR requires different procedures, different tools, and speed above all else.

---

The IR Lifecycle in AWS

Detection → Analysis → Containment → Eradication → Recovery → Lessons Learned

Detection

• GuardDuty finding fires
• Security Hub alert
• CloudTrail anomaly (unusual API calls, new IAM user created)
• Application alert (spike in errors, auth failures)
• External notification (AWS Abuse report, bug bounty)

Analysis

• Understand the scope: what was accessed, what was changed, what was exfiltrated?
• Who is the identity involved?
• What resources are affected?
• Is the attack ongoing?

Containment

• Stop the bleeding - revoke credentials, isolate instances, block IPs
• Preserve evidence - snapshot EBS volumes, export logs before they expire

Eradication

• Remove the attacker’s access and any persistence they established
• Patch the vulnerability that allowed entry

Recovery

• Restore from known-good snapshots or redeploy from IaC
• Monitor closely post-recovery for re-entry attempts

Lessons Learned

• Root cause analysis
• Update runbooks, detection rules, and preventive controls

---

Scenario 1: Compromised IAM Access Key

This is the most common cloud incident - a key leaked in a GitHub repo, a .env file exposed, or malware exfiltrating credentials from an EC2 instance.

Step 1: Detect

GuardDuty finding: UnauthorizedAccess:IAMUser/MaliciousIPCaller or CredentialAccess:IAMUser/AnomalousBehavior

Or you notice it manually - someone emails you a “your AWS key is in this public repo” notification.

Step 2: Identify the Key

# Find the access key ID mentioned in the alert or repo
# Format: AKIA... (long-term) or ASIA... (temporary STS)

# Find which IAM user/role owns it
aws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE

Step 3: Preserve Evidence First

Before touching anything, export relevant CloudTrail events:

# Export all API calls made with this key in the last 24 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time $(date -d '24 hours ago' --iso-8601=seconds) \
  --output json > compromised-key-events.json

Step 4: Contain - Revoke the Key Immediately

# Deactivate the key (reversible)
aws iam update-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive \
  --user-name affected-user

# Delete the key (irreversible: do this after you've saved evidence)
aws iam delete-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --user-name affected-user

Step 5: Check for Attacker Persistence

The attacker may have created resources to maintain access even after the key is revoked:

# Check for new IAM users created recently
aws iam list-users --query 'Users[?CreateDate>`2026-05-01`]'

# Check for new access keys on all users
aws iam list-users --query 'Users[*].UserName' --output text | \
  xargs -I{} aws iam list-access-keys --user-name {}

# Check for new roles with external trust
aws iam list-roles --query \
  'Roles[?AssumeRolePolicyDocument.Statement[?Principal.AWS!=`null`]]'

# Check for new Lambda functions (possible backdoor)
aws lambda list-functions --query 'Functions[?LastModified>`2026-05-01`]'

# Check for EC2 instances launched recently
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[?LaunchTime>`2026-05-01`]'

Step 6: Revoke All Active Sessions

Even after deleting the key, existing STS sessions derived from it may still be valid (up to their expiry):

# Attach a policy to the user that explicitly denies everything
aws iam put-user-policy \
  --user-name affected-user \
  --policy-name DenyAll \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*"}]}'

For roles - update the trust policy to remove external trust or add a condition that denies all.

---

Scenario 2: Compromised EC2 Instance

GuardDuty finding: Backdoor:EC2/C&CActivity.B or CryptoCurrency:EC2/BitcoinTool.B - an instance is communicating with known bad IPs or running a miner.

Step 1: Preserve Evidence

Before isolating the instance, capture its current state:

INSTANCE_ID="i-0abc123def456"

# Create a snapshot of each attached EBS volume
aws ec2 describe-volumes \
  --filters "Name=attachment.instance-id,Values=$INSTANCE_ID" \
  --query 'Volumes[*].VolumeId' \
  --output text | \
  xargs -I{} aws ec2 create-snapshot \
    --volume-id {} \
    --description "IR snapshot $INSTANCE_ID $(date +%Y%m%d)"

Step 2: Isolate the Instance (Quarantine Security Group)

Create a quarantine security group that blocks all traffic:

# Create quarantine SG: no inbound or outbound rules
QUARANTINE_SG=$(aws ec2 create-security-group \
  --group-name "quarantine-ir-$(date +%Y%m%d)" \
  --description "Incident Response Quarantine - blocks all traffic" \
  --vpc-id $VPC_ID \
  --query 'GroupId' \
  --output text)

# Replace ALL security groups on the instance with the quarantine SG
aws ec2 modify-instance-attribute \
  --instance-id $INSTANCE_ID \
  --groups $QUARANTINE_SG

The instance is now isolated - no inbound, no outbound. It’s running but can’t communicate.

Step 3: Revoke the Instance’s IAM Role Credentials

The IAM role on the instance may have been used maliciously. Revoke active sessions:

# Find the instance profile / role
ROLE_NAME=$(aws ec2 describe-instances \
  --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' \
  --output text | cut -d'/' -f2)

# Revoke all active sessions for this role
aws iam put-role-policy \
  --role-name $ROLE_NAME \
  --policy-name DenyAll-IR \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*","Condition":{"DateLessThan":{"aws:TokenIssueTime":"2026-06-11T12:00:00Z"}}}]}'

This denies any API call from sessions issued before the specified time - killing all existing sessions derived from the role without affecting new legitimate ones.

Step 4: Forensic Analysis

With the instance isolated, connect for analysis via SSM (if SSM agent is running):

aws ssm start-session --target $INSTANCE_ID

Or mount the EBS snapshot to a forensic EC2 instance:

# Launch a forensic instance in an isolated VPC
# Attach the snapshot as a volume
aws ec2 attach-volume \
  --volume-id vol-snapshot-id \
  --instance-id forensic-instance-id \
  --device /dev/sdf

What to look for:

# Suspicious processes
ps auxf | grep -v "^\[" | sort -k3 -rn | head -20

# Cron persistence
crontab -l
cat /etc/cron* 2>/dev/null

# Recently modified files
find / -newer /tmp -type f 2>/dev/null | grep -v proc | head -50

# Bash history
cat ~/.bash_history
cat /home/*/.bash_history

# Network connections at time of incident (from VPC flow logs)
# Query from Athena or CloudWatch Logs

---

CloudTrail Forensic Queries

During any incident, Athena is your best tool for querying CloudTrail at scale:

-- What did the compromised identity do?
SELECT eventtime, eventname, sourceipaddress, requestparameters, errorcode
FROM cloudtrail_logs
WHERE useridentity.accesskeyid = 'AKIAIOSFODNN7EXAMPLE'
  AND eventtime > '2026-06-10'
ORDER BY eventtime;

-- Were any IAM changes made?
SELECT eventtime, eventname, requestparameters, useridentity.arn
FROM cloudtrail_logs
WHERE eventsource = 'iam.amazonaws.com'
  AND eventtime > '2026-06-10'
  AND errorcode IS NULL
ORDER BY eventtime;

-- Any new resources created (possible persistence)?
SELECT eventtime, eventname, awsregion, requestparameters
FROM cloudtrail_logs
WHERE eventname IN ('CreateUser','CreateRole','RunInstances','CreateFunction20150331','CreateBucket')
  AND eventtime > '2026-06-10'
ORDER BY eventtime;

-- Console logins from unusual IPs
SELECT eventtime, useridentity.username, sourceipaddress, additionaleventdata
FROM cloudtrail_logs
WHERE eventname = 'ConsoleLogin'
  AND eventtime > '2026-06-01'
  AND sourceipaddress NOT LIKE '10.%'
ORDER BY eventtime DESC;

---

IR Runbook Template

Document your IR procedures as runbooks - executable checklists that any team member can follow during an incident.

## Runbook: Compromised IAM Access Key

**Severity:** P1 - drop everything
**Detection source:** GuardDuty / CloudTrail alert / external report

### Immediate Actions (first 15 minutes)
- [ ] Identify the key ID from the alert
- [ ] Run: `aws iam get-access-key-last-used --access-key-id <KEY_ID>`
- [ ] Export CloudTrail events for the key (last 24h)
- [ ] Deactivate the key: `aws iam update-access-key --status Inactive`
- [ ] Notify security team and management

### Investigation (first hour)
- [ ] Review all API calls made with the key
- [ ] Check for new IAM users, roles, access keys created
- [ ] Check for new EC2, Lambda, RDS instances launched
- [ ] Check for data exfiltration (S3 GetObject on sensitive buckets)
- [ ] Identify how the key was exposed (git, logs, env var)

### Containment
- [ ] Delete the key (after evidence preserved)
- [ ] Add DenyAll inline policy to revoke active sessions
- [ ] Delete any attacker-created IAM users/roles/keys
- [ ] Terminate any attacker-launched resources

### Recovery
- [ ] Issue new credentials to the legitimate user
- [ ] Rotate any secrets that may have been accessed
- [ ] Verify system integrity of any instances the key touched

### Post-Incident
- [ ] Root cause written up
- [ ] Detection gap identified and new GuardDuty/Config rule added
- [ ] Runbook updated with lessons learned

---

Key Takeaways

• Speed is everything - the longer a compromised key is active, the more damage is done
• Preserve evidence before containment - snapshots and CloudTrail exports first
• Quarantine security group is the cleanest EC2 isolation technique - fast and reversible
• The DenyAll inline policy with a DateLessThan condition revokes existing STS sessions without breaking new legitimate access
• Always check for attacker persistence - new IAM users, Lambda backdoors, cron jobs
• Athena + CloudTrail is your forensic platform - practice the queries before you need them
• Runbooks must be written before an incident - not during one

---

References

• AWS Security Incident Response Guide
• Managing IAM Access Keys
• AWS IR Playbooks (GitHub)

---

You can find me online at:

• GitHub: Mhdomer
• LinkedIn: mhd3omar
• Tryhackme: nonlouy