Table of Contents

Problem Statement

As organizations scale their AWS footprint across multiple accounts, maintaining security compliance and consistent configuration standards becomes increasingly complex. Security teams struggle with:

1. Visibility Gaps: No centralized view of resource configurations across 4+ AWS accounts (production, security tools, log archive, container services)
2. Compliance Monitoring: Manual audits are time-consuming and error-prone
3. Security Testing: Lack of safe, isolated environments for penetration testing and vulnerability research
4. Credential Management: Long-lived AWS access keys in CI/CD pipelines create security risks
5. Development Velocity: Remote state backends with locking mechanisms can impede rapid iteration for solo developers

This project establishes a production-grade security foundation that addresses these challenges through automated compliance monitoring, hands-on security testing infrastructure, and credential-less deployments.

Solution Architecture

This infrastructure implements a comprehensive security platform with three core components: organization-wide compliance monitoring via AWS Config, isolated security testing lab environments, and secure CI/CD automation.

Architecture Diagram

graph TB
    subgraph "AWS Organization"
        subgraph "Account: squinks (266735821834)"
            CR1[Config Recorder]
            LAB[Security Lab]
            S3C[S3: Config Data]
            S3M[S3: Macie Findings]

            subgraph "Security Lab"
                ATK[Attacker Instance<br/>Ubuntu + Kali Tools]
                TGT[Target Instance<br/>Vulnerable Services]
                ATK -->|All Ports| TGT
            end

            CR1 --> S3C
        end

        subgraph "Account: sec-tools"
            CR2[Config Recorder]
        end

        subgraph "Account: log-archive"
            CR3[Config Recorder]
        end

        subgraph "Account: container-services"
            CR4[Config Recorder]
        end

        CR1 --> AGG[Config Aggregator<br/>Organization-Wide]
        CR2 --> AGG
        CR3 --> AGG
        CR4 --> AGG

        AGG --> S3C
    end

    subgraph "GitLab CI/CD"
        GL[GitLab Pipeline]
        GL -->|OIDC Token| OIDC[IAM OIDC Provider]
        OIDC -->|AssumeRole| ROLE[DevOps Operator Role]
        ROLE -->|Terraform Apply| CR1
    end

    subgraph "Security Monitoring"
        MACIE[AWS Macie]
        MACIE --> S3M
        AGG -->|Compliance Data| DASH[Compliance Dashboard]
    end

    ADM[Admin Workstation] -->|SSH/RDP/SSM| LAB

    style AGG fill:#ff9900
    style LAB fill:#00ff00
    style OIDC fill:#0066ff
    style S3C fill:#ff6600

Key Components

1. AWS Config Aggregator: Centralized compliance monitoring across all AWS accounts and regions, providing a unified view of resource configurations and compliance status
2. Config Recorders: Deployed in each AWS account to continuously track resource changes and configuration drift
3. Security Testing Lab: Isolated EC2 environment with Kali Linux tools (attacker) and intentionally vulnerable services (target) for hands-on security research
4. GitLab CI/CD with OIDC: Credential-less deployment pipeline using OpenID Connect federation, eliminating long-lived AWS access keys
5. IAM Security Roles: Least-privilege roles for SecurityAnalyst and OrganizationAdmin with cross-account access capabilities
6. S3 Data Lakes: Encrypted storage for Config snapshots (7-year retention) and Macie sensitive data findings

Technical Implementation

Infrastructure as Code

All infrastructure is defined in Terraform with a modular architecture:

• config-aggregator.tf: Organization-wide Config aggregator with IAM service role
• config-recorder.tf: Config recorder with AWS managed policy and inline permissions
• security-lab.tf: EC2 instances with user-data scripts for Kali tools installation
• iam-policies.tf: Security analyst and organization admin roles with least-privilege policies
• s3-buckets.tf: Config and Macie buckets with lifecycle policies and encryption
• oidc.tf: GitLab OIDC identity provider and DevOps operator role

State Management: Migrated from S3/DynamoDB remote backend to local state for simplified iteration and rapid development cycles. This eliminated state locking overhead while maintaining infrastructure tracking.

Security Controls

Multi-Layer Security Architecture:

• IAM Least Privilege: Config recorder uses AWS managed policy (AWS_ConfigRole) plus granular inline policies
• Encryption at Rest: All S3 buckets use AES256 server-side encryption with bucket key optimization
• Network Isolation: Security lab instances run in isolated security groups with explicit allow rules
• IMDSv2 Enforcement: Metadata service v2 required on all EC2 instances to prevent SSRF attacks
• Public Access Blocking: S3 bucket policies explicitly deny unencrypted transport and public access
• EBS Encryption: All volumes encrypted at rest with AWS-managed keys
• Session Manager Access: SSM Session Manager enabled for secure shell access without SSH keys
• S3 Lifecycle Policies:
  • Config data: Standard → Standard-IA (90 days) → Glacier (365 days) → Expire (7 years)
  • Macie findings: Standard → Standard-IA (30 days) → Glacier (90 days) → Expire (1 year)

Automation & CI/CD

GitLab Pipeline with OIDC Federation:

# .gitlab-ci.yml
test-aws-auth:
  stage: test
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    - export AWS_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/devops-operator"
    - export AWS_DEFAULT_REGION="us-west-1"
    - >
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn ${AWS_ROLE_ARN}
      --role-session-name "gitlab-${CI_PROJECT_PATH_SLUG}-${CI_PIPELINE_ID}"
      --web-identity-token ${GITLAB_OIDC_TOKEN}
      --duration-seconds 3600
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text))

  script:
    - aws sts get-caller-identity

Benefits:

• Zero credential storage (no AWS access keys in GitLab variables)
• Short-lived STS credentials (1-hour expiration)
• Project-path restricted trust policy
• Automatic credential rotation

Challenges & Solutions

Challenge 1: GitLab CI/CD Pipeline Failures

Problem: Pipeline failed with error Request ARN is invalid during OIDC authentication. The role ARN construction failed because AWS_ACCOUNT_ID variable was not set in GitLab CI/CD settings, resulting in an invalid ARN: arn:aws:iam::/role/devops-operator.

Root Cause: Missing environment variable in GitLab CI/CD configuration. The pipeline attempted to assume a role with an incomplete ARN.

Solution:

1. Navigate to GitLab Project → Settings → CI/CD → Variables
2. Add variable: AWS_ACCOUNT_ID = 266735821834
3. Ensure variable is not protected (to allow use on all branches)
4. Re-run pipeline

Lesson Learned: When implementing OIDC federation, always verify that required environment variables are configured before pipeline execution. Missing variables can cause cryptic authentication errors that appear to be permission issues rather than configuration problems.

Challenge 2: Config Recorder IAM Policy Warnings

Problem: AWS Config console displayed persistent warning: “Update the IAM Policy used by AWS Config” even after granting comprehensive inline permissions to the Config recorder role.

Root Cause: AWS Config expects either a service-linked role or the official AWS managed policy (AWS_ConfigRole). Custom inline policies alone don’t satisfy Config’s validation checks.

Solution: Attached the AWS managed policy in addition to inline policies:

resource "aws_iam_role_policy_attachment" "config_recorder_managed" {
  role       = aws_iam_role.config_recorder.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWS_ConfigRole"
}

Lesson Learned: AWS services often have specific expectations about IAM policy structure. While inline policies provide flexibility, managed policies ensure compatibility with service-specific validation logic. Use managed policies where they exist, supplemented with inline policies for custom requirements.

Challenge 3: Remote State Backend Overhead

Problem: S3/DynamoDB remote backend added complexity for solo development:

• Required separate state backend infrastructure deployment
• State locking prevented rapid iteration
• Cleanup required deleting versioned S3 objects and DynamoDB tables
• Added ~15 seconds to every Terraform operation

Solution: Migrated to local state management:

1. Downloaded existing state from S3 to preserve resource tracking
2. Deleted S3 object versions and delete markers
3. Removed DynamoDB state lock table
4. Updated providers.tf to use local backend
5. Reinitialized with terraform init

Trade-offs:

• ✅ Faster iteration (45s vs 60s+ Terraform apply)
• ✅ Eliminated state backend infrastructure costs ($1-2/month)
• ✅ Simplified workflow for solo development
• ❌ Lost team collaboration features (state locking, remote access)
• ❌ No automatic state backup (requires manual backup strategy)

Lesson Learned: Remote state backends are valuable for team collaboration but can impede rapid iteration for solo developers. Choose state management based on team size and workflow requirements. For solo projects or rapid prototyping, local state reduces friction while maintaining infrastructure tracking.

Results & Metrics

Compliance Monitoring:

• ✅ Centralized visibility across 4 AWS accounts (squinks, sec-tools, log-archive, container-services)
• ✅ Continuous configuration recording for all supported resource types
• ✅ 7-year retention policy for compliance audits (SOC 2, PCI-DSS requirements)
• ✅ Organization-wide aggregation across all 17 AWS regions
• ✅ Zero IAM policy warnings in AWS Config console

Security Testing Capabilities:

• ✅ Isolated lab environment with Kali Linux tools (Metasploit, Nmap, Hydra, Impacket)
• ✅ Intentionally vulnerable target services for penetration testing practice
• ✅ RDP and SSH access for hands-on security research
• ✅ SSM Session Manager for secure shell access without exposing SSH ports

CI/CD Security & Reliability:

• ✅ Zero long-lived credentials (OIDC federation eliminates access keys)
• ✅ Project-path restricted IAM trust policy prevents unauthorized access
• ✅ Automated Terraform deployments via GitLab pipeline
• ✅ 100% pipeline success rate after configuration fixes

Cost Optimization:

• 💰 EC2 instances stopped when not in use: ~$60/month savings
• 💰 S3 lifecycle policies: ~50% storage cost reduction (Standard-IA transition)
• 💰 Local state management: Eliminated DynamoDB table charges (~$1/month)
• 💰 Estimated monthly cost: ~$15/month (storage only) vs. ~$75/month (running EC2)

Performance Improvements:

• ⚡ Terraform apply time: ~45 seconds (local state) vs. 60+ seconds (remote state)
• ⚡ Config aggregator collection latency: < 15 minutes for new resources
• ⚡ GitLab pipeline execution: ~2 minutes (OIDC authentication + Terraform validation)

Key Takeaways

1. AWS Config Provides Unmatched Visibility: Organization-wide aggregation delivers compliance insights that manual audits cannot match. The ability to query resource configurations across all accounts from a single dashboard reduces audit preparation time from days to minutes.

2. OIDC Federation Eliminates Credential Risks: Removing AWS access keys from CI/CD pipelines eliminates an entire class of credential theft attacks. The initial OIDC setup complexity pays dividends in reduced security risk and simplified credential rotation.

3. State Management Has Trade-offs: Remote state backends enable team collaboration but add overhead for solo development. Local state management provides faster iteration cycles at the cost of collaboration features. Choose based on team size and workflow requirements.

4. AWS Managed Policies Ensure Service Compatibility: While inline policies provide flexibility, AWS managed policies ensure compatibility with service-specific validation logic. Use managed policies where they exist, supplemented with inline policies for custom requirements.

5. Hands-On Security Labs Deepen Understanding: Building a security testing environment with real tools (Kali Linux) and vulnerable services reinforces theoretical knowledge with practical experience. Active exploitation attempts reveal attack vectors that passive learning cannot demonstrate.

6. Infrastructure as Code Enables Reproducibility: The entire security infrastructure can be rebuilt in minutes with terraform apply. This repeatability enables disaster recovery, multi-environment deployments, and portfolio demonstrations without manual configuration.

Future Enhancements

Phase 1: Expand Security Monitoring (Next 30 days)

• AWS GuardDuty threat detection with intelligent alerting
• Security Hub for centralized security findings aggregation
• CloudTrail logging for API call auditing across all accounts

Phase 2: Config Rules & Remediation (Next 60 days)

• AWS Config rules for CIS AWS Foundations Benchmark compliance
• Automated remediation with Lambda functions (e.g., terminate non-compliant instances)
• SNS notifications for compliance violations

Phase 3: Advanced Security Controls (Next 90 days)

• VPC with public/private subnets for network segmentation
• Customer-managed KMS keys for S3 encryption
• VPC endpoints for S3 access (eliminate internet egress)
• AWS Secrets Manager integration for credential rotation

Phase 4: Multi-Account Security Hub (Future)

• Centralized Security Hub aggregator in sec-tools account
• Cross-account finding aggregation from all organization accounts
• Integration with Slack/PagerDuty for real-time security alerts
• Custom Security Hub insights and dashboards

Pipeline Enhancements:

• Terraform plan on merge requests (PR preview environments)
• Automated security scanning with tfsec and checkov
• Cost estimation with Infracost for infrastructure changes
• Automated testing with Terratest for validation

Resources

• Repository: github.com/hmbldv/aws-sec
• GitLab CI/CD: Private GitLab repository for production deployments
• Documentation: Config Aggregator Setup Guide

AWS Services Used:

• AWS Config - Resource configuration recording and compliance
• AWS Config Aggregator - Multi-account aggregation
• AWS IAM Identity Center - SSO authentication
• AWS OIDC - Credential-less CI/CD
• AWS EC2 - Security lab instances
• AWS S3 - Config and Macie data storage
• AWS Macie - Sensitive data discovery
• AWS Systems Manager - Session Manager for secure access

Related Reading:

• CIS AWS Foundations Benchmark
• AWS Security Best Practices
• Terraform AWS Provider Documentation
• GitLab OIDC Authentication

---

This project demonstrates proficiency in AWS security architecture, Infrastructure as Code, compliance automation, DevSecOps CI/CD, multi-account AWS Organizations management, and hands-on penetration testing. Built as part of preparation for AWS Certified Security - Specialty.