Table of Contents

Problem Statement

Modern cloud infrastructure requires secure, collaborative, and automated deployment pipelines. Traditional approaches present several critical challenges:

State Management Issues: Terraform state files contain sensitive data and must be stored securely with proper locking mechanisms to prevent corruption during concurrent operations. Local state storage is unsuitable for team environments and lacks disaster recovery capabilities.

Credential Security: Storing long-lived AWS access keys in CI/CD systems creates significant security risks. Keys can be leaked, compromised, or misused. Traditional credential rotation is manual, error-prone, and often neglected.

Cost Concerns: As a self-funded learning project preparing for AWS Security Specialty certification, minimizing costs while maintaining production-grade quality is essential. Many solutions over-provision resources or use expensive managed services unnecessarily.

Audit and Compliance: Security infrastructure must provide clear audit trails, versioning, and the ability to rollback changes. Without proper controls, tracking who made what changes becomes impossible.

This project establishes a secure, cost-effective foundation for AWS infrastructure that eliminates stored credentials, provides robust state management, and enables automated deployments through GitLab CI/CD.

Solution Architecture

High-Level Design

The architecture implements a secure Terraform state backend using AWS S3 for storage and DynamoDB for state locking, integrated with GitLab CI/CD through OpenID Connect (OIDC) federation for credential-less authentication.

graph TB
    subgraph "GitLab CI/CD"
        A[Pipeline Trigger] --> B[OIDC Token Generation]
    end

    subgraph "AWS Account"
        C[IAM OIDC Provider<br/>gitlab.com]
        D[IAM Role<br/>devops-operator]
        E[S3 Bucket<br/>terraform-state]
        F[DynamoDB Table<br/>state-locks]
    end

    B -->|AssumeRoleWithWebIdentity| C
    C -->|Trust Policy| D
    D -->|Read/Write| E
    D -->|Lock/Unlock| F

    E -->|Versioned State Files| G[State History]
    E -->|Lifecycle Policy| H[STANDARD_IA<br/>after 30 days]
    F -->|LockID Key| I[Concurrent<br/>Protection]

    style C fill:#ff9900
    style D fill:#ff9900
    style E fill:#569a31
    style F fill:#527fff

Key Components

1. Terraform State Backend (S3)

• Encrypted storage using AWS-managed AES256 encryption
• Versioning enabled for complete audit trail and rollback capability
• Public access completely blocked via bucket policies
• Intelligent lifecycle management: transitions to STANDARD_IA after 30 days, expires after 90 days
• Auto-generated bucket names preventing naming conflicts

2. State Locking (DynamoDB)

• Pay-per-request billing mode (no idle costs)
• Required “LockID” attribute for Terraform integration
• Prevents concurrent state modifications
• Immediate consistency for lock operations

3. OIDC Authentication

• Zero stored credentials in GitLab or local environments
• Temporary STS credentials (1-hour validity)
• IAM role restricted by GitLab project path
• Automatic credential rotation with each pipeline run

4. GitLab CI/CD Pipeline

• Automated authentication using OIDC tokens
• Test stage validates AWS connectivity
• Foundation for future Terraform automation stages
• Infrastructure as Code principles applied to deployment pipeline

Technical Implementation

Infrastructure as Code

Terraform State Backend (terraform/state-backend/):

The state backend module creates a secure S3 bucket with comprehensive security controls:

# S3 Bucket with encryption and versioning
resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-state-${var.aws_account_id}-${var.aws_region}"

  lifecycle {
    prevent_destroy = false  # Set to true in production
  }
}

# Server-side encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"  # Using AWS-managed keys for cost savings
    }
  }
}

# Block all public access
resource "aws_s3_bucket_public_access_block" "terraform_state" {
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Cost Optimization through Lifecycle Policies:

resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
  rule {
    id     = "archive-old-versions"
    status = "Enabled"

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "STANDARD_IA"  # 50% cost reduction
    }

    noncurrent_version_expiration {
      noncurrent_days = 90  # Automatic cleanup
    }
  }
}

Security Controls

1. Encryption

• At Rest: AES256 server-side encryption for all S3 objects
• In Transit: TLS enforced for all S3 and DynamoDB operations
• Key Management: AWS-managed keys (future: customer-managed KMS keys)

2. Access Control

• IAM Role: Least-privilege permissions scoped to state operations only
• Path Restriction: Role trust policy limited to specific GitLab project path
• No Static Credentials: OIDC eliminates long-lived access keys
• Temporary Credentials: 1-hour validity with automatic rotation

3. Audit Trail

• S3 Versioning: Complete history of all state changes
• CloudTrail Integration: API calls logged (AWS account-level)
• Git History: Infrastructure code changes tracked in version control
• Pipeline Logs: GitLab CI/CD execution history

4. Operational Security

• State Locking: Prevents concurrent modifications and state corruption
• Rollback Capability: S3 versioning enables recovery from errors
• Disaster Recovery: Cross-region replication possible (future enhancement)
• Cost Monitoring: CloudWatch alarms for unexpected resource creation

OIDC Authentication Flow

GitLab CI/CD Pipeline (.gitlab-ci.yml):

test-aws-auth:
  stage: test
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    - export AWS_ROLE_ARN="arn:aws:iam::ACCOUNT_ID:role/devops-operator"
    - |
      export $(aws sts assume-role-with-web-identity \
        --role-arn ${AWS_ROLE_ARN} \
        --role-session-name "gitlab-${CI_PROJECT_PATH_SLUG}-${CI_PIPELINE_ID}" \
        --web-identity-token ${GITLAB_OIDC_TOKEN} \
        --duration-seconds 3600 \
        --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
        --output text)

  script:
    - aws sts get-caller-identity

IAM Trust Policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/gitlab.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "gitlab.com:aud": "https://gitlab.com"
      },
      "StringLike": {
        "gitlab.com:sub": "project_path:username/aws-sec:*"
      }
    }
  }]
}

The project_path condition ensures only this specific GitLab project can assume the role, preventing unauthorized access even if the OIDC token is compromised.

Challenges & Solutions

Challenge 1: Bootstrapping the State Backend

Problem: Terraform state backends require initial local state creation, creating a “chicken and egg” problem. The state backend must be created before it can store its own state.

Solution:

1. Initial state backend deployment uses local state
2. After S3 and DynamoDB are created, backend configuration is added
3. terraform init -migrate-state moves local state to remote backend
4. Local state file deleted for security

Lesson Learned: Always document bootstrap procedures. Infrastructure foundation requires special handling that differs from application infrastructure.

Challenge 2: OIDC Configuration Complexity

Problem: OIDC trust policies have subtle requirements. Incorrect configuration fails silently or provides cryptic error messages. The relationship between GitLab’s token claims and AWS trust policies isn’t immediately obvious.

Solution:

• Studied AWS Security Specialty content on federation and OIDC
• Used CloudTrail logs to debug failed authentication attempts
• Created test pipeline with verbose logging
• Documented exact trust policy format for future reference

Lesson Learned: Security controls must be tested incrementally. Build authentication first, then add authorization, then add business logic.

Challenge 3: Cost Management

Problem: While preparing for AWS certifications on a limited budget, needed to balance production-grade practices with cost consciousness. S3 storage costs, DynamoDB reads/writes, and data transfer all add up.

Solution:

• S3 lifecycle policies automatically move old state versions to cheaper storage
• DynamoDB PAY_PER_REQUEST instead of provisioned capacity
• AWS-managed encryption keys instead of KMS (saving $1/month per key)
• Aggressive version expiration (90 days) since state can be reconstructed

Result: Infrastructure costs less than $1/month while maintaining production-grade security and reliability.

Lesson Learned: Cost optimization doesn’t mean compromising quality. Smart architecture choices and lifecycle policies enable both.

Results & Metrics

Infrastructure Metrics

Deployment Success:

• ✅ State backend deployed successfully on first apply
• ✅ GitLab CI/CD pipeline authenticated to AWS without stored credentials
• ✅ Terraform operations (init, plan, apply) working with remote backend
• ✅ State locking preventing concurrent modifications

Security Posture:

• ✅ Zero long-lived credentials stored in GitLab
• ✅ All S3 objects encrypted at rest
• ✅ Public access completely blocked
• ✅ OIDC trust policy restricting access to single GitLab project
• ✅ Audit trail through S3 versioning and CloudTrail

Cost Optimization:

• Estimated Monthly Cost: < $1
  • S3 storage: ~$0.02 for small state files
  • S3 requests: ~$0.01 for typical operations
  • DynamoDB: ~$0.25 per million requests (minimal usage)
• Cost Savings: ~95% vs traditional KMS + provisioned DynamoDB approach
• Lifecycle Policy Impact: Old state versions archived to STANDARD_IA (50% savings)

Learning Outcomes

AWS Security Specialty Concepts Applied:

• ✅ IAM roles and federated identity (OIDC)
• ✅ S3 security best practices (encryption, versioning, access control)
• ✅ Infrastructure security through least privilege
• ✅ Cost-aware security architecture
• ✅ Audit and compliance mechanisms

DevSecOps Skills Demonstrated:

• Infrastructure as Code with Terraform
• CI/CD pipeline integration (GitLab)
• Secure credential management (OIDC)
• Git-based workflow for infrastructure
• Production-grade documentation

Key Takeaways

1. OIDC Federation Eliminates Credential Risk

Transitioning from stored credentials to OIDC federation dramatically improved security posture. No long-lived keys to rotate, leak, or compromise. Temporary credentials expire automatically. This pattern applies to any CI/CD system (GitHub Actions, CircleCI, etc.).

2. Cost-Conscious ≠ Low Quality

Production-grade infrastructure doesn’t require expensive managed services. Smart architecture choices—lifecycle policies, pay-per-request billing, AWS-managed encryption—enable both quality and cost efficiency. Essential for personal projects and early-stage startups.

3. Documentation is Infrastructure

Comprehensive README generation (automated with Documentation Agent) transformed a technical project into a portfolio piece. Well-documented infrastructure demonstrates professionalism and attention to detail—critical for job applications and team collaboration.

4. State Management is Critical

Proper Terraform state management prevents countless operational headaches. Remote state with locking isn’t optional for serious infrastructure. This foundation enables team collaboration and prevents state corruption.

5. Security Layers Build on Each Other

Security is additive: encryption + access control + audit + least privilege = defense in depth. No single control provides complete security. Each layer increases attacker cost and reduces risk.

Future Enhancements

Phase 2: Infrastructure Expansion (Planned)

AWS Security Services:

• AWS GuardDuty for threat detection
• Security Hub for centralized findings
• AWS Config for compliance monitoring
• CloudTrail with S3 logging and analysis
• VPC with public/private subnet architecture

Enhanced State Backend:

• Customer-managed KMS keys for S3 encryption
• Cross-region S3 replication for disaster recovery
• S3 access logging for enhanced audit trail
• MFA delete requirement for production state

Phase 3: CI/CD Pipeline Maturity

Automated Terraform Workflow:

• terraform plan on merge requests
• Automated security scanning (tfsec, checkov)
• Cost estimation with Infracost
• Automated testing with Terratest
• Deployment approvals for production

Security Automation:

• SAST integration for Infrastructure as Code
• Secrets scanning in git commits
• Policy-as-Code validation (OPA/Sentinel)
• Automated compliance reporting

Phase 4: Multi-Environment Strategy

Workspace Management:

• Separate Terraform workspaces for dev/staging/prod
• Environment-specific state files
• Progressive deployment strategy
• Blue-green infrastructure deployments

Resources

Project Links

• GitHub Repository: hmbldv/aws-sec
• GitLab Repository (Private): username/aws-sec
• Documentation: Comprehensive README in repository

AWS Services Documentation

• AWS IAM OIDC Identity Providers
• S3 Best Practices for Security
• Terraform S3 Backend
• GitLab CI/CD OIDC

Related Blog Posts

• Coming Soon: “Building Secure Terraform Pipelines with OIDC”
• Coming Soon: “AWS Security Specialty Exam Prep: OIDC Federation”

---

This project demonstrates proficiency in: AWS Security Architecture • Infrastructure as Code • DevSecOps Practices • CI/CD Automation • Cost Optimization • Technical Documentation

Built while preparing for AWS Certified Security - Specialty certification