AWS Secrets Manager: Managing Secrets Infrastructure Without Managing Secrets

Introduction

Here's a counterintuitive truth about secrets management: the infrastructure that holds secrets is not the same as the secrets themselves. Too many teams learn this the hard way when they embed database passwords in Terraform files, commit API keys to Git, or store sensitive tokens in container images. The result? Leaked credentials, compliance violations, and sleepless nights for security teams.

AWS Secrets Manager exists to solve this problem by providing secure, auditable, and rotatable storage for sensitive data like database credentials, API keys, and certificates. But deploying Secrets Manager itself presents a paradox: you need infrastructure-as-code to create secrets containers, yet you must never put actual secret values in that code.

This document explores how to deploy AWS Secrets Manager across different maturity levels—from manual console clicks to production-grade automation—and explains why Project Planton takes a deliberately minimal approach that separates secret infrastructure from secret data.

The Maturity Spectrum: From Quick Fixes to Production Patterns

Level 0: Manual Console Creation (The Quick Start)

What it is: Using the AWS web console to create secrets with a guided UI. You specify a secret name (like myapp/prod/db-password), paste in a value, choose encryption settings, and click "Store secret."

When it works: This is perfectly fine for:

• Getting started or learning AWS Secrets Manager
• One-off secrets in development environments
• Emergency credential storage when production is down
• Small teams with just a handful of secrets

Why it doesn't scale:

• No audit trail in code: Six months later, no one remembers why that secret exists or who created it
• Inconsistent naming: Different team members use different conventions (app1-prod-db vs prod/app1/database vs database_prod_app1)
• Permission drift: Someone creates a secret but forgets to grant the application IAM role access to it
• No repeatability: Rebuilding infrastructure in a new region means manually recreating all secrets

Common pitfalls: Misconfiguring KMS keys (using a custom key without granting decrypt permissions), choosing non-descriptive names, or accidentally creating secrets with a hyphen plus 6 random characters at the end (which confuses AWS's own ARN suffixing system).

Verdict: Use this for learning and emergencies, not for production systems or any environment that needs to be reproducible.

Level 1: AWS CLI and SDK Automation (The Scripting Phase)

What it is: Automating secret creation using aws secretsmanager create-secret commands or SDK calls in Python (Boto3), Java, or other languages.

The upgrade from Level 0:

• Repeatable: Run the same script to create the same secrets
• Scriptable: Integrate into deployment pipelines
• Version-controllable: Store the script (not the secret values) in Git

Critical security practice: Never pass secret values directly in shell commands like this:

# DANGEROUS - This goes in shell history!
aws secretsmanager create-secret --name app/prod/db-pass --secret-string "supersecret123"

Instead, use file-based input:

# Write to temporary file, use it, then shred it
cat > /tmp/secret.txt << EOF
supersecret123
EOF
aws secretsmanager create-secret --name app/prod/db-pass --secret-string file:///tmp/secret.txt
shred -u /tmp/secret.txt

Why it still falls short for production:

• Requires managing AWS credentials for scripts
• No declarative state management (did the secret already exist? What if the name changed?)
• Secret values still need to come from somewhere secure
• Hard to coordinate secret creation with other infrastructure (databases, applications, etc.)

Verdict: A step up from manual work and useful for migration scripts or one-time population, but not a complete infrastructure solution.

Level 2: Configuration Management Tools (Ansible, SaltStack)

What it is: Using tools like Ansible's community.aws.secretsmanager_secret module or SaltStack's Boto3 integration to manage secrets as part of broader system provisioning.

The value proposition:

• Declarative approach: Define desired secret state (present or absent) in playbooks
• Integrated provisioning: Create a secret, then deploy the application that uses it, all in one workflow
• Centralized execution: Ansible can manage secrets across multiple AWS accounts from a single control node

Example Ansible task:

- name: Ensure production database password exists
  community.aws.secretsmanager_secret:
    name: "myapp/prod/DB_PASSWORD"
    description: "Production database credentials"
    tags:
      Environment: prod
      Application: myapp
    state: present
  # Note: This creates the secret container, not the value

The secret value problem persists: Even with configuration management, you face the same challenge: how do you provide the actual secret value without storing it in the playbook? Common solutions:

• Use Ansible Vault to encrypt the value in the playbook (but then the playbook still contains it, just encrypted)
• Prompt for secrets at runtime (doesn't work in CI/CD)
• Read from an external vault system (adds complexity)

Why it's not the final answer:

• Configuration management tools are designed for system state (packages, files, services), not primarily for infrastructure provisioning
• State tracking is done through the tool's own mechanisms, not AWS-native state
• IAM management for the config management system can get complex

Verdict: Useful if you're already heavily invested in Ansible or Salt for infrastructure, but infrastructure-as-code tools are purpose-built for this.

Level 3: Infrastructure-as-Code (The Production Standard)

What it is: Managing secrets as code using Terraform, Pulumi, CloudFormation, or AWS CDK, with proper separation of secret metadata from secret values.

This is where most production systems land, so it deserves deeper examination.

The Core Pattern: Separate Resource from Value

All production-grade IaC follows this principle:

1. Define the secret resource (name, KMS key, tags, rotation settings) in code
2. Omit the actual secret value from code and state
3. Populate the value separately via CI/CD, external vault, or generated passwords

Terraform/OpenTofu Example:

# Create the secret container
resource "aws_secretsmanager_secret" "db_password" {
  name        = "myapp/prod/db-password"
  description = "Production database master password"
  kms_key_id  = aws_kms_key.secrets.id

  tags = {
    Environment = "prod"
    Application = "myapp"
    ManagedBy   = "terraform"
  }
}

# Note: No aws_secretsmanager_secret_version resource here!
# The value is set outside of Terraform via CI/CD

Why this works:

• Terraform state knows the secret exists (ARN, name, encryption key) but never sees the sensitive value
• IAM policies referencing the secret can be defined in the same Terraform code
• The secret can be recreated in any AWS account/region by running the same code

The state file problem (and solutions):

Historically, if you did include a secret value in Terraform, it would be stored in plaintext in the .tfstate file—a major security risk. Modern solutions:

1. Terraform 1.10+ Ephemeral Values: Mark variables as ephemeral so they're never persisted to state:

ephemeral "aws_secretsmanager_secret" "db_creds" {
  secret_id = "myapp/prod/db-password"
}

# Use the secret during apply, but it's never written to state

1. Sensitive Variables: Mark outputs and variables as sensitive = true to prevent them from showing in logs (but this doesn't prevent state storage, just display)

2. External Population: The most common pattern—create the secret with Terraform, populate it with a separate script or pipeline step

Pulumi Approach:

Pulumi takes a different tack by encrypting state. When you use Pulumi's secret types or set config values with --secret, they're stored encrypted in Pulumi's state backend:

import pulumi
import pulumi_aws as aws

# Create secret container
secret = aws.secretsmanager.Secret("db-password",
    name="myapp/prod/db-password",
    kms_key_id=kms_key.id,
    tags={
        "Environment": "prod",
        "Application": "myapp"
    }
)

# The value could come from encrypted Pulumi config
config = pulumi.Config()
db_password = config.require_secret("db_password")  # Encrypted in Pulumi state

# Only use this if the value comes from Pulumi config, not hardcoded!
secret_version = aws.secretsmanager.SecretVersion("db-password-version",
    secret_id=secret.id,
    secret_string=db_password  # This is encrypted in Pulumi state
)

CloudFormation/CDK Pattern:

AWS's native IaC tools support generating random secrets automatically:

# CloudFormation
RestoreModeSecrets:
  Type: AWS::SecretsManager::Secret
  Properties:
    Name: myapp/prod/db-password
    GenerateSecretString:
      PasswordLength: 32
      ExcludeCharacters: '"@/\'
      RequireEachIncludedType: true

CDK equivalent:

import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';

const secret = new secretsmanager.Secret(this, 'DBPassword', {
  secretName: 'myapp/prod/db-password',
  generateSecretString: {
    passwordLength: 32,
    excludeCharacters: '"@/\\',
  },
});

The CDK documentation explicitly warns: If you provide secretStringValue with a literal string, it will be visible in the CloudFormation template to anyone with access. This is almost never what you want.

Advanced IaC Features

Cross-Region Replication:

For multi-region applications or disaster recovery, all IaC tools support replicating secrets:

# Terraform
resource "aws_secretsmanager_secret" "db_password" {
  name = "myapp/prod/db-password"
  
  replica {
    region = "us-west-2"
    kms_key_id = aws_kms_key.secrets_west.id
  }
  
  replica {
    region = "eu-west-1"
    kms_key_id = aws_kms_key.secrets_eu.id
  }
}

Each replica is a read-only copy synchronized automatically. Updates to the primary propagate to replicas within seconds.

Rotation Configuration:

IaC tools can set up automatic rotation, but this requires deploying a Lambda function with the rotation logic:

# Terraform
resource "aws_secretsmanager_secret_rotation" "db_password_rotation" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.rotate_db_password.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

Note: Most teams don't enable rotation at initial deployment. It's added later once the rotation Lambda is tested and the target service (database, API) supports credential updates.

Verdict: IaC is the production standard. Choose Terraform/OpenTofu for multi-cloud and broad provider ecosystem, Pulumi for type-safe code and encrypted state, or CloudFormation/CDK for AWS-native simplicity and deep service integration.

Level 4: Platform Abstraction (Kubernetes-Native and Multi-Cloud)

What it is: Using higher-level controllers that integrate cloud secrets with application platforms, particularly Kubernetes.

External Secrets Operator (ESO)

The most popular pattern for Kubernetes workloads is the External Secrets Operator, which syncs AWS Secrets Manager secrets into Kubernetes Secret objects.

How it works:

1. Deploy the External Secrets Operator in your cluster
2. Create a SecretStore pointing to AWS Secrets Manager (configured with IAM role via IRSA on EKS)
3. Define ExternalSecret resources specifying which AWS secrets to sync
4. The operator fetches secrets from AWS and creates Kubernetes Secrets automatically

Example ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-database
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: myapp-db-secret
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: myapp/prod/db-password

The operator creates a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: myapp-db-secret
  namespace: production
type: Opaque
data:
  password: <base64-encoded-value-from-AWS>

Why this is powerful:

• Applications stay cloud-agnostic: They just read Kubernetes Secrets like always
• Automatic rotation handling: When secrets rotate in AWS, the operator updates the Kubernetes Secret
• Least privilege: Each application's ServiceAccount can have a different IAM role accessing only its secrets
• No SDK required: Applications don't need AWS SDK or credentials

Security consideration: Scope the IAM policy for the External Secrets Operator's role to only the specific secrets needed. Don't grant secretsmanager:GetSecretValue on *.

Crossplane

Crossplane treats cloud resources as Kubernetes CRDs, allowing you to declare AWS secrets in Kubernetes-native YAML:

apiVersion: secretsmanager.aws.crossplane.io/v1alpha1
kind: Secret
metadata:
  name: myapp-db-password
spec:
  forProvider:
    name: myapp/prod/db-password
    description: Production database password
    region: us-east-1
    tags:
      - key: Environment
        value: prod
  providerConfigRef:
    name: aws-provider-config

Crossplane is more about provisioning cloud infrastructure using Kubernetes APIs, whereas ESO is about consuming existing secrets.

Verdict: Use External Secrets Operator for Kubernetes workloads that need to consume AWS secrets. Use Crossplane if you want full cloud infrastructure lifecycle management through Kubernetes control planes.

IaC Tool Comparison: Making the Right Choice

When deploying AWS Secrets Manager via infrastructure-as-code, your choice of tool matters. Here's a structured comparison:

Key Decision Factors

Choose Terraform/OpenTofu if:

• You need multi-cloud support (managing AWS, GCP, Azure secrets in one place)
• You want a large ecosystem of community modules
• You prefer a mature, battle-tested tool (Terraform is 9+ years old)
• You need to integrate with existing Terraform infrastructure

Choose Pulumi if:

• You want to write infrastructure in real programming languages (TypeScript, Python, Go)
• You value type safety and IDE autocomplete
• You need complex logic (loops, conditionals) that's easier in code than HCL
• You want encrypted state by default

Choose CloudFormation if:

• You're AWS-only and want native service integration
• You need AWS-specific features on day one (sometimes AWS features appear in CFN before Terraform)
• You want zero external dependencies or CLIs beyond AWS

Choose AWS CDK if:

• You want CloudFormation's AWS integration with code-based authoring
• You like high-level constructs that bundle best practices (CDK Patterns)
• Your team is already writing TypeScript/Python/Java and prefers that over YAML/HCL

The State and Security Reality

All IaC tools share a common challenge: don't put secret values in code or state.

• Terraform: Even with sensitive = true, values could leak in state until ephemeral values (1.10+). Best practice remains: create the secret resource, populate value via separate pipeline.
• Pulumi: Encrypts state for secrets, which is better, but the value still exists somewhere in state. For maximum security, follow the same pattern: create resource, populate externally.
• CloudFormation/CDK: Templates are stored (in CloudFormation history, S3 for CDK synth). Never include literal secret values. Use parameters with NoEcho or generate random secrets.

Universal best practice: Treat secret values and secret infrastructure as separate concerns. IaC creates the container; a secure pipeline populates the content.

The Project Planton Choice: Minimal and Intentional

Project Planton takes a deliberately minimalist approach to AWS Secrets Manager, grounded in the 80/20 principle: 80% of use cases need only 20% of the available configuration options.

The Philosophy

Secrets infrastructure is not secrets management.

Project Planton's role is to ensure secret containers exist with secure defaults, proper tagging, and correct IAM integration. The actual secret values are populated through secure, auditable processes outside of the infrastructure definition.

This separation provides:

• Security: No secret values ever touch the Project Planton spec or state
• Simplicity: Developers specify what secrets they need, not how to manage AWS KMS, rotation Lambdas, or cross-region replication
• Flexibility: Teams can choose their own secret population strategy (CI/CD, vaults, manual)

The Minimal API

The Project Planton API for AWS Secrets Manager contains exactly what 80% of users need:

message AwsSecretsManagerSpec {
  // List of secret names to create in AWS Secrets Manager.
  // Each name corresponds to a unique secret that will be securely stored.
  repeated string secret_names = 1;
}

That's it. No secret values, no rotation configuration, no replica regions, no KMS keys.

Example usage:

apiVersion: aws.project-planton.org/v1
kind: AwsSecretsManager
metadata:
  name: myapp-prod-secrets
spec:
  secretNames:
    - myapp/prod/DB_PASSWORD
    - myapp/prod/API_KEY
    - myapp/prod/JWT_SECRET

This creates three secrets in AWS Secrets Manager with:

• Default encryption: AWS-managed KMS key (aws/secretsmanager)
• No rotation: Teams enable this later if needed
• Single region: Replicas added manually or via advanced configuration
• No initial value: Populated via secure pipeline or External Secrets Operator

What This Covers (The 80%)

This minimal approach handles:

• Development environments: Quick secret creation for testing
• Production single-region deployments: The majority of applications
• Service credential storage: Database passwords, API keys, tokens
• Integration with Kubernetes: Works perfectly with External Secrets Operator

What It Doesn't Cover (The 20%)

Advanced scenarios require additional configuration or separate steps:

• Automatic rotation: Requires deploying a Lambda function and configuring rotation separately
• Cross-region replication: Can be added manually or through higher-level orchestration
• Custom KMS keys: Default AWS-managed key covers most compliance requirements; custom CMKs for specific use cases
• Resource-based policies: For cross-account access, configured separately

Secure Defaults

Even with minimal configuration, Project Planton ensures production-ready security:

• Encryption at rest: All secrets encrypted with AWS KMS
• IAM integration: Application roles automatically granted least-privilege access to their secrets
• Audit logging: CloudTrail automatically logs all secret access
• Deletion protection: 30-day recovery window on secret deletion

The Value Proposition

By keeping the API minimal, Project Planton:

• Reduces misconfiguration risk: No options means no wrong options
• Accelerates deployment: No decision paralysis about KMS keys or rotation schedules
• Maintains security: Default AWS encryption is production-grade
• Enables iteration: Start simple, add complexity (rotation, replication) when needed

This aligns with modern infrastructure philosophy: make the right thing easy and the wrong thing hard. You can't accidentally leak a secret value in Project Planton because there's no way to specify one.

Production Essentials: What Happens After Deployment

Creating secret containers is step one. Production systems require additional patterns:

Secret Value Population

CI/CD Pipeline Pattern (Recommended):

# GitHub Actions example
- name: Populate AWS Secrets
  env:
    AWS_REGION: us-east-1
  run: |
    # Generate random database password
    DB_PASSWORD=$(openssl rand -base64 32)
    
    # Store in AWS Secrets Manager
    aws secretsmanager put-secret-value \
      --secret-id myapp/prod/DB_PASSWORD \
      --secret-string "$DB_PASSWORD"
    
    # Apply password to RDS instance
    aws rds modify-db-instance \
      --db-instance-identifier myapp-prod \
      --master-user-password "$DB_PASSWORD"

External Vault Integration:

Many organizations use HashiCorp Vault, 1Password, or similar as the source of truth, with a sync job that copies secrets to AWS Secrets Manager for application consumption.

Manual Population (High-Security Environments):

For extremely sensitive secrets, a security engineer might manually populate values using the AWS Console or CLI from a privileged access workstation, ensuring no automation ever sees the value.

Application Integration Patterns

1. Direct SDK Access (Non-Kubernetes):

import boto3
import json

# Get secrets client
client = boto3.client('secretsmanager', region_name='us-east-1')

# Fetch secret
response = client.get_secret_value(SecretId='myapp/prod/DB_PASSWORD')
db_password = json.loads(response['SecretString'])

# Use in application
db_connection = connect_to_database(password=db_password)

Best practice: Use AWS's caching libraries (aws-secretsmanager-caching) to reduce API calls and improve performance. Cache secrets for 1-24 hours depending on rotation frequency.

2. Kubernetes External Secrets Operator:

# ExternalSecret definition
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
  target:
    name: myapp-secrets
  data:
  - secretKey: db_password
    remoteRef:
      key: myapp/prod/DB_PASSWORD
  - secretKey: api_key
    remoteRef:
      key: myapp/prod/API_KEY

Application deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: myapp
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: myapp-secrets
              key: db_password

3. AWS Service Integration (ECS, Lambda):

AWS services can inject secrets directly without application code changes:

{
  "containerDefinitions": [{
    "name": "myapp",
    "secrets": [{
      "name": "DB_PASSWORD",
      "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/prod/DB_PASSWORD"
    }]
  }]
}

Rotation Strategy

Even with minimal initial configuration, plan for rotation:

When to rotate:

• Database credentials: Every 30-90 days via Lambda rotation
• API keys: When team members leave or on schedule if the API supports regeneration
• Certificates: Before expiration (automated via ACM or rotation Lambda)
• Static secrets: Manually during security incidents or staff changes

Rotation pattern:

1. Deploy rotation Lambda (AWS provides templates for RDS, Redshift, etc.)
2. Test rotation in non-production environment
3. Enable rotation via AWS Console or IaC update
4. Monitor CloudWatch for rotation failures

Cost Optimization

Pricing reality:

• $0.40 per secret per month
• $0.05 per 10,000 API calls

For 50 secrets with 100,000 monthly API calls:

• Secret storage: 50 × $0.40 = $20.00/month
• API calls: 10 × $0.05 = $0.50/month
• Total: $20.50/month

Optimization strategies:

1. Consolidate related secrets: Store {"username": "admin", "password": "xyz"} as one JSON secret instead of two separate secrets
2. Use caching: Fetch once at application startup, reuse in memory
3. SSM Parameter Store for non-secrets: Use free tier of Parameter Store for configuration that's not truly sensitive
4. Clean up unused secrets: Delete secrets from decommissioned applications

When to use SSM Parameter Store instead:

• Non-secret configuration (feature flags, endpoints)
• Development/test environments where rotation isn't required
• Thousands of low-value secrets (Parameter Store standard tier is free)

When to use Secrets Manager:

• Production credentials requiring rotation
• Secrets larger than 4KB (Parameter Store standard tier limit)
• Multi-region applications (Secrets Manager has built-in replication)
• Compliance requirements for audit logging and encryption

Monitoring and Compliance

Set up CloudWatch Alarms for:

• Failed rotation attempts
• Unusual secret access patterns
• Secrets not rotated in 90+ days

AWS Config Rules:

• Secrets must have rotation enabled (for production)
• Secrets must use customer-managed KMS keys (if policy requires)
• Secrets must be tagged with environment and owner

CloudTrail Analysis:

• Audit who accessed which secrets when
• Detect secrets read by unauthorized principals
• Track secret creation/deletion events

Conclusion: Infrastructure, Not Secrets

AWS Secrets Manager is a powerful service, but its power comes from what it doesn't do as much as what it does. It doesn't give you secrets; it gives you a secure place to put them. It doesn't manage rotation logic; it gives you hooks to implement it. It doesn't decide what should be secret; it enforces that whatever you deem secret stays protected.

Project Planton embraces this philosophy by providing the minimum viable infrastructure: names of secrets that need to exist. This approach:

• Prevents leaks by making it impossible to embed values in code
• Accelerates deployment by removing decision paralysis
• Maintains flexibility for teams to implement their own secret lifecycle

The journey from hardcoded secrets in application code to production-grade secrets management isn't one big leap—it's a series of thoughtful steps. Start with the infrastructure. Get the containers right. Then focus on secure population, proper rotation, and robust monitoring.

Your secrets are only as secure as your weakest link. Make that link as strong as possible by never letting secret values touch your infrastructure code.