The Challenge

Terraform is a great product for managing infrastructure on AWS however many people start by creating an IAM user and sharing access keys into configuration files. This is really bad from a security aspect as these often get checked into version control and even worse in a public repo.

We can use the AWS ecosystem for your terraform workflow using CodeCommit, CodePipeline, CodeBuild and IAM. We can give the access we need without managing any access keys or secrets.

We store all our terraform code within CodeCommit which ticks our boxes for versioning and traceability for our infrastructure.

We use Terraform S3 and DynamoDB backend capabilities to store state and provide locking to maintain consistency. Changes pulled from CodeCommit are run through a Terraform Validate and Plan operation prior to making any changes to our infrastructure.

Lastly, we run a Terraform Apply to make the requested changes to our infrastructure.

That’s the high level summary of what we are going to build. Here is the Cloudformation template to create it. Yes, we are using Cloudformation however this is only to bootstrap our AWS account so that we can use Terraform moving forward and control all resources via our CodeCommit Repo.

---
AWSTemplateFormatVersion: 2010-09-09
Description: 'Create the supporting resources for Terraform Infrastructure
  state and build pipelines'

Parameters:
  ServiceName:
    Description: 'Specifies the hosted service name used within component
      naming'
    Type: String

  TerraformVersion:
    Description: 'Specifies the Terraform binary version'
    Type: String
    Default: '0.12.10'

  LogsRetentionInDays:
    Description: 'Specifies the number of days you want to retain log
      events in the specified log group'
    Type: Number
    Default: 3
    AllowedValues: [1, 3, 5, 7, 14, 30]

Resources:
  TerraformBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      Tags:
        - Key: Name
          Value: !Join ['-', ['AP2', 'INF', !Ref 'ServiceName', 'S3', 'Terraform']]
      BucketName: !Sub '${ServiceName}-${AWS::Region}-terraform'
      AccessControl: LogDeliveryWrite
      VersioningConfiguration:
        Status: Enabled
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

  TerraformBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref 'TerraformBucket'
      PolicyDocument:
        Id: Content
        Version: '2012-10-17'
        Statement:
          - Action: '*'
            Condition:
              Bool:
                aws:SecureTransport: 'false'
            Effect: Deny
            Principal: '*'
            Resource: !Join ['', ['arn:aws:s3:::', !Ref 'TerraformBucket', '/*']]
            Sid: S3ForceSSL
          - Action: 's3:PutObject'
            Condition:
              'Null':
                s3:x-amz-server-side-encryption: 'true'
            Effect: Deny
            Principal: '*'
            Resource: !Join ['', ['arn:aws:s3:::', !Ref 'TerraformBucket', '/*']]
            Sid: DenyUnEncryptedObjectUploads

  TerraformInfraDynamoDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: 'terraform-infrastructure-locks'
      BillingMode: PAY_PER_REQUEST
      KeySchema:
        -
          AttributeName: "LockID"
          KeyType: "HASH"
      AttributeDefinitions:
        -
          AttributeName: "LockID"
          AttributeType: "S"
      SSESpecification:
        SSEEnabled: true

  TerraformInfrastructureCodeRepository:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Sub 'app-${ServiceName}-infrastructure'
      RepositoryDescription: !Sub 'Repository for ${ServiceName} Infrastructure IaC'

  CodeBuildServiceRole:
    Type: AWS::IAM::Role
    Properties:
      Path: '/managed/'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AdministratorAccess'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Action: 'sts:AssumeRole'
            Effect: Allow
            Principal:
              Service:
                - codebuild.amazonaws.com

  TerraformInfrastructureValidatePlanCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Location: !Sub "https://git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/app-${ServiceName}-infrastructure"
        Type: "CODECOMMIT"
        BuildSpec: 'codebuild/validate_plan.yml'
      Environment:
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:1.0"
        Type: "LINUX_CONTAINER"
        EnvironmentVariables:
          - Name: TERRAFORM_VERSION
            Value: !Ref 'TerraformVersion'
      Description: Verify Terraform Syntax and run Plan
      ServiceRole: !GetAtt CodeBuildServiceRole.Arn
      TimeoutInMinutes: 300

  TerraformInfrastructureValidatePlanLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub '/aws/codebuild/${TerraformInfrastructureValidatePlanCodeBuild}'
      RetentionInDays: !Ref LogsRetentionInDays

  TerraformInfrastructureApplyCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Location: !Sub "https://git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/app-${ServiceName}-infrastructure"
        Type: "CODECOMMIT"
        BuildSpec: 'codebuild/apply.yml'
      Environment:
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:1.0"
        Type: "LINUX_CONTAINER"
        EnvironmentVariables:
          - Name: TERRAFORM_VERSION
            Value: !Ref TerraformVersion
      Description: Apply Terraform Plan
      ServiceRole: !GetAtt CodeBuildServiceRole.Arn
      TimeoutInMinutes: 300

  TerraformInfrastructureApplyLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub '/aws/codebuild/${TerraformInfrastructureApplyCodeBuild}'
      RetentionInDays: !Ref LogsRetentionInDays

  TerraformInfrastructureDestroyCodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Artifacts:
        Type: NO_ARTIFACTS
      Source:
        Location: !Sub "https://git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/app-${ServiceName}-infrastructure"
        Type: "CODECOMMIT"
        BuildSpec: 'codebuild/destroy.yml'
      Environment:
        ComputeType: "BUILD_GENERAL1_SMALL"
        Image: "aws/codebuild/amazonlinux2-x86_64-standard:1.0"
        Type: "LINUX_CONTAINER"
        EnvironmentVariables:
          - Name: TERRAFORM_VERSION
            Value: !Ref TerraformVersion
      Description: Destroy Terraform State - WILL DESTROY YOUR WHOLE CONFIG!
      ServiceRole: !GetAtt CodeBuildServiceRole.Arn
      TimeoutInMinutes: 300

  TerraformInfrastructureDestroyLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub '/aws/codebuild/${TerraformInfrastructureDestroyCodeBuild}'
      RetentionInDays: !Ref LogsRetentionInDays

  PipelineExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      Path: '/managed/'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Action: 'sts:AssumeRole'
            Effect: Allow
            Principal:
              Service:
                - codepipeline.amazonaws.com
      Policies:
        -
          PolicyName: CodePipelinePassRoleAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Action: 'iam:PassRole'
                Effect: Allow
                Resource: !GetAtt CodeBuildServiceRole.Arn
        -
          PolicyName: CodePipelineS3ArtifactAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Action:
                  - 's3:GetObject'
                  - 's3:GetObjectVersion'
                  - 's3:GetBucketVersioning'
                  - 's3:PutObject'
                Effect: Allow
                Resource:
                  - !Sub 'arn:aws:s3:::${TerraformBucket}'
                  - !Sub 'arn:aws:s3:::${TerraformBucket}/*'
        -
          PolicyName: CodePipelineGitRepoAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Action:
                  - 'codecommit:GetBranch'
                  - 'codecommit:GetCommit'
                  - 'codecommit:UploadArchive'
                  - 'codecommit:GetUploadArchiveStatus'
                  - 'codecommit:CancelUploadArchive'
                Effect: Allow
                Resource:
                  - !GetAtt TerraformInfrastructureCodeRepository.Arn
        -
          PolicyName: CodePipelineBuildAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Action:
                  - 'codebuild:StartBuild'
                  - 'codebuild:StopBuild'
                  - 'codebuild:BatchGetBuilds'
                Effect: Allow
                Resource:
                  - !GetAtt TerraformInfrastructureValidatePlanCodeBuild.Arn
                  - !GetAtt TerraformInfrastructureApplyCodeBuild.Arn

  TerraformInfrastructurePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtifactStore:
        Location: !Ref TerraformBucket
        Type: S3
      Name: !Sub ${ServiceName}-terraform-infrastructure
      RoleArn: !GetAtt PipelineExecutionRole.Arn
      Stages:
        -
          Name: GetSource
          Actions:
            -
              Name: TerraformSource
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: '1'
              Configuration:
                RepositoryName: !Sub 'app-${ServiceName}-infrastructure'
                BranchName: master
              OutputArtifacts:
                - Name: SourceArtifact
              RunOrder: 1
        -
          Name: ValidatePlan
          Actions:
            -
              Name: TerraformValidatePlan
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: '1'
              Configuration:
                ProjectName: !Ref 'TerraformInfrastructureValidatePlanCodeBuild'
              InputArtifacts:
                - Name: SourceArtifact
              RunOrder: 1
        -
          Name: Apply
          Actions:
            -
              Name: TerraformApply
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: '1'
              Configuration:
                ProjectName: !Ref 'TerraformInfrastructureApplyCodeBuild'
              InputArtifacts:
                - Name: SourceArtifact
              RunOrder: 1

Outputs:
  TerraformInfrastructureCodeRepository:
    Description: Terraform Infrastructure Code Repository CloneUrlHttp
    Value: !GetAtt 'TerraformInfrastructureCodeRepository.CloneUrlHttp'
    Export:
      Name: TerraformInfrastructureCodeRepositoryCloneUrlHttp

  TerraformBucketName:
    Description: Terraform State Bucket Name
    Value: !Ref 'TerraformBucket'
    Export:
      Name: 'TerraformBucketName'

Cloudformation Resources

ServiceName – Prefix for your CodePipeline and CodeCommit resources

TerraformVersion – Version of Terraform to use

LogsRetentionInDays – CodeBuild logs will be output to CloudWatch. This determines the retention period.

TerraformBucket – S3 bucket used to store Terraform state and CodePipeline artefacts, public access restricted

TerraformBucketPolicy – Policy setting with secure defaults

TerraformInfraDynamoDBTable – DynamoDB table to maintain Terraform locks, SSE enabled

TerraformInfrastructureCodeRepository – CodeCommit repo for all our terraform configuration files

CodeBuildServiceRole – IAM Role for CodeBuild, this has been given the AWS Managed Policy AdministratorAccess however this can be restricted only to be allowed to provision resources that are relevant to your Terraform configuration. Least privilege should be at the forefront of your mind here to please adjust as required.

TerraformInfrastructureValidatePlanCodeBuild – CodeBuild project to perform the Terraform Validate & Plan

TerraformInfrastructureValidatePlanLogGroup – CloudWatch log group for the above CodeBuild project

TerraformInfrastructureApplyCodeBuild – CodeBuild project to perform the Terraform Apply

TerraformInfrastructureApplyLogGroup – CloudWatch log group for the above CodeBuild project

TerraformInfrastructureDestroyCodeBuild –

TerraformInfrastructureDestroyLogGroup –

PipelineExecutionRole – IAM Role for CodePipeline. This requires access to the above CodeBuild projects and access to commit triggers in CodeCommit

TerraformInfrastructurePipeline – Create the pipeline plumbing the above components into a workflow

TerraformInfrastructureCodeRepository – CodeCommit Repo name generated

TerraformBucketName – State Bucket name generated

CodeCommit Structure

Our CodeBuild projects expect the following structure within our CodeCommit repo.

/
/env/terraform_state.tfvars
/env/variables.tfvars
/codebuild
/codebuild/validate_plan.yml
/codebuild/apply.yml
/codebuild.destroy.yml

terraform_state.tfvars – This file contains our references to our AWS components, see the following for more details – https://www.terraform.io/docs/backends/config.html

# /env/terraform_state.tfvars
bucket         = "S3_BUCKET_NAME_FROM_CLOUDFORMATION_OUTPUTS"
key            = "infrastructure/terraform.tfstate"
region         = "ap-southeast-2"
dynamodb_table = "terraform-infrastructure-locks"
encrypt        = true
variables.tfvars – This is where I store variables to inject into the Terraform config. For now this is blank

CodeBuild Buildspec.yml

CodeBuild requires a buildspec file to tell it what actions to take. All our buildspec files live within the codebuild dir in CodeCommit.

validate_plan.yml – The following build spec runs our Terraform Validate & Plan

# /codebuild/validate_plan.yml
version: 0.2

env:
  variables:
    TF_IN_AUTOMATION: "true"

phases:
  install:
    runtime-versions:
      python: 3.7
    commands:
      - "cd /usr/bin"
      - "curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
      - "unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
  build:
    commands:
      - cd "${CODEBUILD_SRC_DIR}"
      - terraform init -backend-config="envs/terraform_state.tfvars" -no-color
      - terraform validate -no-color
      - terraform plan -var-file="envs/variables.tfvars" -no-color

apply.yml – The following build spec runs our Terraform Apply

# /codebuild/apply.yml
version: 0.2

env:
  variables:
    TF_IN_AUTOMATION: "true"

phases:
  install:
    runtime-versions:
      python: 3.7
    commands:
      - "cd /usr/bin"
      - "curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
      - "unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
  build:
    commands:
      - cd "${CODEBUILD_SRC_DIR}"
      - terraform init -backend-config="envs/terraform_state.tfvars" -no-color
      - terraform apply -var-file="envs/variables.tfvars" -auto-approve -no-color

destroy.yml – The following build spec runs a Terraform Destroy. As you will have noted from above this is not part of the standard workflow as generally we won’t be destroying the resources but in the development process it’s likely we will want to teardown everything regularly.

# /codebuild/destroy.yml
version: 0.2

env:
  variables:
    TF_IN_AUTOMATION: "true"

phases:
  install:
    runtime-versions:
      python: 3.7
    commands:
      - "cd /usr/bin"
      - "curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
      - "unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
  build:
    commands:
      - cd "${CODEBUILD_SRC_DIR}"
      - terraform init -backend-config="envs/terraform_state.tfvars" -no-color
      - terraform destroy -var-file="envs/variables.tfvars" -auto-approve -no-color

Let kick off our Cloudformation template and see what gets created. I’ve used the following parameters for our template.

Once the stack has been provisioned we have our CodeCommit Repo.

We have our Terraform CodePipeline. Note the first step has failed. This is because the repo is empty and has no master branch. Let go ahead and populate it

Lets clone our repo and populate our content. We can get our S3 state bucket and CodeCommit repo from our Cloudformation Outputs.

$ git clone  https://git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/app-sjramblings-infrastructure
Cloning into 'app-sjramblings-infrastructure'...
warning: You appear to have cloned an empty repository.

$ ls codebuild/
apply.yml        destroy.yml        validate_plan.yml

$ cat envs/terraform_state.tfvars 
bucket         = "sjramblings-ap-southeast-2-terraform"
key            = "infrastructure/terraform.tfstate"
region         = "ap-southeast-2"
dynamodb_table = "terraform-infrastructure-locks"
encrypt        = true

$ cat envs/variables.tfvars
# Empty file for vars later..
$

For our example we will add Terraform code to create an S3 bucket. We tell Terraform to store our resource state in S3 and create an S3 bucket as follows.

$ cat main.cf 
terraform {
  backend "s3" {}
}

resource "aws_s3_bucket" "b" {
  bucket = "sjramblings-bootstrap-terraform"
  acl    = "private"

  tags = {
    Name        = "My State Bucket"
  }
}

Now we commit and see the pipeline kick into action

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    new file:   codebuild/apply.yml
    new file:   codebuild/destroy.yml
    new file:   codebuild/validate_plan.yml
    new file:   envs/terraform_state.tfvars
    new file:   envs/variables.tfvars
    new file:   main.tf

$ git commit -m 'Initial Commit'
[master efc29fe] Initial Commit
 6 files changed, 75 insertions(+)
 create mode 100644 codebuild/apply.yml
 create mode 100644 codebuild/destroy.yml
 create mode 100644 codebuild/validate_plan.yml
 create mode 100644 envs/terraform_state.tfvars
 create mode 100644 envs/variables.tfvars
 create mode 100644 main.tf

$ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 8 threads
Compressing objects: 100% (9/9), done.
Writing objects: 100% (10/10), 1.20 KiB | 1.21 MiB/s, done.
Total 10 (delta 2), reused 0 (delta 0)
To https://git-codecommit.ap-southeast-2.amazonaws.com/v1/repos/app-sjramblings-infrastructure
   521d7e8..efc29fe  master -> master

We see our commit has been picked up and triggered a pipeline execution.

Our GetSource and ValidatePlan have been successful so we have no errors in our syntax or current state, onto creating out S3 bucket.

As we can see from the CodeBuild run we have our S3 bucket via Terraform as requested.

This is a very simple resource creation example but hopefully shows you the power of wrapping Terraform within AWS CodeCommit, CodeBuild & CodePipeline.

Hope this helps someone else!

Cheers

For more articles on Terraform click here!