Build a Terraform Community Org on GitHub Enterprise

How-to build a Terraform Community Org on GitHub Enterprise

GitHub Community

Infrastructure As Code

Infrastructure as Code (IAC) is great, people can knock up some Terraform and smash out some stacks in next to no time, delivering value to the business, however there is a side effect to this. Regular IT guys and gals are now evolving from sysadmins into... wait for it.... developers, with the power of Domain Admin or root credentials!

With this evolution comes the consideration of the Software Development Life Cycle (SDLC), that previously our powershell or bash scripts didn't give a stuff about. We kept them in our home drives and selfishly kept all our bugs and mistakes close to our chests.

Code reuse is a common practice in software development, there are a million ways to write the same Terraform code that will create an AWS Application Load Balancer for example. As organisations mature in IAC, multiple teams working in isolation will develop differing standards that will make reuse challenging.

Things that will need addressing in the long run...

  • Module Naming
  • Code syntax
  • Documentation
  • Examples
  • Mandatory variables (tagging)

One way to approach the above is not with an iron fist, but to follow the open source model and create a community. Creating a community gives everyone one a voice, maintaining transparency of decisions and direction. There needs to be guidelines and rules, but as long as they are understood and open the whole community will benefit.

The rest of this post describes how we setup some of this at my current employer. Of course, we will use Terraform to manage this as much as possible to make managing the community as easy as possible.

GitHub Enterprise Organisation

GitHub is a great place to build a community, however not all organisations want to do this in the open. We are fortunate enough to have established a GitHub Enterprise deployment that is available company wide privately.

We have created an organisation that will host all the repositories for our Terraform modules.

For more information on GitHub Organisations see the offical documentation

Terraform and GitHub

One thing for me that makes Terraform great, is the number of available providers. I can use the same IAC code to create my cloud resources in AWS or Azure, but I can also use the GitHub Provider to manage our GitHub Organisation!

For more info and setup see https://registry.terraform.io/providers/integrations/github/latest/docs

GitHub Team Management

To manage access to our organsation efficiently we are going to need a Terraform Module to manage our teams and members. See the following GitHub repo for exactly that module-github-team-management

Using the above module, we can define our Organisation Maintainers groups with the following members.

Define Maintainers

maintainers_team = [
  {
    username    = "user1"
    description = "Maintainer 1"
    role        = "maintainer"
  },
  {
    username    = "user2"
    description = "Maintainer 2"
    role        = "maintainer"
  }
]

Now we can add the following to our main.tf to create and populate our GitHub Organisation Team based on the above vars.

Create Maintainers Team

module "Maintainers" {
  source = "git::https://github.com/jonesy1234/module-github-team-management.git"

  team        = "Maintainers"
  description = "Awesome terraform Maintainers team"
  privacy     = "closed"
  members     = var.maintainers_team
}

Our maintainers are going to be an important group, they will be responsible for reviewing Pull Requests and ensuring that commits align to the overall community style guide.

GitHub Repo Management

We are going to have a lot of repositories in this organisation in the long run. We definitely don't want to be managing all these repositories and repository settings independently. See the following GitHub repo for exactly that module-github-repo

Define Repositories

repositories = {
  module-aws-codebuild-superlinter = {
    topics        = ["terraform", "aws", "module", "codebuild", "github", "superlinter"]
    description   = "Terraform Configuration to setup GitHub superlinter on terraform community org repositories"
    visibility    = "public"
    collaborators = []
    maintainers   = ["user1", "user2"]
  },
  module-azure-bastion = {
    topics        = ["terraform", "azure", "module", "bastion"]
    description   = "Terraform Module for Azure Bastion Service"
    visibility    = "private"
    collaborators = ["user45"]
    maintainers   = ["user1", "user2"]
  }
}

Now we can add the following to our main.tf to create our repositories based on the above vars.

Create all community managed repos

module "CreateRepos" {
  source   = "git::https://github.com/jonesy1234/module-github-repo.git"
  for_each = var.repositories

  name        = each.key
  description = each.value.description
  topics      = each.value.topics
  visibility  = each.value.visibility

  template_source = [
    {
      template_owner      = "terraform"
      template_repository = "template-module"
    }
  ]
}

The module will also manage the topics, collaborators and maintainers for each repo all within our Terraform configuration.

Terraform Style Guide alignment

The above as given us a good base for creating repos that are going to contain the Terraform modules for the community, but what about making sure things align to the style guide. Enter the GitHub super linter!

I'm not fortunate enough to have GitHub Actions yet in my Enterprise instance, so lets do the next best thing and use AWS CodeBuild.

Using terraform code similar to the example here https://cloudonaut.io/workaround-codepipeline-for-github-enterprise we can run the superlinter container from dockerhub

version: 0.2
phases:
    install:
      commands:
        - "cd /usr/bin"
        - "curl -O https://releases.hashicorp.com/terraform/${var.TerraformVersion}/terraform_${var.TerraformVersion}_linux_amd64.zip"
        - "unzip terraform_${var.TerraformVersion}_linux_amd64.zip"
    build:
      commands:
          - cd $CODEBUILD_SRC_DIR
          - echo "Run native Terraform commands to validate configuration"
          - terraform init || exit 1
          - terraform validate || exit 1
          - terraform fmt -check -recursive || exit 1
          - echo "Clean up state and plugins"
          - rm -rf .terraform*
          - ls versions.tf || exit 1
          - if [[ `grep -cE "## Usage|## Inputs Notes|## Requirements|## Providers|## Modules|## Resources|## Inputs|## Outputs" README.md` -eq 8 ]]; then echo "README looks good"; else echo "README is missing sections" & exit 1;fi
          - echo "Run superlinter"
          - RUN_LOCAL=true
          - DEFAULT_WORKSPACE=$CODEBUILD_SRC_DIR
          - /action/lib/linter.sh
    BUILDSPEC
  }
}

The above buildspec will perform the following actions that are triggered via a GitHub WebHook on PR creation.

  • Ensure a terraform init runs successfully
  • Ensure a terraform validate runs successfully
  • Ensure a terraform fmt -check -recursive has been performed
  • Ensure we have a versions.tf file
  • Ensure we have the ## Usage|## Inputs Notes|## Requirements|## Providers|## Modules|## Resources|## Inputs|## Outputs headings in our README.MD
  • Perform a run of the superlinter which will run tflint / terrascan for me

Thats all I'll cover for now, but there are additional items such as branch protection that can be implemented with the GitHub Terraform Provider. Go check it out today and automate all aspects of your repo creation for your DevOps lifecycle.

Hope someone else finds this useful.

Cheers