Creating A Custom Terraform Reconciler in GoLang

Creating A Custom Terraform Reconciler in GoLang

Terraform is an open-source product used widely across the industry for infrastructure deployments. There have been recent moves across various organizations to shift to a more proactive approach to maintaining infrastructure and having to deal with draft (changes that are made to resources outside of Terraform).

For example, if you create an S3 bucket with no tags, Terraform now maintains that configuration in it's state file. If someone manually adds a tag to the S3 bucket via the console, your code will not reflect it. Now, the bucket has a configuration that is different from yours defined in code, the states do not match. Doing a terraform plan or refresh will update the state to include those tags, then doing an apply will proceed to bring your state back in line with your original configuration (aka: no tags), however, this is not an active process. You manually have to run that refresh command, or at least a deployment via terraform plan to catch it.

This blog is going to detail the process of writing a custom 'reconciler' in Go (could be thought of more as a Terraform wrapper) that will automatically refresh your Terraform state to maintain a type of deployment similar to Kubernetes: our .tf files will be the single source of truth. The reconciler will work to always ensure our AWS infrastructure matches our original configuration.

Initial Process

What we need to do:

  • Read in Terraform config files (.tf) to understand the desired state

  • Generate a desired_state.tfstate file that contains what our desired state should look like

  • Do the following continuously in a control loop:

    • Perform a Terraform refresh which will update the state file

    • Create an actual_state.tfstate file that represents the actual state

    • Do a diff between the two

    • If changes happened between the desired state and actual state, do a terraform re-deploy

Initial Project Layout

  •   cmd/
          terraform/
              main.go
      config/
          config.go
      examples/
          <various example .tfstate files>
      scripts/
      terraform/
          <various example .tf files>
      state.go
      error.go
      go.mod
    

Reading the Terraform State File

Reading a state file is relatively simple using https://mholt.github.io/json-to-go/ to convert the state json into a structure.

This blog will be updated periodically as I go through the process. Stay tuned for more.