How to store Terraform state in DigitalOcean Spaces

Pushup3-Spud0 We have previously discussed How to use Terraform with DigitalOcean, which guided you through the process of configuring your local workstation to start working Digital Ocean and Terraform.

In this article, we are going to build on what we learned. So before we start using Terraform to create our Infrastructure for our application, we are going to learn about the importance of Terraform state and how we can store state within Digital Ocean spaces.

What are DigitalOcean Spaces

Spaces Object Storage is, as the name implies, an object storage service. Which are DigitalOcean compatible offering, similar to AWS S3 or Azure Blob Storage buckets, enabling you too store and serve large amounts of data, and the built-in CDN minimizes page load times and improves performance. Each Spaces bucket you create within an account has its own URL and can be used as a logical unit for segmenting content.

he Spaces URL naming pattern is spacename.region.digitaloceanspaces.com and region.digitaloceanspaces.com/spacename, where spacename is the name of your Space and region is the region your Space is in.

What is Terraform state

Terraform state is a crucial concept in Terraform's infrastructure as code methodology. It represents the actual state of the infrastructure that Terraform manages.

Terraform state is a file that tracks the current status of the infrastructure managed by Terraform. This state file is essential because:

• Tracking Infrastructure: It keeps a record of all resources created, modified, or destroyed by Terraform.
• Syncing Reality with Configuration: It helps Terraform understand the existing state of infrastructure and compare it with the desired configuration defined in the configuration files. This comparison allows Terraform to plan and apply only the necessary changes.
• Performance Optimization: By knowing the current state, Terraform can determine the most efficient way to reach the desired state, optimizing the creation, modification, and destruction of resources.
• Collaboration: When stored remotely (e.g., in an S3 bucket or Terraform Cloud), the state file allows multiple team members to collaborate safely on infrastructure changes without conflicts.

The state file should be handled securely and backed up, as it contains sensitive information about the infrastructure and is integral to Terraform's operation.

The state file is updated whenever you run a Terraform command, such as terraform apply, terraform plan, or terraform destroy. This ensures that the state file remains synchronized with the actual infrastructure.

Key benefits of Terraform State:

1. Version control: The state file provides a single source of truth for your infrastructure’s state, making it easier to manage and track changes.
2. Consistency: Terraform ensures that the infrastructure is in the desired state by comparing the state file with the actual resources.
3. Idempotence: Terraform’s state-based approach allows for safe and repeatable deployments, as it only applies necessary changes.

Best practices for managing Terraform State

1. Store state files remotely: Use a backend like Amazon S3, Azure Blob Storage, or HashiCorp’s Terraform Cloud to store state files, ensuring they are not version-controlled and avoiding potential conflicts.
2. Use locking mechanisms: Implement locking mechanisms, such as DynamoDB or etcd, to prevent concurrent modifications to the state file.
3. Regularly back up state files: Store backups of state files to ensure data integrity and availability.

Why use Spaces for Terraform state

Using DigitalOcean Spaces for Terraform state storage ensures that your state files are safely stored, easily accessible, and securely managed, supporting efficient infrastructure as code workflows.

• Reliability: DigitalOcean Spaces provides a highly available and durable object storage solution, ensuring that your Terraform state files are safe and resistant to data loss.
• Scalability: Spaces can handle huge amounts of data, supporting the growth of your infrastructure without any performance degradation.
• Accessibility: State files stored in Spaces can be easily accessed and managed from anywhere, facilitating remote collaboration among team members working on the same infrastructure.
• Versioning: Spaces supports versioning, enabling you to keep track of changes to your state file over time and revert to previous versions if needed.
• Security: With robust access controls, encryption at rest, and in transit, DigitalOcean Spaces helps protect sensitive information contained within your state files.
• Integration: Terraform can natively interact with Spaces using backend configurations, making setup straightforward and seamless.
• Cost-Effective: DigitalOcean Spaces offers competitively priced storage, making it an economical choice for state file management without compromising on features.

How to create Digital Ocean Spaces

In order to create a DigitalOcean spaces object store you will need a DigitalOcean Account. Once you have an account you can follow the instructions on their great documentation portal How to Create a Spaces Bucket

Once you have created your spaces object, then you just need to assign Spaces Keys in the Applications & API section

Once you have created your keys you can write them you to .bash_variables file we created in How to use Terraform with DigitalOcean copying both the Access Key and Secret key. However, because DigitalOcean Spaces are S3 Compatible, you'll notice we named the Environment Variables AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY.

echo "export AWS_ACCESS_KEY_ID=<paste your token here>" | tee -a .bash_variables
echo "export AWS_SECRET_ACCESS_KEY=<paste your token here>" | tee -a .bash_variables

### remember to source your current terminal 
. ~/.bashrc

How to start using DigitalOcean spaces for your terraform state

We can now create a really simple Terraform project to test this out. Lets create a new file. Which we will call main.tf. Once you've created the file add the code below to it.

We're not going to do anything too fancy here, all we're doing is configuring the Terraform provider, then we define the remote state using the S3 Driver,

By providing the 3 values

• s3: Update with the region of your bucket
• bucket: name of your bucket.
• key: This is the filename your Terraform state is stored under. In many S3-compatible clients, these are treated as file paths and can contain forward slashes (/) for directories and periods for file extensions. For example, a value of test/example.tfstate would place an example.tfstate file inside the test directory in the root of your bucket.

We also provide some AWS specific value overrides to basically deactivate them.

Lastly, we are going to create a really simple tag in our environment. A Tag is a label that can be applied to a Droplet resource in order to better organize or facilitate the lookups and actions on it.

terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
   backend "s3" {
      endpoints = {
        s3 = "https://ams3.digitaloceanspaces.com"
      }

      bucket = "threenine-terraform-state"
      key    = "threenine.tfstate"

      # Deactivate a few AWS-specific checks
      skip_credentials_validation = true
      skip_requesting_account_id  = true
      skip_metadata_api_check     = true
      skip_region_validation      = true
      skip_s3_checksum            = true
      region                      = "us-east-1"
    }
  }

  resource "digitalocean_tag" "foobar" {
    name = "foobar"
  }
}

Initialize and Apply the Terraform Configuration

In order to test that this all works and that our Remote state will be created, we will run our typical terraform commands

First, we will initialize our configuration using

terraform init

This should return something similar to the below confirming that we were able to initialize our configuration.

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of digitalocean/digitalocean from the dependency lock file
- Using previously-installed digitalocean/digitalocean v2.42.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

We can then execute our plan to verify what we are going to create

terraform plan

which should return something similar to the below confirming resources we are going to create.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_tag.foobar will be created
  + resource "digitalocean_tag" "foobar" {
      + databases_count        = (known after apply)
      + droplets_count         = (known after apply)
      + id                     = (known after apply)
      + images_count           = (known after apply)
      + name                   = "foobar"
      + total_resource_count   = (known after apply)
      + volume_snapshots_count = (known after apply)
      + volumes_count          = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

We can now apply our plan

terraform apply

Which should confirm the execution of our plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_tag.foobar will be created
  + resource "digitalocean_tag" "foobar" {
      + databases_count        = (known after apply)
      + droplets_count         = (known after apply)
      + id                     = (known after apply)
      + images_count           = (known after apply)
      + name                   = "foobar"
      + total_resource_count   = (known after apply)
      + volume_snapshots_count = (known after apply)
      + volumes_count          = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

digitalocean_tag.foobar: Creating...
digitalocean_tag.foobar: Creation complete after 1s [id=foobar]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

After your plan is completed you should now be able to navigate to the DigitalOcean spaces store and see your shiny new state file has been created.

Summary

We provided a detailed guide on storing Terraform state files in DigitalOcean Spaces, an object storage service similar to AWS S3 or Azure Blob Storage. The post starts by explaining DigitalOcean Spaces' capabilities, such as data storage and serving, and the benefits of using its built-in CDN for improved performance.

The article then delves into the importance of Terraform state, which keeps track of the infrastructure managed by Terraform, ensuring synchronization between the actual and desired states of the infrastructure. Key benefits of maintaining Terraform state include tracking infrastructure, ensuring configuration consistency, optimizing performance, and facilitating team collaboration.

Best practices for managing Terraform state are highlighted, such as storing state files remotely to ensure security and accessibility. The post ultimately helps readers understand how to effectively use DigitalOcean Spaces to manage their Terraform state, promoting reliable and scalable infrastructure management.