In this blog post, I'll show you how to set up your Terraform remote state storage using Amazon S3 bucket and DynamoDB table within your existing Terraform project. These will help manage your Terraform state remotely alongside other project resources.

Let's dive into each step together! 🤿

Overview 🌐

This is actually the chicken-and-egg situation of using Terraform to create the backend resources where you want to store your Terraform state on themselves. To make this work, you have to use a two-step process:

1. Write Terraform code to create a S3 bucket and a DynamoDB table, and deploy that code with a local state file (local backend).

2. Go back to the Terraform code, add a s3 remote backend configuration to it to use the newly created S3 bucket and DynamoDB table as a backend state storage, and run terraform init again to move your local state file into S3 bucket.

Later if you would like to destroy the entire stack including both S3 bucket and DynamoDB table for remote backend, you need to do another two-step process in reverse as follows:

1. Go to the Terraform code, remove the backend configuration, and run terraform init -migrate-state to copy the Terraform state back to your local disk.

2. Run terraform destroy to delete the S3 bucket and DynamoDB table along with other resources from the entire stack.

With this roadmap in mind, let's begin our exploration of each step in detail.

NOTE: I have already developed Terraform configurations here to illustrate the steps outlined above in detail. You can visit my repository, create a codespace based on it, and conduct your workshop there.

Migration 🚚

In this section, I will explain how to modify the existing Terraform code project to transition the Terraform state information from the local state file to the S3 bucket, which will also be defined within the same project. Here's how:

Step 1: Define & Apply additional resources for backend

First, in the Terraform code, I define a S3 bucket as a remote location to store the Terraform state file. Here are resource blocks that I configure for this S3 bucket in my repository:

########################
# Secure state storage #
########################

resource "aws_s3_bucket" "backend_bucket" {
  bucket = "terraform-backend-<random-string>"

  # After migrating the remote state file back to the local one, the remote state file still exists on the backend bucket.
  # So, with `force_destroy = true`, we can run `terraform destroy` to destroy the backend bucket even the remote state file still exists.
  force_destroy = true

  tags = {
    Name = "S3 Remote Terraform State Store"
  }
}

resource "aws_s3_bucket_versioning" "backend_bucket_versioning" {
  bucket = aws_s3_bucket.backend_bucket.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "backend_bucket_lifecycle" {
  bucket = aws_s3_bucket.backend_bucket.id

  rule {
    id = "noncurrent-version-expiration-rule"

    noncurrent_version_expiration {
      # NOTE: The number of days an object is noncurrent before Amazon S3 can perform the associated action.
      noncurrent_days = 7

      # NOTE: The number of noncurrent versions Amazon S3 will retain.
      # newer_noncurrent_versions = 10
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "backend_bucket_sse" {
  bucket = aws_s3_bucket.backend_bucket.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256" # NOTE: Using server-side encryption with Amazon S3-managed encryption keys (SSE-S3)
    }
  }
}

resource "aws_s3_bucket_public_access_block" "backend_bucket_public_access_block" {
  bucket = aws_s3_bucket.backend_bucket.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

I don't just define the S3 bucket with its default properties; I also enhance its security with additional layers of protection. Each aspect of this S3 bucket can be described as follows:

• resource "aws_s3_bucket" "backend_bucket"

  • bucket = "terraform-backend-<random-string>": This specifies the name of the S3 bucket. Note that S3 bucket names must be globally unique among all AWS customers, so please choose the <random-string> wisely.

  • force_destroy = true: This ensures that even if the remote state file still exists, running terraform destroy will remove the backend bucket.

• resource "aws_s3_bucket_versioning" "backend_bucket_versioning": This enables versioning on the S3 bucket, ensuring that every update to a file in the bucket creates a new version. This allows you to view older versions of the remote state file and revert to them if needed, serving as a useful fallback mechanism.

• resource "aws_s3_bucket_lifecycle_configuration" "backend_bucket_lifecycle": This prevents too many noncurrent versions of the remote state file from being retained.

• resource "aws_s3_bucket_server_side_encryption_configuration" "backend_bucket_sse": This ensures that your state files, along with any contained secrets, are always encrypted on disk when stored in the S3 bucket.

• resource "aws_s3_bucket_public_access_block" "backend_bucket_public_access_block": This will block all public access to the S3 bucket. Since Terraform state files may contain sensitive data and secrets, adding this extra layer of protection ensures that no one on your team can accidentally make this S3 bucket public.

If you're collaborating on a project with multiple team members and using the S3 bucket as a remote state backend, DynamoDB is utilized to support the locking mechanism. It contains a single attribute named "LockID," indicating whether operations on the state file can proceed.

During Terraform operations (such as plan, apply, destroy), the state file is locked for the duration of the operation. If another developer attempts to execute operations concurrently, the request is denied. Operations can resume once the current operation is complete, and the lock on the state file is released from the DynamoDB table.

Here's how we define the DynamoDB table for the locking purpose:

#######################
# State locking table #
#######################

resource "aws_dynamodb_table" "backend_state_lock_tbl" {
  name = "terraform-backend-state-locking". # NOTE: You can change the table name as desired.

  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S" # NOTE: String type
  }

  tags = {
    "Name" = "DynamoDB Terraform State Lock Table"
  }
}

NOTE: Setting billing_mode = "PAY_PER_REQUEST" means that we opt for the on-demand capacity mode in the DynamoDB table, where you pay for the actual request usage. This is ideal for scenarios where the table is primarily used for testing purposes.

Let's begin by creating the remote backend resources on AWS and saving the state locally. Run the following commands:

terraform plan -out /tmp/tfplan
terraform apply /tmp/tfplan

You should notice that a terraform.tfstate file is created, containing the state information corresponding to the newly created backend resources. Proceed to the next step to transfer its contents to the remote storage.

Step 2: Configure backend & Reinitiate Terraform project

Next, add a backend configuration for the S3 bucket to your Terraform code. This configuration is specific to Terraform and is placed within a terraform block. Here's how you can structure it:

terraform {

  # ... <Other blocks> ...

  backend "s3" {
    bucket         = "terraform-backend-<random-string>"
    region         = "ap-southeast-1"
    key            = "path/to/remote_terraform.tfstate" # NOTE: Change the object key for the state file as desired.
    # NOTE: `encrypt = true` as a second layer in addition to `backend_bucket_sse` to ensure that the state file is always encrypted on the S3 bucket.
    encrypt        = true
    dynamodb_table = "terraform-backend-state-locking"
  }
}

NOTE: Ensure to replace <random-string> with the same name suffix of the S3 bucket you created earlier.

To instruct Terraform to store your state file in this S3 bucket, you'll need to run the terraform init command again. This command not only downloads provider code but also configures your Terraform backend. Since the init command is idempotent, it's safe to run it multiple times.

Additionally, you need a separate credential to migrate the state file to the S3 bucket. If you haven't specified AWS access keys in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, you'll need to partially configure the backend block externally in a separate variable file, such as backend.tfvars:

access_key = "<aws-account-access-key>"
secret_key = "<aws-account-secret-key>"

Then, you'll need to add the -backend-config=backend.tfvars option to the terraform init command so that Terraform has the necessary permissions to migrate the state file:

terraform init -backend-config=backend.tfvars

After run the above command, Terraform will recognize that you already have a state file locally and prompt you to copy it to the new S3 backend bucket. Confirm by typing "yes" and your Terraform state will be stored in the S3 bucket. You can verify this by navigating to the S3 bucket on the Amazon S3 web console.

Clean Up 🧹

As mentioned earlier, to delete all resources, including the backend resources, you'll also need to perform two separate tasks:

Step 1: Remove backend configuration & Reinitiate Terraform project

Begin by deleting the entire backend "s3" block inside the terraform block from your code. Then, copy the Terraform state information back to your local disk by executing the following command:

terraform init -migrate-state

You'll notice that the state information is now restored to the terraform.tfstate file.

Step 2: Destroy all resources

To proceed with resource deletion on AWS, simply execute the Terraform CLI:

terraform destroy

Conclusion 🏁

In summary, this blog post has shown you how to set up a remote Terraform state backend using Amazon S3 bucket and DynamoDB table in your existing project. With this setup, you can manage your infrastructure more efficiently and work better with your team.

Happy Terraforming! 🤗

References 🔗

• github.com / prasitstk / terraform-aws-practices-s3-backend

• Terraform / Backend / S3

• Terraform / Backend Configuration / Partial Configuration

• How to manage Terraform state

• Terraform S3 Backend Best Practices

• How to Manage Terraform S3 Backend – Best Practices

• How to Store Terraform State on S3

In this blog post, I'll walk you through the process and show you how to use my Terraform code from my solution to simplify the resource definition for this task.

Let's get started and simplify data migration! 🚀

Architecture 🕍

The objective of this AWS pattern is to create multi-account resources that allow us to migrate objects or files from an Amazon S3 bucket in an AWS account to another S3 bucket in a different account or region. Let's see the architecture of this pattern:

• The source S3 bucket allows GetObject permission through an attached resource policy, typically a S3 Bucket Policy.

• In the target account, an IAM user (temp-user) needs to assume an IAM role granting PutObject permission for the target S3 bucket and GetObject permission for the source S3 bucket.

• Finally, execute copy or sync commands on behalf of temp-user to transfer data from the source S3 bucket to the target S3 bucket.

Prerequisites 📋

There are a few things we need before we demonstrate how to utilize my Terraform code to support copying or syncing S3 objects across accounts:

• An active source AWS account with an IAM user (assuming named as src-tf-developer) that has required permissions to create/update/delete resources by Terraform CLI.

• An active target AWS account with an IAM user (assuming named as tgt-tf-developer) that has required permissions to create/update/delete resources by Terraform CLI.

• Access keys of those IAM Users from both accounts above.

• An S3 bucket of the source AWS account in a particular region with some objects to be copied.

• An empty S3 bucket of the target AWS account in a particular region.

• Terraform (version >= 0.13)

Terraform Environment Setup 🛠️

I will demonstrate how to set up the environment for our Terraform project based on my repository here. First, navigate to my repository, click "Code" button and click "Create codespace on main" to start your own codespace on this repository:

After the codespace initializes successfully, create terraform.tfvars (ignored from. git) on the root directory of the project to define variables for Terraform as follows:

# Source AWS account configuration
# - Source AWS account region code: "ap-southeast-1" (= Singapore), for example.
# - AWS access key ID and secret key of src-tf-developer in the source account.
# - Source S3 bucket name, "src-bucket.prasitio.com", for example.
src_aws_region     = "<source-aws-account-region>"
src_aws_access_key = "<source-aws-account-access-key>"
src_aws_secret_key = "<source-aws-account-secret-key>"
src_s3_bucket_name = "<source-aws-s3-bucket-name>"

# Target AWS account configuration
# - Target AWS account region code: "ap-southeast-1" (= Singapore), for example.
# - AWS access key ID and secret key of tgt-tf-developer in the target account.
# - Target S3 bucket name, "tgt-bucket.prasitio.com", for example.
tgt_aws_region     = "<target-aws-account-region>"
tgt_aws_access_key = "<target-aws-account-access-key>"
tgt_aws_secret_key = "<target-aws-account-secret-key>"
tgt_s3_bucket_name = "<target-aws-s3-bucket-name>"

NOTE: This repository configure devcontainer to make AWS CLI, Terraform CLI, and necessary VS Code extensions available to be used. So, no need to install all of them again.

Infrastructure Deployment 🏗️

Initialize the Terraform project on the current working directory and download/install AWS provider by:

terraform init

Then, run the following command to create an execution plan and write it into a temporary folder. This will let you preview the infrastructure changes that Terraform plans to make to your infrastructure before applying the actual changes:

terraform plan -out /tmp/tfplan

Run the following command to apply the actual infrastructure changes based on the exported plan from the previous step onto both source and target AWS accounts:

terraform apply /tmp/tfplan

Note that by default terraform apply will automatically create a state file called terraform.tfstate which contains information about all resources created or modified by Terraform. It is actually not a best practice to keep the state file locally when you collaborate with others to work with the same Terraform project. In the future blog post, I will write about the best practice to keep the Terraform state file remotely on S3 bucket. Stay tune!

Explanation 💬

Before we perform the copy or sync task from the source S3 objects, let me explain more about what are inside my Terraform project, what resources Terraform creates and refers to, and why they are needed for our task.

Providers

Terraform relies on plugins called providers to interact with cloud providers, here is AWS. In this project, I declare AWS provider to be installed in providers.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

In providers.tf, from the installed AWS provider, I also configure multiple settings for each source and target AWS accounts separately. I use the alias meta-argument to provide an extra name for each configuration. The data source and resource blocks needs to refer to these aliases by using provider = aws.<alias> to differentiate the destination of the AWS account that those blocks are working on. Here are the provider configurations for each account in the providers.tf:

provider "aws" {
  alias      = "source"
  region     = var.src_aws_region
  access_key = var.src_aws_access_key
  secret_key = var.src_aws_secret_key
}

provider "aws" {
  alias      = "target"
  region     = var.tgt_aws_region
  access_key = var.tgt_aws_access_key
  secret_key = var.tgt_aws_secret_key
}

Input variables

Terraform input variables are used to assign dynamic values to resource attributes in a Terraform module. With Input variables, you can customize modules without altering your Terraform code.

In my project, I define the input variables of the root module in variables.tf file and assign their value in terraform.tfvars as I mentioned in the previous section. Here is how variables are defined:

variable "src_aws_region" {
  type        = string
  description = "AWS Region of your source AWS account"
}

variable "src_aws_access_key" {
  type        = string
  description = "AWS Access Key of your source AWS account"
}

variable "src_aws_secret_key" {
  type        = string
  description = "AWS Secret Key of your source AWS account"
}

variable "src_s3_bucket_name" {
  type        = string
  description = "S3 bucket name of your source AWS account"
}

variable "tgt_aws_region" {
  type        = string
  description = "AWS Region of your target AWS account"
}

variable "tgt_aws_access_key" {
  type        = string
  description = "AWS Access Key of your target AWS account"
}

variable "tgt_aws_secret_key" {
  type        = string
  description = "AWS Secret Key of your target AWS account"
}

variable "tgt_s3_bucket_name" {
  type        = string
  description = "S3 bucket name of your target AWS account"
}

NOTE: Terraform variables can be other data types rather than string. You can find more information here.

Data sources

Data sources provide information about entities that are not managed by the current Terraform configuration. In this project, our data sources are source and target S3 buckets because I try to simulate the general problem of copying and syncing S3 objects across existing S3 buckets, not the new ones. Later, by using this Terraform project defined in main.tf, we can create new resources that associate with both S3 buckets to support only for copying and syncing S3 objects operation.

Here is how we define those data sources in data.tf:

###############
# Datasources #
###############

#----------------#
# Source account #
#----------------#

data "aws_s3_bucket" "src_s3_bucket" {
  provider = aws.source
  bucket   = var.src_s3_bucket_name
}

#----------------#
# Target account #
#----------------#

data "aws_s3_bucket" "tgt_s3_bucket" {
  provider = aws.target
  bucket   = var.tgt_s3_bucket_name
}

The data block is to define how we query a data of a particular resource in a cloud provider. You can see more about how to get information about AWS S3 Bucket from a particular account here.

Resources

As I mentioned before, main.tf is a file that define all of the AWS resources to be created on both source and target AWS accounts only for copying and syncing S3 objects operation. All of them are security resources that limit permissions of AWS CLI to do only those tasks. The below section will describe each resource:

Source account :: S3 Bucket Policy

resource "aws_s3_bucket_policy" "app_bucket_policy" {
  provider = aws.source
  bucket   = data.aws_s3_bucket.src_s3_bucket.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        "Sid" : "DelegateS3Access",
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : aws_iam_role.s3_migration_role.arn
        },
        "Action" : [
          "s3:ListBucket",
          "s3:GetObject",
          "s3:GetObjectTagging",
          "s3:GetObjectVersion",
          "s3:GetObjectVersionTagging"
        ],
        "Resource" : [
          "arn:aws:s3:::${data.aws_s3_bucket.src_s3_bucket.id}/*",
          "arn:aws:s3:::${data.aws_s3_bucket.src_s3_bucket.id}"
        ]
      }
    ]
  })
}

This resource block defines a S3 Bucket Policy to be attached to the source S3 bucket on the source AWS account. The policy allows the IAM Role aws_iam_role.s3_migration_role from the target account to get S3 objects from the source bucket.

Target account :: IAM User

resource "aws_iam_user" "temp_user" {
  provider = aws.target
  name     = "temp-user"
  path     = "/"
}

resource "aws_iam_access_key" "temp_user_access_key" {
  provider = aws.target
  user     = aws_iam_user.temp_user.name
}

The first resource block defines an IAM User on the target account named temp-user. Another resource block will generate temp-user's access key ID and secret key to be used on AWS CLI to perform copy or sync task later.

Target account :: IAM Policy

resource "aws_iam_policy" "s3_migration_policy" {
  provider = aws.target
  name     = "S3MigrationPolicy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        "Effect" : "Allow",
        "Action" : [
          "s3:ListBucket",
          "s3:GetObject",
          "s3:GetObjectTagging",
          "s3:GetObjectVersion",
          "s3:GetObjectVersionTagging"
        ],
        "Resource" : [
          "arn:aws:s3:::${data.aws_s3_bucket.src_s3_bucket.id}",
          "arn:aws:s3:::${data.aws_s3_bucket.src_s3_bucket.id}/*"
        ]
      },
      {
        "Effect" : "Allow",
        "Action" : [
          "s3:ListBucket",
          "s3:PutObject",
          "s3:PutObjectAcl",
          "s3:PutObjectTagging",
          "s3:GetObjectTagging",
          "s3:GetObjectVersion",
          "s3:GetObjectVersionTagging"
        ],
        "Resource" : [
          "arn:aws:s3:::${data.aws_s3_bucket.tgt_s3_bucket.id}",
          "arn:aws:s3:::${data.aws_s3_bucket.tgt_s3_bucket.id}/*"
        ]
      }
    ]
  })
}

This resource block defines the IAM Policy S3MigrationPolicy that allow any attaching principals to get S3 objects from the source S3 bucket and put S3 objects to the target S3 bucket. For our task, we attach it to the IAM Role S3MigrationRole defined below.

Target account :: IAM Role

resource "aws_iam_role" "s3_migration_role" {
  provider = aws.target
  name     = "S3MigrationRole"
  path     = "/"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        "Sid" : "AllowTempUserToAssumeRole",
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : aws_iam_user.temp_user.arn
        },
        "Action" : "sts:AssumeRole"
      }
    ]
  })

  managed_policy_arns = [
    aws_iam_policy.s3_migration_policy.arn
  ]
}

The last resource block defines an IAM Role S3MigrationRole attaching the IAM Policy S3MigrationPolicy to allow only temp-user to assume to the role to perform copy or sync operation by AWS CLI.

Outputs

In the last file of our Terraform setup, outputs.tf, we define output variables to display crucial information in the console after successful applying the root module. These outputs are specifically tailored for the copy or sync tasks executed through the AWS CLI:

output "tgt_temp_user_access_key_id" {
  value = aws_iam_access_key.temp_user_access_key.id
}

output "tgt_temp_user_access_key_secret" {
  value = aws_iam_access_key.temp_user_access_key.secret
  sensitive = true
}

output "tgt_s3_migration_role_arn" {
  value = aws_iam_role.s3_migration_role.arn
}

output "src_aws_region" {
  value = var.src_aws_region
}

output "src_s3_bucket_name" {
  value = data.aws_s3_bucket.src_s3_bucket.id
}

output "tgt_aws_region" {
  value = var.tgt_aws_region
}

output "tgt_s3_bucket_name" {
  value = data.aws_s3_bucket.tgt_s3_bucket.id
}

You can see the sensitive = true attribute for the tgt_temp_user_access_key_secret output. This is important because Terraform considers this generated secret key as sensitive data. Terraform requires us to confirm our intent to display it in the console by forcing us to add sensitive = true for this output block; otherwise, the following error will occur:

To view all outputs, you can use the command terraform output:

Although tgt_temp_user_access_key_secret remains masked, you can explicitly display sensitive data like this by running terraform output tgt_temp_user_access_key_secret.

For the next steps, most commands will rely on the terraform output <output-variable-name> command to assign each output variable to its corresponding environment variable. We also use the -raw flag to print values in a format suitable for assigning environment variables, without quotes or a newline character.

Copy or Sync 🔄

Run the following commands to set up environment variables for AWS CLI with AWS access keys of the newly created temp-user IAM User of the target account:

# Set up AWS access keys of temp-user
export AWS_ACCESS_KEY_ID=$(terraform output -raw tgt_temp_user_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(terraform output -raw tgt_temp_user_access_key_secret)

Assume S3MigrationRole by temp-user and replace environment variables for AWS CLI with the newly received temporary access keys from aws sts assume-role command:

# Assume S3MigrationRole by temp-user and set up temporary access keys
export TARGET_S3_MIGRATION_ROLE_ARN=$(terraform output -raw tgt_s3_migration_role_arn)
export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"  \
  $(aws sts assume-role \
    --role-arn "$TARGET_S3_MIGRATION_ROLE_ARN" \
    --role-session-name AWSCLI-Session \
    --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
    --output text))

Verify that currently AWS CLI actually assumes the S3MigrationRole role by:

aws sts get-caller-identity

Before running copy or sync command, set up the following variables:

export SRC_BUCKET_NAME=$(terraform output -raw src_s3_bucket_name)
export SRC_REGION=$(terraform output -raw src_aws_region)
export TGT_BUCKET_NAME=$(terraform output -raw tgt_s3_bucket_name)
export TGT_REGION=$(terraform output -raw tgt_aws_region)

Run the following command to copy all objects from a folder in the source S3 bucket into another folder on the target S3 bucket:

aws s3 cp s3://$SRC_BUCKET_NAME/src-folder-path/ \
  s3://$TGT_BUCKET_NAME/tgt-folder-path/ \
  --recursive --source-region \(SRC_REGION --region \)TGT_REGION

Check whether all of the objects on a specified folder are copied into the target S3 bucket folder path.

Or, you can sync all of the objects in the source S3 bucket with the target S3 bucket:

aws s3 sync "s3://$SRC_BUCKET_NAME/" \
  "s3://$TGT_BUCKET_NAME/" \
  --source-region \(SRC_REGION --region \)TGT_REGION

Then, verify whether all objects from the source S3 bucket are copied into the target S3 bucket.

Clean Up 🧹

To clean up the whole infrastructure by deleting all of the resources from both source and target AWS accounts, excluding the source and target S3 bucket and their objects inside, run the following command:

terraform destroy

If -auto-approve is set, then the destroy confirmation will not be shown:

terraform destroy -auto-approve

Conclusion 🏁

This blog post shows an easy way to copy or sync data between AWS S3 buckets across accounts using Terraform. I've outlined the necessary steps, from setting up prerequisites to executing the migration task. With Terraform, it's simple to manage the necessary parts. By using AWS CLI and Terraform outputs, you can smoothly copy or sync your data. Following cleanup steps is also important to keep things tidy and save costs.

By following this guide, you can automate S3 data migration, reducing manual work and error risks. Terraform's declarative approach allows easy adaptation and scaling to fit your needs, giving you the power to manage data migration tasks smoothly across your AWS setups.

Stay tuned for more helpful tips in our next blog post! 😎

References 🔗

Main Resources

• github.com / prasitstk / terraform-aws-patterns-copy-data-s3-bucket-across-accounts

• AWS Prescriptive Guidance Patterns: Copy data from an S3 bucket to another account and Region by using the AWS CLI

AWS

• AWS sts assume role in one command

• Creating an S3 bucket (AmazonS3 documentation)

• Amazon S3 bucket policies and user policies (Amazon S3 documentation)

• IAM identities (users, groups, and roles) (IAM documentation)

• cp command (AWS CLI documentation)

• sync command (AWS CLI documentation)

Terraform

• Terraform - Types and Values

• aws_s3_bucket | Data Sources | hashicorp/aws

• Terraform - Providers

• Terraform - Providers - alias: Multiple Provider Configurations

"Dev containers" (Development containers) are Docker containers tailored to provide comprehensive development environments. When working in a Codespace, you're essentially using a dev container on a virtual machine. These containers can be configured for specific repositories, ensuring Codespaces offer a customized environment with all necessary tools and runtimes for the project.

In this post, I'll leverage Dev containers to create a basic development environment in Codespace for a Terraform-AWS repository. This setup includes:

• AWS access keys for AWS CLI

• Terraform VS Code extension

• AWS CLI

• Terraform CLI

First, I'll cover setting up AWS access keys and the Terraform VS Code extension for a dev container. Then, I'll detail three methods for installing AWS CLI and Terraform CLI:

1. "postCreateCommand" property of Dev container

2. Dev containers features

3. Custom Dockerfile [Recommended 👍]

By following these steps, you'll be ready to apply it to your real-world Terraform for AWS projects on Codespace.

Let's get started! 🚀

Define AWS access keys on Dev Container 🔑

AWS access keys are credentials comprising an access key ID and secret access key. We utilize them to grant access for AWS CLI to manage AWS resources on our AWS account from our Codespace environment.

To begin, log in to your GitHub account and navigate to the Terraform-AWS repository where you wish to set up a dev container. Next, click the "Code" button, choose the "Codespaces" tab, and select the ellipsis icon "...". From the dropdown menu, opt for "Configure dev container" as illustrated below:

After that, GitHub will try to create a new directory .devcontainer and a new file inside devcontainer.json. The configuration files for a dev container are contained in a .devcontainer directory in your repository and devcontainer.json is the primary one. Define the "recommended" personal secrets by adding "secrets" property into this JSON file as follows:

{
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  "features": {
  },
  // ADDED //
  "secrets": {
    "AWS_ACCESS_KEY_ID": {
      "description": "AWS access key associated with an IAM user or role.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    },
    "AWS_SECRET_ACCESS_KEY": { 
      "description": "the secret key associated with the access key (AWS_ACCESS_KEY_ID). This is essentially the password for that access key.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    }
    ///////////
  }
}

Proceed by clicking the "Commit changes..." button, entering the commit message, and confirming the changes by clicking "Commit changes":

It's worth noting that AWS access keys are not directly defined in the Dev container configuration files but are "recommended" instead. Users must create these secrets in their personal Codespaces settings. If not, the "secrets" property will prompt them to do so when using the advanced options method to create a Codespace as follows:

Then the names of the recommended secrets are only listed on the below page when the dev container configuration on the selected branch specifies these secrets.

By selecting "Associate with repository?" for certain secrets and clicking "Create codespace", these secrets will be added to your personal Codespaces settings, along with creating a new codespace. Navigate to your GitHub account's personal settings page, select "Codespaces" from the left-hand menu, and you'll find the two secrets automatically created under "Codespaces secrets":

Update each secret by pasting your AWS access keys and clicking "Save changes." These secrets won't be available as environment variables until you click "Reload to apply" button on the dialog box that just appears below your codespace:

Once AWS access keys from your Codespaces secrets are confirmed, verify their presence using the echo command:

Your AWS access keys should now appear in the VS Code Terminal on Codespace. With AWS CLI installed, it will utilize these keys to manage your AWS resources accordingly.

Define Terraform VS Code extension on Dev Container 🧩

On the step 8 in my previous post, I demonstrated manual installation of the Terraform extension for VS Code. However, you can streamline this process by defining it directly in the devcontainer.json configuration file. Here's how:

Add the "customizations" property to your devcontainer.json file:

{
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  "features": {
  },
  "secrets": {
    "AWS_ACCESS_KEY_ID": {
      "description": "AWS access key associated with an IAM user or role.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    },
    "AWS_SECRET_ACCESS_KEY": { 
      "description": "the secret key associated with the access key (AWS_ACCESS_KEY_ID). This is essentially the password for that access key.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    }
  },
  // ADDED //
  "customizations": {
    "vscode": {
      "extensions": ["hashicorp.terraform"]
    }
  }
  ///////////
}

After adding the property, you'll see a dialog appear. Click "Rebuild now" to proceed:

Alternatively, you can select "Rebuild Container" from the menu as follows:

Once completed, the Terraform VS Code Extension will be installed automatically here:

Note that to find the extension ID for inclusion in the "extensions" array, visit the extension's marketplace page. You can find this information under the "More Info" section:

Install AWS & Terraform CLIs on Dev Container 🛠

Method#1: "postCreateCommand" property of Dev container

You can add a new "postCreateCommand" property to the devcontainer.json file to run commands or scripts after the container is created. Start by creating a shell script file .devcontainer/post-create.sh to install AWS and Terraform CLIs:

#!/usr/bin/env bash

# Install Terraform CLI
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

# Install AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
rm -f awscliv2.zip
rm -rf aws

Next, include "postCreateCommand" in .devcontainer/devcontainer.json to execute the shell script after container setup automatically:

{
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  "features": {
  },
  "secrets": {
    "AWS_ACCESS_KEY_ID": {
      "description": "AWS access key associated with an IAM user or role.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    },
    "AWS_SECRET_ACCESS_KEY": { 
      "description": "the secret key associated with the access key (AWS_ACCESS_KEY_ID). This is essentially the password for that access key.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    }
  },
  "customizations": {
    "vscode": {
      "extensions": ["hashicorp.terraform"]
    }
  },
  // ADDED //
  "postCreateCommand": "bash .devcontainer/post-create.sh"
  ///////////
}

Once added, rebuild the container by clicking "Rebuild now" or selecting "Rebuild Container" from the menu.

Note that, in this method, after the codespace is rebuilt, you need to wait for a while until you see both awscliv2.zip file and aws/ directory are deleted. This can be inconvenient and I do not recommend this method. However, I would like to demonstrate this method to show you how to work with "postCreateCommand" on the Dev container configuration. Hopefully, it should be more useful for other use cases.

To verify installation, run aws --version and terraform -version commands:

With this setup, you can now utilize Terraform to manage AWS resources. Let's explore the second method by using Dev container features.

Method#2: Dev container features

Dev container offers a built-in method for adding additional software via the "features" property in the devcontainer.json file. This enables the installation of various tools to support your development, either from a predefined set of Features or custom ones. To make AWS and Terraform CLIs available, define features in the devcontainer.json file:

{
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  // UPDATED //
  "features": {
    "ghcr.io/devcontainers/features/aws-cli:1": {},
    "ghcr.io/devcontainers/features/terraform:1": {}
  },
  /////////////
  "secrets": {
    "AWS_ACCESS_KEY_ID": {
      "description": "AWS access key associated with an IAM user or role.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    },
    "AWS_SECRET_ACCESS_KEY": { 
      "description": "the secret key associated with the access key (AWS_ACCESS_KEY_ID). This is essentially the password for that access key.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    }
  },
  "customizations": {
    "vscode": {
      "extensions": ["hashicorp.terraform"]
    }
  }
}

Once added, rebuild the container by clicking "Rebuild now" or selecting "Rebuild Container" from the menu.

With this method, not only is software installed, but also VS Code extensions supporting the software. However, this can result in excessive extensions being installed, potentially consuming more memory or CPU than anticipated and impacting development efficiency. You can view all installed extensions as follows:

Again, to verify installation, run aws --version and terraform -version commands:

This setup also allows you to utilize Terraform for managing AWS resources. Now, let's explore the third method, which I recommend 😉.

Method#3: Custom Dockerfile [Recommended 👍]

Instead of starting with an existing image, this method involves creating a custom image using a Dockerfile. This file extends the image by executing additional shell commands to install AWS and Terraform CLIs during the container image building process. First, create a Dockerfile in the .devcontainer directory:

FROM mcr.microsoft.com/devcontainers/universal:2

# Install Terraform CLI
RUN wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg \
    && echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list \
    && sudo apt update && sudo apt install terraform

# Install AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
    && unzip awscliv2.zip \
    && sudo ./aws/install \
    && rm -f awscliv2.zip \
    && rm -rf aws

In the devcontainer.json file, replace "image" property with "build" property:

{
  // REPLACE //
  //"image": "mcr.microsoft.com/devcontainers/universal:2",
  "build": {
    "dockerfile": "Dockerfile"
  },
  /////////////
  "features": {
  },
  "secrets": {
    "AWS_ACCESS_KEY_ID": {
      "description": "AWS access key associated with an IAM user or role.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    },
    "AWS_SECRET_ACCESS_KEY": { 
      "description": "the secret key associated with the access key (AWS_ACCESS_KEY_ID). This is essentially the password for that access key.",
      "documentationUrl": "https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list"
    }
  },
  "customizations": {
    "vscode": {
      "extensions": ["hashicorp.terraform"]
    }
  }
}

Rebuild the container by clicking "Rebuild now" or selecting "Rebuild Container" from the menu.

Again, to verify installation, run aws --version and terraform -version commands:

This method enables you to utilize Terraform for managing AWS resources efficiently. I recommend this method for the following reasons:

1. Unlike Method #1, there is no need for a wait time after the container is running, as AWS and Terraform CLIs are already installed in the custom image.

2. Unlike Method #2, no excessive extensions are installed, ensuring a streamlined development environment.

Bonus: Make the selected method a GitHub template 🎁

After choosing your preferred method, it's time to commit changes:

git add .
git commit -m "Add Dev container configuration"
git push origin main

Now, anyone wanting to create a new codespace from your Terraform-AWS repository won't need to manually set up everything. They can simply click to create it from your repository, and voilà! Everything they need is ready!

Alternatively, if you plan to create multiple Terraform-AWS repositories for future projects, you can make your repository a template:

• Navigate to the main page of the repository.

• Under your repository name, click "Settings" tab.

• Select "Template repository".

Now, every time you create a new repository, you can use this repository as a template:

Conclusion 🏁

Congratulations! 🎉 You now have your Dev container configuration for your Codespace setup without any manual tasks. What's even better is that you also have your pre-defined template for your future Terraform-AWS projects. I hope you find this post useful for your work and enjoy joining me on my upcoming blog posts! 😎

References 🔗

AWS

• Environment variables to configure the AWS CLI

Dev containers

• Introduction to dev containers

• Using the default dev container configuration

• Create a Dev Container

• Dev Container metadata reference

• Dev containers > Editors > Visual Studio Code

• Specifying recommended secrets for a repository

• Dev Container Features reference

• Available Dev Container Features

• Adding features to a devcontainer.json file

• Custom Dev Container Features

• Dev containers > Using Images, Dockerfiles, and Docker Compose

GitHub

• Specifying recommended secrets for a repository

• Creating a template repository

• Creating a repository from a template

Embarking on this journey, we'll navigate the basic concept of AWS, Terraform, and GitHub Codespaces. Starting by setting up our GitHub Codespace environment, we'll then streamline the creation of a simple AWS infrastructure using Terraform. Finally, we'll explore a one-step cleanup process for these resources.

Join me in simplifying infrastructure management with Terraform and GitHub Codespaces, wherever creativity takes you 😉.

Exploring AWS, Terraform, and GitHub Codespaces 🔍

What is AWS?

AWS (Amazon Web Services) is one of the Amazon's cloud computing platform, offering a wide range of services like computing power, storage, networking, databases, machine learning, AI, and more. Primarily, it allows businesses to grow without the need for costly upfront infrastructure investments, offering flexibility and cost-effectiveness. You can explore the full range of benefits AWS offers businesses by clicking on this link.

What is Terraform?

Terraform, created by HashiCorp, is an open-source tool for managing infrastructure as code. It lets users define and provision resources using a simple configuration language, called HCL (HashiCorp configuration language). With Terraform, you can manage your infrastructure through code, enabling version control, collaboration, and automation. It supports multiple cloud providers like AWS, Azure, and Google Cloud, making it adaptable for various environments.

What is GitHub Codespaces?

GitHub Codespaces offers cloud-hosted development environments for coding from anywhere via any web browser, seemingly integrating with GitHub repositories. It eliminates the need for local setups, enabling developers to work on projects from any device with internet access, including my iPad Air 😎.

Get Started 🚀

Step 1: Create a Codespace on a branch in a GitHub repository.

Open your browser, go to https://github.com/login and login to your GitHub account.

Browse to a repository where you’d like to create the Codespace from. Press "Code" button, select the "Codespaces" tab, and press "Create codespace on main" button to let GitHub initiate a Codespace VS Code environment based on your main branch.

Within a few seconds your GitHub Codespace should be launched in the new browser tab as follows:

Step 2: Install Terraform CLI in the Codespace

First, check the OS release of the Codespace by typing the command cat /etc/os-release on the "Terminal" tab as follows:

For me, it shows "Ubuntu 20.04.6 LTS". Actually I do this because I would like to make sure that I select the correct Terraform installation commands from the web page here:

Based on my OS release of my Codespace, I will copy the commands on the "Ubuntu/Debian" tab and press on the "Terminal" tab on my Codespace VS Code and run all of them to install the Terraform CLI (Command Line Interface) as follows:

# Installation commands for Ubuntu/Debian
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

To verify that we install the Terraform CLI successfully, type the terraform -version on the "Terminal" tab, it should display the version of the installed Terraform CLI as follows:

Step 3: Install AWS CLI

Installing the AWS CLI (Command Line Interface) after the Terraform CLI is necessary to enable Terraform to interact with AWS resources effectively. Based on the installation commands from this AWS page on the "Linux" section, run the following commands on the Terminal tab:

# Installation commands for Linux x86 (64-bit)
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# ADDED: Clean up after installation
rm -f awscliv2.zip
rm -rf aws

After the installation is complete, verify the installation of the AWS CLI by typing the command aws --version on the "Terminal" tab, it should show the version of the installed AWS CLI as follows:

Step 4: Sign up for a new AWS account if not available yet

If you already have your own AWS account, skip this step.

If not, create your new AWS account by following the steps here.

Step 5: Create a new IAM User for Terraform development

Sign in to your AWS account, then type "IAM" on the search text box at the top to navigate to "IAM Dashboard":

On the left-hand side menu, under the "Access management", click on the "Users" menu link.

On the "IAM Users" page, click the "Create user" button, then provide the information of the new IAM User as follows:

• User name: terraform-developer

• Attach policies directly > Select AdministratorAccess

BEST PRACTICE 👍

Note that using the AdministratorAccess policy for an IAM user managing AWS resources with Terraform isn't usually recommended. This policy gives full access to all AWS services and resources, which can be risky if the user's credentials are hacked.

Instead, it's better to stick to the principle of least privilege. Only give IAM users the permissions they really need, like creating or changing certain resources. This makes things safer and reduces the chance of someone getting into your AWS resources without permission.

Step 6: Create access keys for the IAM User

From "IAM Users" page, select terraform-developer IAM User we've just created, scroll down until you see the "Access keys" box, then press "Create access key" button as follows:

Select the "Use case" as "Command Line Interface (CLI)", acknowledge the warning below, and click "Next" button below as follows:

Optionally set the description tag, then click "Create access key" button below.

Your access key and secret access key now have been successfully generated. It's important to note that once you click the "Done" button below, you won't be able to access this "Retrieve access keys" page again. Be sure to copy or download these keys securely, as they won't be accessible later.

Step 7: Configure AWS CLI

To enable Terraform CLI for AWS resource management, configure your AWS credentials using the access keys generated earlier. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in your GitHub Codespace Terminal session. To do this, navigate to your repository "Settings > Secrets and variables > Codespace", and create new repository secrets with the following details:

• AWS_ACCESS_KEY_ID = <Access key>

• AWS_SECRET_ACCESS_KEY =<Secret access key>

After configuring these secrets, your Codespace will automatically detect them. You'll receive a notification in your Codespace stating, "Your codespace secrets have changed. Reload to apply." Simply click the 'Reload to apply' button for the changes to take effect.

After the Codespace reloads, access keys for AWS CLI are configured. Terraform CLI is now effectively authorized to manage AWS resources using the terraform-developer IAM User. We're all set to define and create our AWS infrastructure.

IMPORTANT 🔐

Note that AWS credentials or any secrets stored in the "Repository secrets" setting are shared with all GitHub collaborators. For privacy, store them in your personal "Codespaces secrets" setting instead.

Reference: Managing your account-specific secrets for GitHub Codespaces

Step 8: Install the Terraform Extension for VS Code

On your Codespace, install the Terraform extension for VS Code by:

1. Select "Extensions" menu on the left-hand side.

2. Search for the "terraform" on the marketplace.

3. Press "Install" button on the "HashiCorp Terraform" extension.

Step 9: Define AWS infrastructure in Terraform configuration file

In the root directory of the Codespace, create a file named main.tf to define our AWS infrastructure using HCL (HashiCorp Configuration Language).

In main.tf, add the following code to create a basic Amazon VPC (Virtual Private Cloud) virtual network. Remember, creating a VPC incurs no cost, but it's recommended to clean it up afterward.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "ap-southeast-1"
}

resource "aws_vpc" "example_vpc" {
  cidr_block = "10.0.0.0/16"
}

The Terraform code provided above is from the example on the AWS Provider page linked here.

We can explain each block in the code as follows:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

• The terraform block specifies the required AWS provider. AWS provider is a Terraform plugin that enables Terraform to interact with a specific service or technology platform, such as AWS, or Azure. This block will also map its local provider name called aws to a source address "hashicorp/aws" and a version constraint "~> 5.0".

provider "aws" {
  region = "ap-southeast-1"
}

• The provider "aws" block will configure the AWS Provider with default region of Singapore ("ap-southeast-1"). Note that the default region in this block is only optional. You can define it by either AWS_REGION or AWS_DEFAULT_REGION environment variable with ap-southeast-1 value through Secrets and variables setting from step 7 as well.

resource "aws_vpc" "example_vpc" {
  cidr_block = "10.0.0.0/16"
}

• This resource block defines a VPC resource (by "aws_vpc" resource type) named "example_vpc" with IPv4 CIDR block of "10.0.0.0/16".

Step 10: Create AWS infrastructure

Now we have our configuration file to define our AWS infrastructure. We will then create the specified AWS resources on your AWS account on behalf of terraform-developer IAM User. Here are the commands to:

• Initialize the Terraform project in the root directory to download the AWS provider plugin and set up the working directory by:

terraform init

• Preview the actions that Terraform will take before applying any changes to the infrastructure by:

terraform plan

• After you're satisfied with the changes proposed by the terraform plan command, apply those changes to your AWS account by:

terraform apply

Note that Terraform will prompt you to confirm applying the changes. Simply type yes and press Enter to proceed.

Verify the result ✅

• Sign in to your AWS account.

• On the AWS console, type "VPC" on the top search bar and click "VPC" link as follows:

• Click "Your VPCs" menu, then you should see the newly created VPC whose IPv4 CIDR is "10.0.0.0/16" as specified in the Terraform code as follows:

If you see the new VPC on your VPC console, congratulations 🎉🎉🎉! You’ve successfully applied and managed your infrastructure as code using Terraform for AWS on GitHub Codespaces, even on the go 🚶🏻.

Clean up 🧹

After experimenting with Terraform, it's crucial to clean up your resources to avoid unexpected costs. To destroy the resources created earlier, run:

terraform destroy

Terraform will request confirmation before destroying the resources. Type yes and press Enter to proceed.

Conclusion 🏁

In this guide, we've navigated the setup of Terraform for AWS on GitHub Codespaces, enabling flexible infrastructure management from any device. By following simple steps, we established our environment, created AWS resources, and verified our setup.

As we finish up, remember that your learning with Terraform and AWS doesn't stop here. Now that you have the basics, you can start trying out more advanced setups and improvements to better manage your infrastructure alongside me 😁.

Keep coding and exploring! 🚀

References 🔗

Terraform

• Install | Terraform | HashiCorp Developer | Linux

• Docs overview | hashicorp/aws | Terraform | Terraform Registry

AWS

• AWS | Benefits at a Glance

• AWS CLI | Install or update to the latest version of the AWS CLI

• AWS Knowledge Center | How do I create and activate a new AWS account?

GitHub Codespaces

• Managing your account-specific secrets for GitHub Codespaces

Topics to be covered

In this blog, I'll be covering a wide range of topics, including:

• Personal learning experiences

• Cloud and software engineering tutorials and tips

• Productivity and self-improvement

• Technology news and updates

• And much more!

Get in Touch

I'm eager to hear from you! Whether you have questions, suggestions, or just want to say hello, feel free to leave a comment on any of my posts or connect with me on social media. I'm looking forward to engaging with fellow bloggers and readers.

Thank you for joining me on this adventure! I hope you find value and inspiration in my posts. Let's learn and grow together.