How To Import Resources into Terraform Control
Motivation
If you are like most organizations and users of Terraform, chances are you have a lot of cloud resources managed by Terraform, as well as resources that are not currently managed by Terraform. The benefits of Terraform, and Infrastructure as Code (IAC) more generally, is to have cloud infrastructure that is well documented, standardized, and easily controllable with code changes in the future.
If a larger portion of the cloud environment is not controlled by Terraform, these benefits erode. As a result, there is a natural inclination to want Terraform to control previously uncontrolled resources.
(Brief) Review of How Terraform Works
Before we go through how Terraform creates resources, it will be valuable to briefly review the major components of Terraform. These are Terraform Code, Terraform State, and Remote Cloud Resources.
- Terraform Code: This is HCL code that declaratively defines remote cloud resources.
- Remote Cloud Resources: These are the actual storage buckets, servers, databases, etc. from a cloud provider like AWS, Azure, GCP.
- Terraform State: Terraform state can be thought of, at a high level, as a required mapping of information between written Terraform code and the corresponding remote cloud resource. That is, it says: “This particular HCL code block corresponds to this particular remote s3 bucket”.
When we have uncontrolled cloud resources, we have the remote cloud resource, but are missing the corresponding Terraform code, as well as the appropriate state file updates linking Terraform code to the remote cloud resource.
To proceed, we have the following options:
Delete the Remote Cloud Resource and Recreate the Resource with Terraform
One option that involves no state file manipulation is to delete the remote cloud resource and recreate it identically via a regular Terraform workflow.
While perhaps easier for developers to execute, this is often not a great option; for storage resources, this can result in a loss of data, and for compute resources this can result in unwanted application downtime.
Manually Import an Uncontrolled Resource
- Identify the uncontrolled resource that you wish to bring into Terraform control.
- Write a corresponding piece of Terraform code in HCL that matches the uncontrolled resource.
- Identify the state file which should control the newly Terraformed resource.
- Write a state migration statement that updates the chosen state file to link the new Terraform code with the appropriate remote resource.
This involves running a migration statement via the CLI in a directory where terraform init has already been run.
This import migration command takes the following format:
Example:
We have an uncontrolled S3 bucket in AWS and want to bring it into Terraform control.
First we write the Terraform:
Then within a directory where we have already run terraform init, we run the state migration command:
Downsides with this Approach:
- Fairly manual process, from identification of remote cloud resources to running CLI commands
- There is no written record of state migrations run via the CLI. Since state migrations are notoriously finicky in Terraform, this can be risky.
- The Terraform code being linked to the remote resource must match the remote resource exactly. Otherwise, the next time terraform apply is run within that directory/workspace, changes will be made to the remote cloud resource.
Importing Uncontrolled Resources using dragondrop.cloud
dragondrop.cloud is a software service consumed as a container running within your cloud environment. It automates the lifecycle of identifying and importing uncontrolled resources into Terraform control by performing the following:
- Identifying cloud resources that are currently uncontrolled by Terraform ad-hoc or on a cron job schedule.
- Generating the needed HCL Terraform code and corresponding state migration commands.
- Selecting the most appropriate state file for controlling each identified cloud resource using machine learning.
- Outputting recommendations via a Pull Request into your Version Control System, providing a concrete record of all state migrations that were previously run via the CLI.
- Using the GitHub Action provided by dragondrop.cloud, migrations are only allowed to be merged into your code base if the HCL Terraform code identically matches the current remote cloud resource identically for an extra layer of security.
Downsides with this Approach:
- There is a bit of a start-up cost with dragondrop.cloud. It requires creating some cloud infrastructure in your cloud to host dragondrop, as well as configuring some environment variables on that created compute instance. Once set up, however all uncontrolled resources can be identified in one pull request, vs. the one-off nature of the manual approach described above.
dragondrop.cloud provides Terraform modules to spin up the infrastructure required for hosting its container, as well as client-side environment variable validation in their web application. This helps cap the onboarding process to less than an hour for the majority of customers.
Conclusion
We hope this helps you better understand options for bringing uncontrolled cloud infrastructure into Terraform control.
–
dragondrop.cloud’s mission is to automate developer best practices while working with Infrastructure as Code. Our flagship product regularly scans and identifies resource changes that have occurred outside of a Terraform workflow (e.g. drift) so that dev teams can have a Cloud environment that is fully represented as code. All of our tools are self-hosted by our customers, with no data ever leaving their servers. To learn more, schedule a demo or get started today!
Learn More About Terraform
Terraform Variable Management
We've previously discussed the syntax for creating variables within Terraform configuration. While this helps us with syntax, it leaves open questions about how variable values are actually passed into our Terraform workflow. CLI Specification When running terraform...
What is Terraform? How Does Terraform Work?
What is Terraform? Terraform is the leading Infrastructure as Code (IaC) tool (see our article for a review of IaC). It is fully open-sourced, and managed by HashiCorp. Over 1000+ different infrastructure providers can be controlled via Terraform, and new providers...
Quickstart: Writing Terraform
In this article we discuss how the basics of writing organized Terraform infrastructure configuration. Specifying Terraform's Configuration We recommend keeping a given Terraform module's requirements within their own versions.tf file. Within versions.tf, you can...