How To Import Resources into Terraform Control
Motivation
If you are like most organizations and users of Terraform, chances are you have a lot of cloud resources managed by Terraform, as well as resources that are not currently managed by Terraform. The benefits of Terraform, and Infrastructure as Code (IAC) more generally, is to have cloud infrastructure that is well documented, standardized, and easily controllable with code changes in the future.
If a larger portion of the cloud environment is not controlled by Terraform, these benefits erode. As a result, there is a natural inclination to want Terraform to control previously uncontrolled resources.
(Brief) Review of How Terraform Works
Before we go through how Terraform creates resources, it will be valuable to briefly review the major components of Terraform. These are Terraform Code, Terraform State, and Remote Cloud Resources.
- Terraform Code: This is HCL code that declaratively defines remote cloud resources.
- Remote Cloud Resources: These are the actual storage buckets, servers, databases, etc. from a cloud provider like AWS, Azure, GCP.
- Terraform State: Terraform state can be thought of, at a high level, as a required mapping of information between written Terraform code and the corresponding remote cloud resource. That is, it says: “This particular HCL code block corresponds to this particular remote s3 bucket”.
When we have uncontrolled cloud resources, we have the remote cloud resource, but are missing the corresponding Terraform code, as well as the appropriate state file updates linking Terraform code to the remote cloud resource.
To proceed, we have the following options:
Delete the Remote Cloud Resource and Recreate the Resource with Terraform
One option that involves no state file manipulation is to delete the remote cloud resource and recreate it identically via a regular Terraform workflow.
While perhaps easier for developers to execute, this is often not a great option; for storage resources, this can result in a loss of data, and for compute resources this can result in unwanted application downtime.
Manually Import an Uncontrolled Resource
- Identify the uncontrolled resource that you wish to bring into Terraform control.
- Write a corresponding piece of Terraform code in HCL that matches the uncontrolled resource.
- Identify the state file which should control the newly Terraformed resource.
- Write a state migration statement that updates the chosen state file to link the new Terraform code with the appropriate remote resource.
This involves running a migration statement via the CLI in a directory where terraform init has already been run.
This import migration command takes the following format:
Example:
We have an uncontrolled S3 bucket in AWS and want to bring it into Terraform control.
First we write the Terraform:
Then within a directory where we have already run terraform init, we run the state migration command:
Downsides with this Approach:
- Fairly manual process, from identification of remote cloud resources to running CLI commands
- There is no written record of state migrations run via the CLI. Since state migrations are notoriously finicky in Terraform, this can be risky.
- The Terraform code being linked to the remote resource must match the remote resource exactly. Otherwise, the next time terraform apply is run within that directory/workspace, changes will be made to the remote cloud resource.
Importing Uncontrolled Resources using dragondrop.cloud
dragondrop.cloud is a software service consumed as a container running within your cloud environment. It automates the lifecycle of identifying and importing uncontrolled resources into Terraform control by performing the following:
- Identifying cloud resources that are currently uncontrolled by Terraform ad-hoc or on a cron job schedule.
- Generating the needed HCL Terraform code and corresponding state migration commands.
- Selecting the most appropriate state file for controlling each identified cloud resource using machine learning.
- Outputting recommendations via a Pull Request into your Version Control System, providing a concrete record of all state migrations that were previously run via the CLI.
- Using the GitHub Action provided by dragondrop.cloud, migrations are only allowed to be merged into your code base if the HCL Terraform code identically matches the current remote cloud resource identically for an extra layer of security.
Downsides with this Approach:
- There is a bit of a start-up cost with dragondrop.cloud. It requires creating some cloud infrastructure in your cloud to host dragondrop, as well as configuring some environment variables on that created compute instance. Once set up, however all uncontrolled resources can be identified in one pull request, vs. the one-off nature of the manual approach described above.
dragondrop.cloud provides Terraform modules to spin up the infrastructure required for hosting its container, as well as client-side environment variable validation in their web application. This helps cap the onboarding process to less than an hour for the majority of customers.
Conclusion
We hope this helps you better understand options for bringing uncontrolled cloud infrastructure into Terraform control.
–
dragondrop.cloud’s mission is to automate developer best practices while working with Infrastructure as Code. Our flagship OSS product, cloud-concierge, allows developers to codify their cloud, detect drift, estimate cloud costs and security risks, and more — while delivering the results via a Pull Request. For enterprises running cloud-concierge at scale, we provide a management platform. To learn more, schedule a demo or get started today!
Learn More About Terraform
Everything Everywhere All as Code
“Everything as Code” Definition Everything as Code is a philosophy for managing IT infrastructure where all components of infrastructure are created, managed, and deleted using code. This applies to container definitions, cloud infrastructure, on-premise server...
Open Source driftctl Alternatives
What is driftctl? driftctl is an OSS CLI tool that enables users to identify Terraform drift as well as unmanaged resources within a cloud environment. It is a quite popular tool and has collected over two thousand stars on GitHub. Why Would We Want a Replacement?...
Why We Are Not Supporting OpenTF
Background On August 10, HashiCorp changed the license to their previously “Open Source” projects to a Business Source License (BSL), making them now “source available” for all future releases. We discusssed in detail reasons and motivations for this change here. On...