What is an ECS Fargate Task?

An ECS Fargate Task is serverless compute for long-running container executions. This is similar to GCP’s Cloud Run Job service, or Azure Container Instances.

Why ECS Fargate vs. Lambda w/Container?

Both services are marketed as serverless compute for containerized applications, and Lambda functions can directly be granted a function url, so why would ever need to use ECS Fargate Tasks and limit ourselves. The reason is that Lambda functions defined by containers do have significant have other limits:

• Lambda execution is time bound to only 15 minutes. Fargate tasks are not time-bound.
• Lambdas that reference containers require containers that implement the Lambda runtime. Fargate task containers do not.
• Lambda is limited to ECR hosted containers within the same region as the Lambda. Fargate task containers can be public images hosted by other services (like DockerHub or Google Artifact Repository)

Executing ECS Fargate Tasks via HTTP request

The Motivation

We’d like both strengths of Lambda and ECS Fargate Tasks: HTTPS triggerable, and long running containerized execution that can reference virtually any container. Furthermore, the consumer of the provided endpoint should be able to affect the ECS Fargate Task via different input parameters supplied through the HTTP request. Lastly, we don’t want to manage long-running servers to manage the HTTP requests, that is, we want to keep everything serverless.

Is this actually a real use case? Absolutely. Imagine a data processing service that a web application wants to offer for its users. The data processing service is dynamic. It depends on the input data to analyze, the type of analysis the customer wants to perform, and the user may want to execute it at any point in time. The data processing service is long-running, so it cannot be placed directly within an API, but we still want to be able to trigger it dynamically (like whenever a user clicks “Analyze Data”).

The Challenge

It is not possible to directly run ECS Fargate Tasks via an HTTPS request. While AWS allows us to trigger on a schedule or based on CloudWatch Events, we cannot send an https POST request to our task, for example, and have it spin up with a new configuration.

The Solution

We can create a solution satisfying all requirements with a Lambda function, and a Lambda function URL that serves as the endpoint for running our ECS Fargate task:

1) The Lambda function receives environment variable values within a POST request JSON Body.

2) Use Boto3 (or the corresponding AWS SDK, depending on your Lambda runtime) to run the ECS Fargate task with the appropriate

The lambda function can be defined with a single file, because our only import is Boto3, and Lambda includes the corresponding SDK pre-packaged, we do not need to configure any Layers etc.

More specifically, our solution is as follow:

a) The code for the lambda function:

b) The IAM role assigned to the Lambda must be able to both execute the desired ECS Fargate Task, but also pass the ECS Fargate Task’s IAM role.

c) Create a Lambda URL for the function, with any required security restrictions.

The Result

The consumer of the endpoint of this Lambda can now interact with a long-running, containerized job via an HTTP POST request and include specific information for that unique job execution in the request body! Furthermore, everything is serverless, keeping the costs of this solution as low as possible.

At dragondrop, we have an open-sourced implementation of the lambda function, along with supporting a supporting Terraform module that creates all of the requisite infrastructure to host our HTTPS triggerable long-running job within our customer’s cloud environments.

Conclusion

At present, there is no native way on AWS to trigger via HTTP POST request an ECS Fargate Task. In this post, however, we show how to build our own serverless HTTP endpoint for dynamically configuring and triggering ECS Fargate jobs using a Lambda function. This makes it much easier for other authorized services to kick off the ECS Fargate Task, and all that is required is a new lambda function and a splash of code.

If you have any questions or suggestions as to how we can improve this post, please comment below.

–

dragondrop.cloud’s mission is to automate developer best practices while working with Infrastructure as Code. Our flagship product regularly scans and identifies resource changes that have occurred outside of a Terraform workflow (e.g. drift) so that dev teams can have a Cloud environment that is fully represented as code. All of our tools are self-hosted by our customers, with no data ever leaving their servers. To learn more, schedule a demo or get started today!