Making a Consumable HTTP Endpoint for Cloud Run Jobs

What are Cloud Run jobs?
Cloud Run jobs are GCP’s compute service for executing long-running containerized workloads from start to finish. More succinctly, they are serverless compute for containerized jobs. This functionality is similar to what AWS offers through its Fargate service.
Why Cloud Run jobs vs. Cloud Run services?
Both services are marketed as serverless compute for containerized applications, and Cloud Run services have been around for a long time, so why do we need something like Cloud Run jobs in the first place?
Cloud Run services provide compute for containerized APIs that continuously listens for and serves responses to HTTP requests. On the other hand, Cloud Run jobs provide compute for an executable container, and when that container executes, it only runs once from start until termination. Importantly, Cloud Run jobs do not accept HTTP requests, and instead are executed via either gcloud commands, the Cloud Run jobs API, or Cloud Scheduler.
Because of these differences, Cloud Run jobs fill the essential role of supporting long-running containerized tasks. Think, model-training tasks, parallel data processing jobs, or a single, long-running batch data engineering pipeline. These are the types of tasks which should not be run within a Cloud Run hosted API due to memory and timeout constraints. Doing so would at best result in slow task execution, and at worst, knock down your API.
Executing Cloud Run jobs via HTTP request
The Motivation
We want to be able to execute our Cloud Run jobs via a simple HTTP request. Furthermore, the consumer of the provided endpoint should be able to affect the Cloud Run job via different input parameters supplied through the HTTP request. Lastly, we don’t want to manage long-running servers to manage the HTTP requests, that is, we want to keep everything serverless.
Is this actually a real use case? Absolutely. Imagine a data processing service that a web application wants to offer for its users. The data processing service is dynamic. It depends on the input data to analyze, the type of analysis the customer wants to perform, and the user may want to execute it at any point in time. The data processing service is long-running, so it cannot be placed directly within an API, but we still want to be able to trigger it dynamically (like whenever a user clicks “Analyze Data”).
The Challenge
Google Cloud explicitly notes that Cloud Run jobs cannot serve HTTP requests. The only ways that a Cloud Run job can be natively executed through the following methods:
- Running the `gcloud beta run jobs execute {JOBNAME} — region={REGION}` command
- Sending a POST request to the jobs.run API method
- Executing the job on a Cloud Scheduler trigger
None of these methods satisfy both our dynamic triggering and custom parameterization requirements.
The Solution
We can create a solution satisfying all requirements with a new Cloud Run service with a single post-request endpoint that performs the following:
1) Runs a gcloud command to update the environment variables on the Cloud Run job with the values passed to the Cloud Run endpoint
2) Runs a gcloud command to kick off the desired Cloud Run job
This Cloud run service can be defined by three files (in our example we are using a Python Flask API within Cloud Run, but you can adapt the logic of the solution to any other language’s API server in your container).
- The Dockerfile for the Cloud Run Service:

2. The application factory in app.py

3. And most interestingly the endpoint in root.py

The Result
The consumer of the endpoint from this Cloud Run Service can now interact with a long-running, containerized job via an HTTP POST request and include specific information for the job in the request body! Furthermore, everything is serverless, keeping the costs of this solution as low as possible.
At dragondrop, we have an open-sourced implementation of this solution, along with supporting a supporting Terraform module that creates all of the requisite infrastructure to host our HTTP triggerable long-running job within our customer’s cloud environments.
Conclusion
At present, there is no native way on Google Cloud to trigger a Cloud Run job via HTTP request. This precludes native Cloud Run jobs from being used within several key scenarios. In this post, however, we show how to build our own serverless HTTP endpoint for dynamically triggering and configuring Cloud Run jobs. This makes it much easier for other services to kick off the Cloud Run job, and all that is required is a new Cloud Run service instance and a splash of code.
–
dragondrop.cloud’s mission is to automate developer best practices while working with Infrastructure as Code. Our flagship OSS product, cloud-concierge, allows developers to codify their cloud, detect drift, estimate cloud costs and security risks, and more — while delivering the results via a Pull Request. For enterprises running cloud-concierge at scale, we provide a management platform. To learn more, schedule a demo or get started today!
Learn More About Google Cloud (GCP)
Google Kubernetes Engine (GKE) vs. Cloud Run
What is Google Kubernetes Engine (GKE)? Kubernetes (K8s) is an extremely popular, open source, container orchestration platform. GKE is a managed offering for running Kubernetes clusters in GCP. GKE handles a lot of the heavy lifting, fully managing the K8s cluster...
AWS Fargate Tasks vs. GCP Cloud Run Jobs
When it comes to running containerized, long-duration workloads in the cloud, AWS Fargate Tasks and GCP Cloud Run Jobs are two options worth considering. AWS Fargate Tasks have been around for quite some time in cloud years (since 2018 ), while GCP Cloud Run Jobs were...
Securely Accessing Our Google Cloud SQL Instances
Securely accessing Google Cloud SQL instances while minimizing system complexity and cost.