In the world of software development, Continuous Integration (CI) and Continuous Deployment (CD) have become essential practices.
They help teams deliver code faster, with fewer bugs, and with greater confidence. However, as projects grow in complexity, so do the build pipelines.
This is where Earthly and its Earthfile come into play. Earthly is a modern build tool that simplifies and accelerates the CI/CD process, making it easier to manage complex builds. In this article, we’ll explore what Earthly is, how Earthfile works, and how it compares to Dockerfile.
In a previous article, I wrote about Helmfile and how it can be useful in container deployment. This article is on Earthfile and how it manages CI/CD workflow. Let us start by defining what Earthly is.
What is Earthly?
Earthly is an open-source build automation tool designed to make CI/CD pipelines more efficient, reproducible, and portable.
It combines the best aspects of Docker and Makefile to create a unified build process that works across different environments.
Earthly uses a declarative syntax called Earthfile, which is similar to Dockerfile but with additional features tailored for build automation.
Earthly is particularly useful for:
– Simplifying complex build pipelines.
– Ensuring builds are reproducible across different environments.
– Reducing build times by leveraging caching and parallelism.
– Integrating seamlessly with existing CI/CD tools like GitHub Actions, GitLab CI, and Jenkins.
What is an Earthfile?
An Earthfile is a configuration file used by Earthly to define build steps. It looks similar to a Dockerfile but is more powerful and flexible. Earthfile allows you to define targets (similar to Makefile targets) and dependencies between them. Each target can include commands, copy files, and even run containers.
Here’s a simple example of an Earthfile:
# Earthfile
VERSION 0.7 # Base target FROM golang:1.20-alpine WORKDIR /app # Build target build: COPY main.go . RUN go build -o myapp main.go SAVE ARTIFACT myapp AS LOCAL ./myapp # Test target test: COPY --from=build /app/myapp . RUN ./myapp --test # Default target default: BUILD +build
In the example above:
– The `FROM` command specifies the base image (similar to Dockerfile).
– The `build` target compiles a Go application.
– The `test` target runs tests using the compiled binary.
– The `default` target is the entry point and builds the `build` target.
Key Features of Earthfile
1. Target-Based Builds:
One of the most powerful features of Earthly is its ability to define multiple targets within a single Earthfile. Each target represents a specific build step or a set of related steps, allowing you to organize your build logic in a modular and reusable way. This approach not only improves readability but also enhances maintainability and scalability of your build process. Let’s dive deeper into how Earthfile targets work and why they are so beneficial.
In Earthly, a target is a named block of instructions that performs a specific task, such as building a binary, running tests, or packaging an application. Targets are similar to functions in programming—they encapsulate logic, can accept inputs, and can be reused across different parts of the build process.
2. Caching:
Earthly’s automatic caching mechanism is a game-changer for optimizing build performance, as it caches the results of each build step and reuses them in subsequent runs, significantly reducing build times and resource consumption. This capability is especially valuable in CI/CD (Continuous Integration/Continuous Deployment) pipelines, where builds are executed frequently, and efficiency is critical to maintaining rapid development cycles. By caching steps such as dependency installation, code compilation, and testing, Earthly ensures that only the parts of the build process affected by changes are re-executed, while unchanged steps are reused from the cache. For example, if a developer modifies a single source file, Earthly will re-run only the steps dependent on that file, skipping the rest and leveraging cached results. This not only speeds up builds but also reduces the load on CI/CD infrastructure, saving time and costs. Additionally, Earthly’s caching ensures build consistency and reproducibility, as the same cached results are used across different environments, eliminating discrepancies between local and remote builds. By enhancing efficiency, reliability, and scalability, Earthly’s caching mechanism makes it an indispensable tool for modern CI/CD workflows, enabling teams to deliver software faster and more reliably.
3. Parallel Execution:
Earthly’s ability to execute independent targets in parallel is a powerful feature that significantly speeds up the overall build process by leveraging concurrency. When multiple targets in an Earthfile do not depend on each other, Earthly can run them simultaneously, rather than sequentially, which is especially beneficial in complex projects with numerous build steps. For example, if you have separate targets for linting, unit testing, and integration testing, and none of these targets depend on the others, Earthly can execute them in parallel, reducing the total build time. This parallelism is particularly advantageous in CI/CD pipelines, where faster builds lead to quicker feedback loops and improved developer productivity. Additionally, Earthly’s parallel execution works seamlessly with its caching mechanism, ensuring that only the necessary steps are executed while reusing cached results for unchanged steps. By optimizing resource utilization and minimizing idle time, Earthly’s parallel execution capability makes it an efficient and scalable solution for modern build automation, enabling teams to deliver software faster and more reliably.
4. Artifact Management:
Earthly’s SAVE ARTIFACT command is a powerful feature that enables the saving and sharing of artifacts—such as binaries, libraries, or configuration files—between targets, making it easier to create modular and reusable build pipelines. When a target produces an artifact, such as a compiled binary or a packaged application, the SAVE ARTIFACT command allows you to store that artifact and make it available to other targets or even export it to the local filesystem. For example, a build target might compile a Go application and save the resulting binary as an artifact, which can then be used by a test target for running tests or a package target for creating a deployable bundle. This eliminates the need to rebuild or duplicate artifacts across targets, ensuring consistency and efficiency. Additionally, artifacts can be shared across different Earthfiles or even different projects, promoting reusability and reducing redundancy. By facilitating the seamless flow of artifacts between targets, Earthly simplifies complex build processes, enhances collaboration, and ensures that each step in the pipeline has access to the necessary resources, ultimately leading to faster and more reliable builds.
5. Reproducibility:
Earthly ensures build reproducibility by isolating each step of the build process in a containerized environment, which guarantees that builds are consistent and independent of the host system’s state. By running each step inside a container, Earthly encapsulates the build environment, including dependencies, tools, and configurations, preventing inconsistencies caused by differences in local setups or external factors. This isolation ensures that the same Earthfile will produce identical results regardless of where it is executed—whether on a developer’s machine, a CI/CD server, or a production environment. For example, if a build step requires a specific version of a compiler or library, Earthly ensures that the exact version is used every time, eliminating the “it works on my machine” problem. Additionally, Earthly’s containerized approach integrates seamlessly with Docker, leveraging its ecosystem for managing images and dependencies. By combining containerization with its caching and parallel execution features, Earthly provides a robust and reliable build system that is not only reproducible but also efficient and scalable, making it an ideal choice for modern software development workflows.
6. Integration with Docker:
Earthly seamlessly integrates with Docker, allowing you to use Docker images as base layers for your builds and even export the final build outputs as Docker images, making it a versatile tool for containerized workflows. By leveraging Docker images as base layers, Earthly ensures that your builds start from a consistent and well-defined environment, reducing the risk of discrepancies between development, testing, and production. For example, you can use an official Python or Node.js Docker image as the foundation for your build, ensuring that all dependencies and tools are preconfigured and standardized. Once the build process is complete, Earthly can export the final output—such as a compiled application or a packaged service—as a Docker image, ready to be deployed to any container orchestration platform like Kubernetes or Docker Swarm. This tight integration with Docker not only simplifies the build and deployment process but also enhances portability, as the resulting images can be run anywhere Docker is supported. By combining the reproducibility and isolation of Earthly with the flexibility and ecosystem of Docker, Earthly provides a powerful solution for building, testing, and deploying containerized applications efficiently and reliably.
Earthfile vs Dockerfile: What’s the Difference?
While Earthfile and Dockerfile share some similarities, they serve different purposes and have distinct features:
Use Cases for Earthfile
1. Multi-Language Projects:
Earthly is ideal for projects that use multiple programming languages. For example, you can compile a Go backend, build a React frontend, and run tests in a single Earthfile.
2. Complex CI/CD Pipelines:
If your CI/CD pipeline involves multiple steps (e.g., linting, testing, building, and deploying), Earthly can simplify and streamline the process.
3. Reproducible Builds:
Earthly ensures that builds are consistent across different environments, reducing the “it works on my machine” problem.
4. Faster Builds:
By leveraging caching and parallel execution, Earthly can significantly reduce build times.
Sample Earthfile for a CI Pipeline
Let’s say you have a Node.js project with the following steps:
1. Install dependencies.
2. Lint the code.
3. Run tests.
4. Build the application.
Here’s how you can define this in an Earthfile:
VERSION 0.7 FROM node:16 WORKDIR /app # Install dependencies deps: COPY package.json package-lock.json . RUN npm install # Lint the code lint: COPY --from=deps /app . COPY .eslintrc.js . RUN npm run lint # Run tests test: COPY --from=deps /app . COPY . . RUN npm test # Build the application build: COPY --from=deps /app . COPY . . RUN npm run build SAVE ARTIFACT dist AS LOCAL ./dist # Default target default: BUILD +deps BUILD +lint BUILD +test BUILD +build
How Earthly Compares to Dockerfile
While Dockerfile is great for creating container images, it lacks the flexibility and power needed for complex build pipelines. Earthly fills this gap by providing:
– Targets: Organize build steps into reusable targets.
– Caching: Speed up builds with automatic caching.
– Parallelism: Execute independent steps in parallel.
– Artifact Management: Easily share artifacts between steps.
For example, in a Dockerfile, you’d typically define a single build process. In contrast, an Earthfile allows you to define multiple targets (e.g., `deps`, `lint`, `test`, `build`) and manage dependencies between them.
How to Execute an Earthfile
Earthly allows you to build all the steps at the same time, or build a single step. But first the earthly CLI needs to be installed to be able to run builds.
To install earthly on Mac use the following command
brew install earthly earthly bootstrap
Linux and Windows users can follow the installation instructions provided in the Earthly documentation.
Once Earthly is installed, you can run an Earthfile by navigating to its directory and executing the desired target. Earthly uses a syntax similar to Makefile, where targets are defined within the Earthfile. To run a specific target, use the following command:
earthly +<target>
Using our example above, if we want to run only the deps section of the Earthfile we can simply do this
earthly +deps
The screenshot above shows a successful run of Earthly on an Earthfile to run npm install which is a stage defined in the Earthfile. Similar to Dockerfile, the package.json was located in the root directory with the Earthfile
Conclusion
Earthly and its Earthfile offer a modern approach to build automation, making CI/CD pipelines faster, more efficient, and easier to manage.
While Dockerfile is excellent for creating container images, Earthly goes a step further by providing features like targets, caching, and parallelism, which are essential for complex builds.
