Fundamentals of Hermetic Builds

Fundamentals of Hermetic Builds

Master hermetic builds with free flashcards and spaced repetition practice. This lesson covers reproducibility principles, dependency isolation, and environment consistency—essential concepts for reliable software development and deployment pipelines.

Welcome 💻

Have you ever heard a developer say "it works on my machine" when software fails in production? Or struggled to rebuild an old project because dependencies changed? Hermetic builds solve these problems by creating completely self-contained, reproducible build environments.

In this lesson, you'll learn what makes a build hermetic, why hermetic builds are critical for modern software development, and how to implement hermetic principles in your own projects. Whether you're working on a small application or managing complex distributed systems, understanding hermetic builds will transform how you think about software reproducibility.

What is a Hermetic Build? 🔒

A hermetic build is a build process that is:

1. Completely self-contained - All dependencies are explicitly declared and versioned
2. Deterministic - The same inputs always produce the same outputs
3. Isolated - No dependence on the host system's state or configuration
4. Reproducible - Anyone can rebuild it with identical results, anywhere, anytime

The term "hermetic" comes from Hermes Trismegistus and "hermetically sealed" - meaning airtight and completely isolated from the outside environment. Just as a hermetically sealed container prevents air from entering, a hermetic build prevents external factors from affecting the build process.

💡 Key Insight: A truly hermetic build should produce byte-for-byte identical outputs when given the same inputs, regardless of when or where it runs.

Core Principles of Hermetic Builds 🎯

1. Explicit Dependencies 📦

Every dependency must be:

• Explicitly declared in configuration files
• Version-pinned to specific releases (not "latest" or version ranges)
• Content-addressed using checksums or hashes
• Stored reliably in artifact repositories

❌ Non-Hermetic	✅ Hermetic
npm install express	npm install express@4.18.2 + lockfile
Uses system Python	Downloads Python 3.11.4 from verified source
Pulls Docker tag latest	Uses digest @sha256:abc123...

2. Environment Isolation 🏝️

The build must not depend on:

• System environment variables (except explicitly passed ones)
• Global installations (system packages, tools)
• Network access during build (all deps fetched beforehand)
• Filesystem state outside the build directory
• Current time/date (unless explicitly needed and controlled)
• Random number generators with unpredictable seeds

┌────────────────────────────────────────────┐
│         HERMETIC BUILD SANDBOX             │
│                                            │
│  ┌──────────────────────────────────┐     │
│  │  Build Process                   │     │
│  │  ┌─────────┐   ┌─────────┐      │     │
│  │  │ Source  │   │  Deps   │      │     │
│  │  │  Code   │   │(pinned) │      │     │
│  │  └────┬────┘   └────┬────┘      │     │
│  │       │             │            │     │
│  │       └──────┬──────┘            │     │
│  │              ↓                   │     │
│  │         ┌─────────┐              │     │
│  │         │ Build   │              │     │
│  │         │ Output  │              │     │
│  │         └─────────┘              │     │
│  └──────────────────────────────────┘     │
│                                            │
│  ❌ No system access                      │
│  ❌ No network during build               │
│  ❌ No global state                       │
└────────────────────────────────────────────┘

3. Deterministic Outputs 🎲

The build process must produce identical results every time:

• No timestamps embedded in artifacts (unless from source control)
• Sorted file lists (filesystem order varies by OS)
• Fixed ordering in archives, manifests, and metadata
• Reproducible randomness (fixed seeds when needed)
• Normalized paths (no absolute paths that vary by machine)

4. Content Addressing 🔑

Dependencies are referenced by their content hash rather than just name and version:

Reference Method	Example	Hermetic?
Name only	lodash	❌ Version unspecified
Name + version	lodash@4.17.21	⚠️ Version could be republished
Name + version + hash	lodash@4.17.21 + SHA256	✅ Content verified

💡 Pro Tip: Modern package managers like npm (package-lock.json), Cargo (Cargo.lock), and Go modules (go.sum) automatically track content hashes.

Why Hermetic Builds Matter 🌟

Reproducibility Benefits

Security Auditing: If you can reproduce a build from 6 months ago, you can verify whether a security vulnerability was present.

Debugging: Reproduce the exact binary that failed in production to debug issues.

Compliance: Prove that released software matches audited source code.

Collaboration: Every team member gets identical build outputs.

The Cost of Non-Hermetic Builds

THE DEPENDENCY DRIFT PROBLEM

Day 1:        Day 30:       Day 90:
┌─────┐      ┌─────┐       ┌─────┐
│Build│      │Build│       │Build│
│ ✅  │      │ ⚠️  │       │ ❌  │
└──┬──┘      └──┬──┘       └──┬──┘
   │            │              │
   ↓            ↓              ↓
lib@1.0.0   lib@1.0.1    lib@2.0.0
(works)     (warning)    (breaking)

Without version pinning, dependencies
change underneath you!

Core Concepts in Detail 🔍

Build Caching 🗄️

Hermetic builds enable safe caching because:

1. Input hashing: Hash all inputs (source + dependencies)
2. Cache lookup: Check if this input hash was built before
3. Cache hit: Reuse previous output if hash matches
4. Cache miss: Build and store result with input hash

CACHE DECISION FLOW

  📂 Source Code + 📦 Dependencies
           |
           ↓
    🔑 Hash Inputs
           |
           ↓
    🔍 Check Cache
           |
     ┌─────┴─────┐
     ↓           ↓
  ✅ Hit      ❌ Miss
     |           |
     ↓           ↓
  Reuse      Build It
  Output         |
                 ↓
            Store in Cache

Why hermetic builds make caching safe: Non-hermetic builds might read system state or network resources, so cached outputs could be wrong. Hermetic builds guarantee that identical inputs = identical outputs.

Sandboxing 📦

Build tools enforce isolation through:

Containerization: Docker, Podman isolate filesystem and network

Virtual environments: Python venv, Node.js node_modules, Ruby bundler

Build sandboxes: Bazel sandbox, Nix store paths

Tool	Isolation Method	Hermetic Level
Bazel	Sandboxed execution + explicit deps	🟢 High
Nix	Functional package management	🟢 High
Docker	Container isolation	🟡 Medium (if used correctly)
Make	Dependency tracking only	🔴 Low

The Dependency Graph 📊

Hermetic builds require understanding the complete dependency graph:

     Your Application
           │
           ↓
    ┌──────┴──────┐
    │             │
    ↓             ↓
  Library A    Library B
    │             │
    ↓             ↓
  Library C    Library C  ← Diamond dependency
    │             │
    └──────┬──────┘
           ↓
    (Must resolve to same version)

Transitive dependencies are dependencies of your dependencies. Hermetic builds must pin these too!

Real-World Examples 🌍

Example 1: Non-Hermetic Docker Build ❌

FROM python:latest
RUN apt-get update && apt-get install -y build-essential
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

Problems:

• python:latest changes over time
• apt-get update fetches latest package lists
• pip install without lockfile installs latest compatible versions
• Different results if built today vs. next month

Example 1: Hermetic Docker Build ✅

FROM python@sha256:a4b5c6d7e8f9...
RUN apt-get update && apt-get install -y \
    build-essential=12.9ubuntu3 \
  && rm -rf /var/lib/apt/lists/*
COPY requirements.txt requirements-lock.txt .
RUN pip install --no-deps -r requirements-lock.txt \
  && pip check
COPY app.py .
CMD ["python", "app.py"]

Improvements:

• Base image pinned to SHA256 digest
• Package versions explicitly specified
• Lockfile with exact versions and hashes
• --no-deps ensures no surprise transitive deps
• Reproducible across time and machines

Example 2: JavaScript Build Evolution 🔄

Stage 1: Non-Hermetic

{
  "dependencies": {
    "express": "^4.0.0",
    "lodash": "*"
  }
}

⚠️ Caret (^) and star (*) allow version updates

Stage 2: Better (with lockfile)

npm install  # Generates package-lock.json

💡 package-lock.json pins exact versions and integrity hashes

Stage 3: Fully Hermetic Build Script

#!/bin/bash
set -euo pipefail

## Verify lockfile matches package.json
npm ci  # Clean install from lockfile only

## Build with fixed NODE_ENV
NODE_ENV=production npm run build

## Verify output hash
sha256sum dist/bundle.js > dist/bundle.js.sha256

Key difference: npm ci fails if lockfile is out of sync, ensuring consistency.

Example 3: Bazel Build Configuration ⚙️

Bazel is designed for hermetic builds from the ground up:

## BUILD file
load("@rules_python//python:defs.bzl", "py_binary")

py_binary(
    name = "app",
    srcs = ["app.py"],
    deps = [
        "@pip_deps//flask",  # From locked requirements
        "//lib:utils",       # Internal dependency
    ],
    python_version = "PY3",
)

Hermetic features:

• Explicit dependency declarations
• Sandboxed execution (process isolation)
• Content-based caching
• Remote execution support
• Reproducible across machines

Example 4: Go Modules 🐹

// go.mod
module github.com/user/project

go 1.21

require (
    github.com/gin-gonic/gin v1.9.1
    github.com/lib/pq v1.10.9
)

// go.sum (auto-generated)
github.com/gin-gonic/gin v1.9.1 h1:4idEAncQnU5cB7BeOkPtxjfCSye0AAm1R0RVIqJ+Jmg=
github.com/gin-gonic/gin v1.9.1/go.mod h1:hPrL7YrpYKXt5YId3A/Tnip5kqbEAP+KLuI3SUcPTeU=

Hermetic properties:

• go.mod declares direct dependencies
• go.sum contains cryptographic hashes of all deps (including transitive)
• go build verifies checksums before building
• Module proxy caches ensure availability

Common Mistakes ⚠️

Mistake 1: Using Floating Tags/Versions

❌ Problem:

image: nginx:latest

✅ Solution:

image: nginx@sha256:4c0fdaa8b6341bfdeca5f18f7837462c80cff90527ee35ef185571e1c327beac

Mistake 2: Network Access During Build

❌ Problem:

build:
	curl https://example.com/latest-config.json > config.json
	gcc -o app main.c

✅ Solution:

## Fetch dependencies in separate step
fetch:
	curl https://example.com/config-v1.2.3.json > config.json
	sha256sum -c config.json.sha256

build: config.json
	# Build without network access
	gcc -o app main.c

Mistake 3: Timestamp Embedding

❌ Problem:

VERSION = f"1.0.0-{datetime.now().isoformat()}"

Building at different times produces different outputs!

✅ Solution:

## Use Git commit info (deterministic)
VERSION = f"1.0.0-{os.getenv('GIT_COMMIT_SHA', 'dev')}"

Mistake 4: System Tool Dependencies

❌ Problem:

## Assumes system has gcc installed
gcc -o app main.c

What version? What architecture?

✅ Solution:

## Specify exact compiler version
FROM gcc:11.3.0 AS builder
WORKDIR /build
COPY main.c .
RUN gcc -o app main.c

Mistake 5: Mutable Remote Resources

❌ Problem:

## Pip install from Git branch (mutable)
pip install git+https://github.com/user/repo.git@main

✅ Solution:

## Pin to specific commit hash
pip install git+https://github.com/user/repo.git@a1b2c3d4e5f6...

Mistake 6: Build Output Varies by Filesystem

❌ Problem:

## File order depends on filesystem
tar czf archive.tar.gz *

✅ Solution:

## Sort files explicitly
find . -type f | sort | tar czf archive.tar.gz -T -

Tools and Ecosystem 🛠️

Tool	Purpose	Hermetic Features
Bazel	Multi-language build system	Sandboxing, content addressing, remote caching
Nix	Package manager	Purely functional, isolated store paths
Docker	Containerization	Image layers, digest addressing
Pants	Python/JVM build system	Hermetic by default, fine-grained caching
Buck2	Meta's build system	Virtual filesystem, action caching

Implementing Hermetic Builds: Step-by-Step 📝

Step 1: Audit Current Build

• List all dependencies (direct and transitive)
• Identify system dependencies
• Find network calls during build
• Check for embedded timestamps/randomness

Step 2: Pin All Dependencies

• Use lockfiles (package-lock.json, Cargo.lock, go.sum)
• Pin base images to digests
• Version system tools

Step 3: Isolate the Build Environment

• Use containers or sandboxes
• Disable network during build phase
• Set fixed environment variables

Step 4: Make Outputs Deterministic

• Remove timestamps (or use from source control)
• Sort file lists
• Normalize paths
• Use fixed random seeds if needed

Step 5: Verify Reproducibility

• Build on different machines
• Build at different times
• Compare output hashes
• Automate verification in CI/CD

Key Takeaways 🎓

✅ Hermetic builds are self-contained, deterministic, and reproducible

✅ Pin all dependencies with exact versions and content hashes

✅ Isolate builds from system state and network resources

✅ Ensure outputs are deterministic (same inputs → same outputs)

✅ Use modern tools designed for hermeticity (Bazel, Nix, proper lockfiles)

✅ Hermetic builds enable safe caching, easier debugging, and better security

✅ Avoid floating tags, mutable resources, and embedded timestamps

✅ Test reproducibility by building in different environments

🤔 Did You Know?

Google builds over 2 billion builds per week using their hermetic build system (Blaze, the predecessor to Bazel). Every build is reproducible and cached, allowing engineers to reuse build outputs across the entire organization. This saves massive amounts of compute time and ensures consistency across thousands of engineers!

📚 Further Study

• Bazel Build System Documentation - Comprehensive guide to hermetic builds with Bazel
• Reproducible Builds Project - Tools and practices for verifiable, reproducible software
• Docker Build Best Practices - Creating reproducible container images