Hermetic CI/CD Pipelines

Hermetic CI/CD Pipelines

Master hermetic CI/CD pipelines with free flashcards and spaced repetition practice. This lesson covers containerized build environments, dependency isolation, reproducible deployments, and cache invalidation strategies—essential concepts for ensuring consistent, reliable production releases.

Welcome 🚀

Welcome to the world of hermetic CI/CD pipelines! If you've ever experienced the frustration of a build that works on one machine but fails on another, or a deployment that succeeds in staging but crashes in production, you've encountered the problems that hermetic builds solve. In this lesson, we'll explore how to create CI/CD pipelines that are completely self-contained, reproducible, and isolated from their environment.

A hermetic build is one that depends only on explicitly declared inputs and produces identical outputs regardless of where or when it runs. When applied to CI/CD pipelines, this principle ensures that your deployments are predictable, debuggable, and safe. We'll dive deep into the techniques, tools, and patterns that make this possible.

Core Concepts 💡

What Makes a Pipeline Hermetic?

A truly hermetic CI/CD pipeline has four critical characteristics:

1. Input Isolation 🔒
All inputs to the build and deployment process must be explicitly declared and versioned. This includes:

• Source code (specific commits, not "latest")
• Dependencies (pinned versions, not ranges)
• Build tools (containerized, not system-installed)
• Configuration files (versioned, not environment variables)
• External data (checksummed artifacts, not live downloads)

2. Environment Reproducibility 🔄
The build environment must be identical across all runs:

• Same operating system version
• Same installed packages
• Same environment variables
• Same filesystem state
• Same network isolation

3. Output Determinism 🎯
Given the same inputs, the pipeline must produce byte-for-byte identical outputs:

• No timestamps in artifacts
• No random identifiers
• No order-dependent operations
• No undeclared dependencies

4. Side-Effect Freedom 🧹
The pipeline must not depend on or modify external state:

• No shared filesystems
• No global caches (unless explicitly managed)
• No network calls to unversioned resources
• No reliance on system time or locale

The Hermetic Build Container

The foundation of a hermetic CI/CD pipeline is the build container—a Docker image that contains everything needed to build and deploy your application:

┌─────────────────────────────────────────────┐
│         HERMETIC BUILD CONTAINER            │
├─────────────────────────────────────────────┤
│                                             │
│  📦 Base OS (pinned version)               │
│  ├─ Debian 11.6, not "latest"              │
│  └─ SHA256 checksum verified               │
│                                             │
│  🔧 Build Tools (explicit versions)        │
│  ├─ Node.js 18.16.0                        │
│  ├─ Go 1.20.4                              │
│  └─ Python 3.11.3                          │
│                                             │
│  📚 Dependencies (locked)                   │
│  ├─ package-lock.json                      │
│  ├─ go.sum                                 │
│  └─ requirements.txt (with hashes)         │
│                                             │
│  ⚙️ Configuration (baked in)               │
│  ├─ Build flags                            │
│  ├─ Compiler settings                      │
│  └─ Deployment scripts                     │
│                                             │
└─────────────────────────────────────────────┘

Dependency Management in Hermetic Pipelines

Proper dependency management is crucial for hermetic builds. Here's how different ecosystems handle it:

Ecosystem	Lock File	Verification	Hermetic Tool
Node.js	package-lock.json	SHA-512 integrity	npm ci, pnpm
Python	requirements.txt + hashes	SHA-256 checksums	pip-tools, poetry
Go	go.sum	Module checksums	go mod vendor
Rust	Cargo.lock	Crate checksums	cargo build
Java	gradle.lockfile	Artifact verification	Gradle, Maven

💡 Pro Tip: Always commit your lock files to version control! They're the key to reproducibility.

Cache Invalidation Strategy

Caching is essential for fast builds, but it must be hermetic. The cache key should be a cryptographic hash of all inputs:

CACHE KEY COMPOSITION
┌─────────────────────────────────────────┐
│                                         │
│  🔑 Hash(                               │
│     source_code_commit_sha,            │
│     dependency_lock_file_hash,         │
│     build_tool_versions,               │
│     build_script_content,              │
│     compiler_flags                     │
│  )                                      │
│                                         │
│  ↓                                      │
│                                         │
│  sha256:a3f9c8d7e2b4...                │
│                                         │
└─────────────────────────────────────────┘

  If ANY input changes → Cache MISS
  If ALL inputs same → Cache HIT

⚠️ Common Mistake: Using timestamps or "latest" tags in cache keys breaks hermeticity!

The Hermetic Pipeline Architecture

A complete hermetic CI/CD pipeline follows this pattern:

┌────────────────────────────────────────────────────────┐
│              HERMETIC CI/CD PIPELINE                   │
└────────────────────────────────────────────────────────┘

    📝 Code Commit (SHA: abc123)
           │
           ↓
    ┌──────────────────┐
    │  1. BUILD IMAGE  │  ← Dockerfile with pinned versions
    │     PREPARATION  │
    └────────┬─────────┘
             │
             ↓
    ┌──────────────────┐
    │  2. DEPENDENCY   │  ← Install from lock files only
    │     RESOLUTION   │  ← Verify checksums
    └────────┬─────────┘
             │
             ↓
    ┌──────────────────┐
    │  3. COMPILATION  │  ← Deterministic flags
    │     & BUILD      │  ← No timestamps
    └────────┬─────────┘
             │
             ↓
    ┌──────────────────┐
    │  4. TESTING      │  ← Isolated test environment
    │                  │  ← No external services
    └────────┬─────────┘
             │
             ↓
    ┌──────────────────┐
    │  5. ARTIFACT     │  ← Sign with checksum
    │     CREATION     │  ← Store immutably
    └────────┬─────────┘
             │
             ↓
    ┌──────────────────┐
    │  6. DEPLOYMENT   │  ← Pull exact artifact
    │                  │  ← Verify signature
    └──────────────────┘
             │
             ↓
    🚀 Production (reproducible)

Container Layer Caching

Docker's layer caching can speed up builds while maintaining hermeticity:

Layer Structure for Optimal Caching:

## Layer 1: Base OS (changes rarely)
FROM debian:11.6-slim@sha256:abc123...

## Layer 2: System packages (changes rarely)
RUN apt-get update && apt-get install -y \
    ca-certificates=20210119 \
    curl=7.74.0-1.3 \
    && rm -rf /var/lib/apt/lists/*

## Layer 3: Build tools (changes occasionally)
COPY --from=golang:1.20.4@sha256:def456... /usr/local/go /usr/local/go

## Layer 4: Dependencies (changes frequently)
COPY go.mod go.sum ./
RUN go mod download && go mod verify

## Layer 5: Source code (changes most frequently)
COPY . .

## Layer 6: Build
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o app

💡 Optimization Strategy: Order layers from least to most frequently changing to maximize cache hits.

Handling External Dependencies

External dependencies are the biggest threat to hermeticity. Here's how to manage them:

1. Vendoring 📦
Copy all dependencies into your repository:

go mod vendor          # Go
npm ci --cache .cache  # Node.js with local cache
pip download -r requirements.txt -d ./wheels  # Python

2. Private Mirrors 🪞
Host your own package registries:

• Artifactory
• Nexus
• Private npm registry
• PyPI mirror

3. Content-Addressable Storage 🔐
Reference dependencies by hash, not version:

{
  "dependencies": {
    "lodash": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz#sha512:abc..."
  }
}

Real-World Examples 🌍

Example 1: Node.js Hermetic Pipeline

Let's build a complete hermetic pipeline for a Node.js application:

Dockerfile (hermetic build container):

## Use specific SHA digest, not tags
FROM node:18.16.0-alpine3.17@sha256:a1b2c3d4e5f6...

## Install specific versions of system dependencies
RUN apk add --no-cache \
    git=2.38.4-r1 \
    python3=3.10.11-r0 \
    make=4.3-r1

## Set reproducible environment
ENV NODE_ENV=production \
    NPM_CONFIG_LOGLEVEL=error \
    # Disable npm update checks
    NPM_CONFIG_UPDATE_NOTIFIER=false \
    # Use deterministic install order
    NPM_CONFIG_PREFER_OFFLINE=true

## Create app directory
WORKDIR /build

## Copy dependency manifests first (for layer caching)
COPY package.json package-lock.json ./

## Install exact versions from lock file
## npm ci is hermetic, npm install is not!
RUN npm ci --only=production --ignore-scripts

## Copy source code
COPY . .

## Build with deterministic output
RUN npm run build

## Create minimal runtime image
FROM node:18.16.0-alpine3.17@sha256:a1b2c3d4e5f6...
WORKDIR /app
COPY --from=0 /build/dist ./dist
COPY --from=0 /build/node_modules ./node_modules
COPY package.json ./

USER node
CMD ["node", "dist/index.js"]

.gitlab-ci.yml (hermetic pipeline definition):

variables:
  # Pin Docker version
  DOCKER_VERSION: "24.0.2"
  # Use content-addressable image references
  BUILD_IMAGE: "$CI_REGISTRY_IMAGE/builder@sha256:$BUILD_IMAGE_DIGEST"
  # Disable git checkout optimizations that break hermeticity
  GIT_STRATEGY: clone
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - build
  - test
  - deploy

build:
  stage: build
  image: docker:${DOCKER_VERSION}
  services:
    - docker:${DOCKER_VERSION}-dind
  before_script:
    # Verify we're building from a tagged commit
    - |
      if [ -z "$CI_COMMIT_TAG" ]; then
        echo "ERROR: Must build from a tagged commit"
        exit 1
      fi
  script:
    # Build with explicit cache directives
    - |
      docker build \
        --build-arg BUILDKIT_INLINE_CACHE=1 \
        --cache-from $BUILD_IMAGE \
        --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA \
        --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG \
        .
    # Generate reproducible SBOM
    - docker sbom $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA > sbom.json
    # Push with digest
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
  artifacts:
    paths:
      - sbom.json
    expire_in: 1 year

Why This Is Hermetic:

• ✅ Base image referenced by SHA256 digest
• ✅ System packages pinned to exact versions
• ✅ npm ci installs exact dependency versions
• ✅ Build runs in isolated container
• ✅ No external network calls during build
• ✅ Artifacts tagged by commit SHA
• ✅ SBOM (Software Bill of Materials) generated for auditability

Example 2: Go Service with Bazel

Bazel is designed for hermetic builds. Here's a Go service pipeline:

WORKSPACE (Bazel workspace definition):

workspace(name = "myapp")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

## Pin Go rules with SHA256
http_archive(
    name = "io_bazel_rules_go",
    sha256 = "ae013bf35bd23234d1dea46b079f1e05ba74ac0321423830119d3e787ec73483",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.40.0/rules_go-v0.40.0.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.40.0/rules_go-v0.40.0.zip",
    ],
)

## Pin gazelle with SHA256
http_archive(
    name = "bazel_gazelle",
    sha256 = "727f3e4edd96ea20c29e8c2ca9e8d2af724d8c7778e7923a854b2c80952bc405",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.30.0/bazel-gazelle-v0.30.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.30.0/bazel-gazelle-v0.30.0.tar.gz",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")

go_rules_dependencies()

## Pin exact Go version
go_register_toolchains(version = "1.20.4")

gazelle_dependencies()

BUILD.bazel (build definition):

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
load("@bazel_gazelle//:def.bzl", "gazelle")

gazelle(name = "gazelle")

go_library(
    name = "app_lib",
    srcs = ["main.go"],
    importpath = "github.com/myorg/myapp",
    visibility = ["//visibility:private"],
    deps = [
        "@com_github_gin_gonic_gin//:gin",
        "@com_github_sirupsen_logrus//:logrus",
    ],
)

go_binary(
    name = "app",
    embed = [
        ":app_lib",
    ],
    visibility = ["//visibility:public"],
    # Reproducible build flags
    gc_linkopts = [
        "-s",  # Strip symbol table
        "-w",  # Strip debug info
        "-X main.version={STABLE_VERSION}",
        "-X main.commit={STABLE_COMMIT}",
    ],
)

Jenkinsfile (hermetic pipeline):

pipeline {
    agent {
        docker {
            // Pin build image by digest
            image 'gcr.io/bazel-public/bazel:6.2.0@sha256:xyz789...'
            args '--network=none --read-only --tmpfs /tmp:exec'
        }
    }
    
    environment {
        // Hermetic build flags
        BAZEL_OPTS = '--sandbox_default_allow_network=false --experimental_repository_cache_hardlinks'
        // Reproducible timestamps
        SOURCE_DATE_EPOCH = sh(script: 'git log -1 --format=%ct', returnStdout: true).trim()
    }
    
    stages {
        stage('Build') {
            steps {
                sh '''
                    bazel build \
                        --stamp \
                        --workspace_status_command=./build/workspace_status.sh \
                        --define version=${GIT_TAG} \
                        //...
                '''
            }
        }
        
        stage('Test') {
            steps {
                sh 'bazel test --test_output=errors //...'
            }
        }
        
        stage('Package') {
            steps {
                sh '''
                    # Create OCI image with exact content hash
                    bazel run //:push_image -- \
                        --tag=${DOCKER_REGISTRY}/myapp:${GIT_COMMIT} \
                        --tag=${DOCKER_REGISTRY}/myapp:${GIT_TAG}
                '''
            }
        }
    }
}

Why Bazel Is Hermetic:

• ✅ All external dependencies defined with SHA256 checksums
• ✅ Sandboxed execution prevents undeclared dependencies
• ✅ Content-addressable caching
• ✅ Network isolation during build
• ✅ Reproducible across machines and platforms

Example 3: Python Application with Nix

Nix provides operating-system-level hermetic builds. Here's a Python data pipeline:

default.nix (Nix build definition):

{ pkgs ? import <nixpkgs> { } }:

let
  # Pin exact nixpkgs version
  pinnedPkgs = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/23.05.tar.gz";
    sha256 = "1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0t1u2v3w4x5y6z";
  }) {};
  
  python = pinnedPkgs.python311;
  
  pythonPackages = python.pkgs;
  
  # Define exact Python dependencies
  myPythonEnv = python.withPackages (ps: with ps; [
    pandas
    numpy
    requests
    pytest
  ]);
  
in
pinnedPkgs.stdenv.mkDerivation {
  name = "data-pipeline";
  version = "1.0.0";
  
  src = ./.;
  
  buildInputs = [ myPythonEnv ];
  
  buildPhase = ''
    # Compile Python to bytecode for determinism
    python -m compileall -b .
  '';
  
  installPhase = ''
    mkdir -p $out/bin
    mkdir -p $out/lib/python
    
    # Install bytecode, not source
    cp -r *.pyc $out/lib/python/
    
    # Create wrapper script
    cat > $out/bin/pipeline << EOF
    #!${pinnedPkgs.bash}/bin/bash
    export PYTHONPATH=$out/lib/python
    exec ${myPythonEnv}/bin/python $out/lib/python/main.pyc "\$@"
    EOF
    
    chmod +x $out/bin/pipeline
  '';
  
  # Run tests as part of build
  checkPhase = ''
    pytest tests/
  '';
  
  doCheck = true;
}

CircleCI config (Nix-based pipeline):

version: 2.1

jobs:
  build:
    docker:
      # Use Nix-enabled image
      - image: nixos/nix:2.15.0@sha256:abc123...
    
    steps:
      - checkout
      
      - restore_cache:
          keys:
            # Cache key includes nix configuration hash
            - nix-store-v1-{{ checksum "default.nix" }}-{{ checksum "shell.nix" }}
      
      - run:
          name: Build with Nix
          command: |
            # Build in pure mode (no external dependencies)
            nix-build --pure --show-trace
      
      - run:
          name: Generate closure
          command: |
            # Create self-contained closure
            nix-store --export $(nix-store -qR result) > pipeline.closure
      
      - save_cache:
          key: nix-store-v1-{{ checksum "default.nix" }}-{{ checksum "shell.nix" }}
          paths:
            - /nix/store
      
      - store_artifacts:
          path: pipeline.closure
      
      - persist_to_workspace:
          root: .
          paths:
            - result
            - pipeline.closure
  
  deploy:
    docker:
      - image: nixos/nix:2.15.0@sha256:abc123...
    
    steps:
      - attach_workspace:
          at: .
      
      - run:
          name: Deploy closure
          command: |
            # Import and activate on target system
            nix-store --import < pipeline.closure
            nix-env --install ./result

Why This Is Hermetic:

• ✅ Nix pins all dependencies including system libraries
• ✅ Pure builds prevent external state access
• ✅ Content-addressable store ensures reproducibility
• ✅ Closures are completely self-contained
• ✅ Can reproduce exact build on any Nix system

Example 4: Multi-Stage Hermetic Deployment

Here's a complete multi-environment deployment pipeline:

Deployment Flow:

┌─────────────────────────────────────────────────────┐
│          HERMETIC DEPLOYMENT PIPELINE               │
└─────────────────────────────────────────────────────┘

  🔐 Git Tag: v1.2.3
       │
       ↓
  ┌─────────────────┐
  │  BUILD STAGE    │  Image: app@sha256:abc...
  │  (immutable)    │  Artifacts signed with GPG
  └────────┬────────┘
           │
           ├────────────────┬────────────────┐
           ↓                ↓                ↓
    ┌──────────┐     ┌──────────┐     ┌──────────┐
    │   DEV    │     │ STAGING  │     │   PROD   │
    │          │     │          │     │          │
    │ Deploy   │────→│ Deploy   │────→│ Deploy   │
    │ sha:abc  │     │ sha:abc  │     │ sha:abc  │
    │          │     │          │     │          │
    │ ✓ Tests  │     │ ✓ Tests  │     │ Manual   │
    │ ✓ Auto   │     │ ✓ Smoke  │     │ Approval │
    └──────────┘     └──────────┘     └──────────┘
         │                │                │
         ↓                ↓                ↓
    Same artifact deployed to all environments
    (only configuration differs)

GitHub Actions workflow:

name: Hermetic Deploy

on:
  push:
    tags:
      - 'v*.*.*'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build:
    runs-on: ubuntu-22.04
    outputs:
      image-digest: ${{ steps.build.outputs.digest }}
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          # Fetch full history for reproducible builds
          fetch-depth: 0
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          version: v0.11.0  # Pin buildx version
      
      - name: Log in to registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=semver,pattern={{version}}
            type=sha,prefix=,format=long
      
      - name: Build and push
        id: build
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          # Reproducible build settings
          build-args: |
            SOURCE_DATE_EPOCH=${{ github.event.head_commit.timestamp }}
            VERSION=${{ github.ref_name }}
            COMMIT=${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          # Generate provenance
          provenance: true
          sbom: true
      
      - name: Sign image
        run: |
          # Sign with cosign for supply chain security
          cosign sign --key env://COSIGN_KEY \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
        env:
          COSIGN_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
  
  deploy-dev:
    needs: build
    runs-on: ubuntu-22.04
    environment: development
    
    steps:
      - name: Verify image signature
        run: |
          cosign verify --key env://COSIGN_PUBLIC_KEY \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.image-digest }}
        env:
          COSIGN_PUBLIC_KEY: ${{ secrets.COSIGN_PUBLIC_KEY }}
      
      - name: Deploy to dev
        run: |
          # Deploy exact digest, not tag
          kubectl set image deployment/myapp \
            app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.image-digest }} \
            --namespace=dev
  
  deploy-staging:
    needs: [build, deploy-dev]
    runs-on: ubuntu-22.04
    environment: staging
    
    steps:
      - name: Deploy to staging
        run: |
          kubectl set image deployment/myapp \
            app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.image-digest }} \
            --namespace=staging
      
      - name: Run smoke tests
        run: |
          # Tests against staging environment
          ./scripts/smoke-tests.sh https://staging.example.com
  
  deploy-prod:
    needs: [build, deploy-staging]
    runs-on: ubuntu-22.04
    environment: production
    
    steps:
      - name: Deploy to production
        run: |
          # Same digest deployed everywhere
          kubectl set image deployment/myapp \
            app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.image-digest }} \
            --namespace=production

Key Hermetic Principles:

• ✅ Build once, deploy everywhere (same artifact)
• ✅ Reference by digest, never by tag
• ✅ Cryptographically sign artifacts
• ✅ Verify signatures before deployment
• ✅ Generate SBOM and provenance
• ✅ Immutable artifacts stored permanently

Common Mistakes ⚠️

Mistake 1: Using Floating Version Tags

❌ Wrong:

FROM node:18
FROM python:latest
RUN apt-get install curl

✅ Right:

FROM node:18.16.0-alpine3.17@sha256:a1b2c3d4...
FROM python:3.11.3-slim-bullseye@sha256:e5f6g7h8...
RUN apt-get install curl=7.74.0-1.3

Why it matters: "latest" and major version tags change over time. Your build in January will be different from your build in June.

Mistake 2: Installing Dependencies Without Verification

❌ Wrong:

pip install requests numpy pandas
go get github.com/gin-gonic/gin

✅ Right:

pip install --require-hashes -r requirements.txt
go mod download && go mod verify

Why it matters: Without verification, a compromised package registry could inject malicious code into your builds.

Mistake 3: Relying on System Time or Locale

❌ Wrong:

import datetime
BUILD_TIME = datetime.datetime.now().isoformat()

✅ Right:

import os
## Use SOURCE_DATE_EPOCH for reproducible timestamps
BUILD_TIME = os.environ.get('SOURCE_DATE_EPOCH', '0')

Why it matters: Timestamps make builds non-reproducible. Two builds from the same source will differ.

Mistake 4: Using Shared Caches Without Keys

❌ Wrong:

cache:
  paths:
    - node_modules/
    - .cache/

✅ Right:

cache:
  key:
    files:
      - package-lock.json
      - Dockerfile
  paths:
    - node_modules/

Why it matters: Unkeyed caches can serve stale dependencies when inputs change.

Mistake 5: Allowing Network Access During Build

❌ Wrong:

RUN curl https://install.example.com/script.sh | sh
RUN npm install --registry=https://registry.npmjs.org

✅ Right:

## Copy vendored dependencies
COPY vendor/ ./vendor/
## Install from local copy
RUN npm ci --offline --cache ./vendor/npm-cache

Why it matters: Network resources can change or become unavailable, breaking reproducibility.

Mistake 6: Not Committing Lock Files

❌ Wrong:

package-lock.json
Cargo.lock
go.sum
poetry.lock

✅ Right:

## Keep lock files!
## package-lock.json
## Cargo.lock
## go.sum

Why it matters: Lock files are the source of truth for dependency versions. Without them, builds aren't reproducible.

Mistake 7: Using npm install Instead of npm ci

❌ Wrong:

npm install

✅ Right:

npm ci

Why it matters: npm install can update packages within semver ranges. npm ci installs exact versions from the lock file.

Mistake 8: Ignoring Build Tool Versions

❌ Wrong:

script:
  - make build
  - go build

✅ Right:

image: golang:1.20.4@sha256:abc123...
script:
  - go version  # Verify version
  - go build

Why it matters: Different compiler versions can produce different outputs, even from identical source code.

Key Takeaways 🎯

The Four Pillars of Hermetic CI/CD

1. Explicit Inputs 📝
   Every dependency must be declared and versioned. No implicit dependencies from the environment.

2. Isolated Execution 🔒
   Builds run in containers with no access to host system state or network resources.

3. Deterministic Outputs 🎯
   Same inputs always produce byte-identical outputs. No timestamps, random IDs, or environmental variation.

4. Verifiable Artifacts ✅
   All outputs are cryptographically signed and have complete provenance chains.

Benefits of Hermetic Pipelines

• Debugging: Can reproduce any build locally
• Security: Supply chain attacks are detectable
• Reliability: No "works on my machine" problems
• Speed: Aggressive caching with confidence
• Compliance: Full audit trail of what's deployed
• Rollback: Can rebuild old versions identically

When to Use Hermetic Builds

Always use for:

• Production deployments
• Security-sensitive applications
• Regulated industries (finance, healthcare)
• Open source projects
• Long-lived applications

May skip for:

• Prototype/demo projects
• Personal experiments
• Applications with < 1 year lifespan
• Environments where reproducibility isn't critical

Hermetic Build Maturity Model

    HERMETIC MATURITY LEVELS
    
    Level 5: 🏆 PLATINUM
    ├─ Bit-identical rebuilds across platforms
    ├─ Complete supply chain attestation
    ├─ Automated vulnerability scanning
    └─ Zero-trust artifact verification
    
    Level 4: ⭐ GOLD
    ├─ All dependencies pinned by hash
    ├─ Sandboxed build execution
    ├─ Content-addressable caching
    └─ Cryptographic signatures
    
    Level 3: 🥈 SILVER
    ├─ Lock files for all dependencies
    ├─ Containerized build environments
    ├─ Version-pinned base images
    └─ Reproducible timestamps
    
    Level 2: 🥉 BRONZE
    ├─ Pinned major versions
    ├─ Documented dependencies
    ├─ Consistent environments
    └─ Basic caching
    
    Level 1: 🌱 BASIC
    ├─ Manual dependency management
    ├─ Local builds
    ├─ No reproducibility guarantees
    └─ Ad-hoc caching

💡 Pro Tip: Start at Level 3 (Silver) for production applications. It provides the best balance of effort vs. benefit.

📚 Further Study

1. Reproducible Builds Project - https://reproducible-builds.org/
   Comprehensive documentation on achieving bit-identical rebuilds across various ecosystems.

2. SLSA Framework - https://slsa.dev/
   Supply-chain Levels for Software Artifacts—a framework for ensuring artifact integrity.

3. Bazel Documentation - https://bazel.build/
   In-depth guide to the build tool designed for hermetic, reproducible builds at scale.

---

Congratulations! 🎉 You now understand how to build CI/CD pipelines that are reproducible, secure, and reliable. Hermetic builds are the foundation of trustworthy software delivery—start implementing these principles in your projects today!