Reproducibility vs Determinism

Reproducibility vs Determinism in Hermetic Builds

Master hermetic builds with free flashcards and structured practice to understand reproducibility, determinism, and their critical role in modern software engineering. This lesson covers the fundamental differences between reproducible and deterministic builds, common implementation challenges, and practical strategies for achieving both properties in your build systems.

Welcome to Build Reliability 💻

When you compile your code today and get app.exe, then compile the same code tomorrow and get a different app.exe, something has gone wrong. Modern software development demands that builds be both reproducible and deterministic—but these terms, while related, mean different things. Understanding this distinction is crucial for creating reliable, secure, and verifiable software systems.

Hermetic builds aim to isolate the build process from external influences, but achieving true hermeticity requires understanding what we're actually trying to guarantee. Are we trying to get the same output file? The same behavior? Both? Let's untangle these concepts.

Core Concept: What is Reproducibility? 🔄

Reproducibility means that given the same source code and build configuration, you can generate functionally equivalent outputs across different times, machines, and environments. The key word here is functionally equivalent—the outputs behave the same way, even if they're not byte-for-byte identical.

Think of it like baking cookies from a recipe:

• Reproducible baking: Following the recipe produces cookies that taste the same, look similar, and have the same texture—even if you bake them in different ovens on different days
• Not reproducible: Sometimes you get chocolate chip cookies, sometimes you get oatmeal raisin, even though you followed the same recipe

Characteristics of Reproducible Builds:

💡 Key Insight: Reproducibility focuses on outcomes and behavior, not perfect binary matching. Two builds might differ in metadata but still be reproducible.

What Makes Builds Non-Reproducible?

Common culprits that break reproducibility:

1. Unstable Dependencies 🎲

   • Using latest tags instead of pinned versions
   • Pulling from registries without checksums
   • Network failures causing partial downloads

2. Environmental Leakage 🌍

   • Reading from /etc/hostname or environment variables
   • Different tool versions on different machines
   • Varying system libraries

3. Temporal Dependencies ⏰

   • Build timestamps embedded in output
   • Generated IDs based on current time
   • Date-dependent compilation flags

4. Non-Hermetic Resources 📡

   • Fetching data from internet during build
   • Reading from mutable file paths
   • Depending on global system state

┌─────────────────────────────────────────────────┐
│         REPRODUCIBILITY REQUIREMENTS            │
├─────────────────────────────────────────────────┤
│                                                 │
│  📋 Same Source Code                           │
│           ↓                                     │
│  🔒 Pinned Dependencies                        │
│           ↓                                     │
│  🐳 Controlled Environment                     │
│           ↓                                     │
│  ⚙️  Isolated Build Process                    │
│           ↓                                     │
│  ✅ Functionally Equivalent Output             │
│                                                 │
└─────────────────────────────────────────────────┘

Core Concept: What is Determinism? 🎯

Determinism is a stronger property than reproducibility. A deterministic build produces byte-for-byte identical outputs given the same inputs. Every single bit matches, including metadata, timestamps, and file ordering.

Back to our baking analogy:

• Deterministic baking: Not only do the cookies taste identical, but every sugar crystal is in exactly the same position, every air bubble is the same size, and the browning pattern is pixel-perfect identical
• This level of precision is nearly impossible with real cookies but achievable with builds!

Characteristics of Deterministic Builds:

Why Determinism Matters for Security 🔒

Scenario: You download a binary claiming to be built from open-source code. How do you know it actually comes from that source and hasn't been tampered with?

With deterministic builds:

1. You build the source code yourself
2. Compare your build's hash to the published hash
3. If they match: SHA256(your_build) == SHA256(official_build) ✅
4. You have cryptographic proof the binary is legitimate

Without determinism: You can't prove anything. Even legitimate builds from the same source produce different hashes.

DETERMINISTIC BUILD VERIFICATION

   Developer A          Developer B          Developer C
       │                    │                    │
       ↓                    ↓                    ↓
   [Build]              [Build]              [Build]
       │                    │                    │
       ↓                    ↓                    ↓
   SHA256:              SHA256:              SHA256:
   a3f8b2...            a3f8b2...            a3f8b2...
       │                    │                    │
       └────────────────────┴────────────────────┘
                           │
                    ALL HASHES MATCH! ✅
                    Build is verified

What Breaks Determinism?

Even subtle differences create non-deterministic builds:

1. Embedded Timestamps 📅

   # Non-deterministic
   build_time = datetime.now().isoformat()
   header = f"Built on {build_time}"
   

2. Random Values 🎲

   # Non-deterministic
   build_id = uuid.uuid4()
   

3. Hash Map Iteration 🗺️

   # Non-deterministic (dict order varies in Python <3.7)
   for key in unsorted_dict:
       output.write(key)
   

4. File System Ordering 📁

   # Non-deterministic (readdir() order is undefined)
   for file in *.c; do
       compile $file
   done
   

5. Parallel Build Race Conditions ⚡

   • Multiple threads writing to shared output
   • Non-deterministic task scheduling

6. Floating Point Operations 🔢

   • Different CPU architectures may compute slightly different results
   • Optimization flags can change operation order

💡 Pro Tip: Tools like reprotest can help identify sources of non-determinism by building under different conditions and comparing outputs.

The Relationship: Determinism ⊂ Reproducibility 📊

Here's the crucial relationship:

Think of it as nested sets:

┌─────────────────────────────────────────┐
│     ALL BUILDS                          │
│                                         │
│  ┌─────────────────────────────────┐   │
│  │  REPRODUCIBLE BUILDS            │   │
│  │                                 │   │
│  │  ┌─────────────────────────┐   │   │
│  │  │ DETERMINISTIC BUILDS    │   │   │
│  │  │                         │   │   │
│  │  │  • Byte-identical       │   │   │
│  │  │  • Cryptographically    │   │   │
│  │  │    verifiable           │   │   │
│  │  │                         │   │   │
│  │  └─────────────────────────┘   │   │
│  │                                 │   │
│  │  • Functionally equivalent      │   │
│  │  • May differ in metadata       │   │
│  │                                 │   │
│  └─────────────────────────────────┘   │
│                                         │
│  • May vary across environments         │
│  • Unpredictable outputs                │
│                                         │
└─────────────────────────────────────────┘

Comparison Table

Example 1: Reproducible but Not Deterministic 🏗️

Let's see a concrete example of a build that's reproducible but not deterministic:

## build.py
import datetime
import subprocess

def build_app(source_files, output_binary):
    # Embed build timestamp (breaks determinism)
    timestamp = datetime.datetime.now().isoformat()
    
    # Create version string
    version_code = f'''
    const char* BUILD_TIME = "{timestamp}";
    const char* VERSION = "1.0.0";
    '''
    
    # Write version header
    with open('version.h', 'w') as f:
        f.write(version_code)
    
    # Compile with pinned compiler version
    subprocess.run([
        'gcc-11.2.0',  # Specific version
        '-o', output_binary,
        *source_files,
        '-O2'
    ])

Why it's reproducible:

• Uses specific compiler version (gcc-11.2.0)
• Deterministic optimization level (-O2)
• Same source files produce same functionality
• Application behaves identically when run

Why it's NOT deterministic:

• BUILD_TIME differs on each build
• Binary contains different timestamp strings
• SHA-256(build1) ≠ SHA-256(build2)
• Cannot cryptographically verify

Making it deterministic:

import os

def build_app(source_files, output_binary):
    # Use SOURCE_DATE_EPOCH for determinism
    timestamp = os.environ.get('SOURCE_DATE_EPOCH', '1577836800')
    
    version_code = f'''
    const char* BUILD_TIME = "{timestamp}";
    const char* VERSION = "1.0.0";
    '''
    
    with open('version.h', 'w') as f:
        f.write(version_code)
    
    subprocess.run([
        'gcc-11.2.0',
        '-o', output_binary,
        *sorted(source_files),  # Ensure consistent ordering
        '-O2',
        '-Wl,--build-id=none'  # Remove build ID
    ])

💡 SOURCE_DATE_EPOCH: A standard environment variable containing a Unix timestamp. Set it to your last git commit time for deterministic timestamps.

Example 2: Docker Image Builds 🐳

Docker images are a common place where the reproducibility vs. determinism distinction matters:

Non-reproducible Dockerfile:

FROM ubuntu:latest

RUN apt-get update && apt-get install -y \
    python3 \
    python3-pip

RUN pip3 install flask

COPY app.py /app/
CMD ["python3", "/app/app.py"]

Problems:

• ubuntu:latest changes over time (tag moves)
• apt-get update fetches current package lists
• pip3 install flask gets newest compatible version
• Not reproducible OR deterministic

Reproducible Dockerfile:

FROM ubuntu:22.04@sha256:abc123...

RUN apt-get update && apt-get install -y \
    python3=3.10.6-1~22.04 \
    python3-pip=22.0.2+dfsg-1

COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt

## requirements.txt contains:
## flask==2.3.2
## werkzeug==2.3.6
## jinja2==3.1.2

COPY app.py /app/
CMD ["python3", "/app/app.py"]

Improvements:

• Base image pinned with SHA-256 hash
• Specific package versions
• Pinned Python dependencies
• Now reproducible across builds

Still not fully deterministic because:

• Layer timestamps differ
• File metadata (modification times) varies
• Image creation timestamp embedded

Making it more deterministic:

FROM ubuntu:22.04@sha256:abc123...

## Use fixed timestamp for all operations
ENV SOURCE_DATE_EPOCH=1577836800

RUN apt-get update && apt-get install -y \
    python3=3.10.6-1~22.04 \
    python3-pip=22.0.2+dfsg-1 \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt /tmp/
RUN pip3 install --no-cache-dir -r /tmp/requirements.txt

COPY app.py /app/

## Strip timestamps from files
RUN find /app -exec touch -d "@${SOURCE_DATE_EPOCH}" {} +

CMD ["python3", "/app/app.py"]

🎯 Best Practice: Use tools like buildkit with --build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) for deterministic Docker builds.

Example 3: JavaScript/Node.js Builds 📦

Non-reproducible package.json:

{
  "dependencies": {
    "express": "^4.18.0",
    "lodash": "~4.17.0",
    "axios": "*"
  }
}

Problems:

• ^ allows minor version updates (4.18.0 → 4.19.0)
• ~ allows patch updates (4.17.0 → 4.17.21)
• * accepts any version
• Different developers get different versions

Making it reproducible:

## Generate lockfile
npm install  # Creates package-lock.json

## Commit package-lock.json to git
git add package-lock.json
git commit -m "Add lockfile for reproducibility"

## Other developers use exact versions
npm ci  # Uses lockfile, doesn't update it

package-lock.json ensures:

• Exact versions recorded: "express": "4.18.2"
• Entire dependency tree locked
• npm ci installs identical dependencies
• Now reproducible!

For determinism, also need:

## Set cache location
export npm_config_cache=/tmp/npm-cache

## Use consistent NODE_ENV
export NODE_ENV=production

## Build with webpack
webpack --mode production --config webpack.config.js

Webpack config for determinism:

module.exports = {
  mode: 'production',
  output: {
    filename: '[name].[contenthash].js',
    hashFunction: 'sha256',
    hashDigestLength: 20
  },
  optimization: {
    moduleIds: 'deterministic',  // Stable module IDs
    chunkIds: 'deterministic'     // Stable chunk IDs
  },
  plugins: [
    new webpack.DefinePlugin({
      'process.env.BUILD_TIME': JSON.stringify(
        process.env.SOURCE_DATE_EPOCH || '1577836800'
      )
    })
  ]
}

Example 4: Java/Maven Builds ☕

Java builds have their own reproducibility challenges:

Reproducible Maven configuration:

<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>myapp</artifactId>
  <version>1.0.0</version>
  
  <properties>
    <maven.compiler.source>11</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    
    <!-- For deterministic builds -->
    <project.build.outputTimestamp>2020-01-01T00:00:00Z</project.build.outputTimestamp>
  </properties>
  
  <dependencies>
    <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
      <version>31.1-jre</version>  <!-- Exact version, no ranges -->
    </dependency>
  </dependencies>
  
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.2.2</version>
        <configuration>
          <archive>
            <manifestEntries>
              <!-- Fixed build metadata -->
              <Build-Time>${project.build.outputTimestamp}</Build-Time>
            </manifestEntries>
          </archive>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Key elements:

• project.build.outputTimestamp: Makes JAR timestamps deterministic
• Exact dependency versions (no [1.0,2.0) ranges)
• Fixed source encoding
• Specific plugin versions

Verification:

## Build twice
mvn clean package
cp target/myapp-1.0.0.jar build1.jar

mvn clean package
cp target/myapp-1.0.0.jar build2.jar

## Compare
sha256sum build1.jar build2.jar
## Should produce identical hashes!

Common Mistakes and How to Avoid Them ⚠️

Mistake 1: Confusing the Two Concepts

❌ Wrong thinking: "My builds are reproducible, so they must be deterministic."

✅ Correct thinking: "My builds are reproducible (same behavior), but I need to verify if they're deterministic (same bytes) by comparing hashes."

How to check:

## Build twice
./build.sh
mv output.bin output1.bin

./build.sh  
mv output.bin output2.bin

## Test reproducibility (functional)
./test_suite output1.bin  # All tests pass
./test_suite output2.bin  # All tests pass

## Test determinism (binary)
sha256sum output1.bin output2.bin
## If hashes match → deterministic
## If hashes differ → reproducible but not deterministic

Mistake 2: Using "Latest" Tags

❌ Anti-pattern:

FROM node:latest
RUN npm install

✅ Best practice:

FROM node:18.16.0-alpine3.17@sha256:f77...
RUN npm ci --only=production

Mistake 3: Ignoring Parallel Build Ordering

❌ Non-deterministic Makefile:

app: *.o
	ld -o app $^

%.o: %.c
	gcc -c $<

Problem: $^ (all prerequisites) has undefined order when built in parallel.

✅ Deterministic version:

SOURCES := $(sort $(wildcard *.c))
OBJECTS := $(SOURCES:.c=.o)

app: $(OBJECTS)
	ld -o app $(sort $^)

%.o: %.c
	gcc -c $<

Mistake 4: Embedding Build Paths

❌ Problem code:

#define BUILD_PATH __FILE__
// Embeds absolute path: /home/alice/project/src/main.c

Building in /home/alice/project vs /home/bob/project produces different binaries.

✅ Solution:

## Strip path prefixes
gcc -ffile-prefix-map=/home/alice/project=. -c main.c

Mistake 5: Timezone Dependencies

❌ Non-deterministic:

import datetime
build_date = datetime.datetime.now().strftime("%Y-%m-%d")
## Different results in different timezones!

✅ Deterministic:

import datetime
import os

timestamp = int(os.environ.get('SOURCE_DATE_EPOCH', '1577836800'))
build_date = datetime.datetime.utcfromtimestamp(timestamp).strftime("%Y-%m-%d")

Mistake 6: Ignoring Locale Settings

❌ Non-deterministic sort:

find . -name '*.txt' | sort
## Different sort order with different LC_COLLATE!

✅ Deterministic sort:

LC_ALL=C find . -name '*.txt' | sort

When to Choose Each Approach 🤔

Choose Reproducibility (not full determinism) when:

• You need reliable deployments across environments
• Development velocity is priority over perfect verification
• Build metadata (timestamps, debug info) provides value
• Tools don't easily support determinism
• Team is learning hermetic build practices

Example use case: Internal microservice deployed to staging/production. You want consistent behavior but don't need cryptographic verification.

Choose Determinism when:

• Security and supply chain verification are critical
• Building public software that others will verify
• Regulatory compliance requires auditable builds
• Creating packages for Linux distributions
• Contributing to projects like Debian, Arch, or Bitcoin Core

Example use case: Open-source security tool where users need to verify binaries match published source code.

Do users need to verify your builds?
    │
    ├─ NO → Reproducibility is sufficient
    │       ✓ Faster to implement
    │       ✓ Some metadata OK
    │       ✓ Focus on functional consistency
    │
    └─ YES → Need Determinism
            ✓ Byte-identical outputs
            ✓ Cryptographic verification
            ✓ Supply chain security
            ✓ More complex to achieve

Key Takeaways 🎓

1. Reproducibility ≠ Determinism: Reproducibility means functionally equivalent outputs; determinism means byte-identical outputs

2. Hierarchy: All deterministic builds are reproducible, but not all reproducible builds are deterministic

3. Verification Methods:

   • Reproducible builds: Test with functional test suites
   • Deterministic builds: Compare cryptographic hashes (SHA-256)

4. Common Enemies:

   • Timestamps → Use SOURCE_DATE_EPOCH
   • Unstable dependencies → Pin versions with lockfiles
   • File ordering → Sort explicitly
   • Embedded paths → Strip or normalize
   • Random values → Use seeded deterministic generation

5. Trade-offs: Determinism is harder to achieve but provides stronger security guarantees

6. Tooling Support: Modern build systems (Bazel, Nix, Buck2) prioritize both properties

7. Incremental Approach: Start with reproducibility, then progressively eliminate sources of non-determinism

8. Testing Strategy: Always build twice and compare—catches problems early

📋 Quick Reference Card

## Set deterministic timestamp from git
export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)

## Build twice and compare
sha256sum build1 build2

## Check for embedded timestamps
strings binary | grep -E '[0-9]{4}-[0-9]{2}-[0-9]{2}'

## Find sources of non-determinism
reprotest --vary=time,path,user 'make build'

📚 Further Study

1. Reproducible Builds Project: https://reproducible-builds.org/ - Comprehensive guide with tools and best practices
2. SOURCE_DATE_EPOCH Specification: https://reproducible-builds.org/docs/source-date-epoch/ - Standard for deterministic timestamps
3. Bazel's Approach to Hermetic Builds: https://bazel.build/basics/hermeticity - Modern build system design patterns

💡 Remember: Start by making your builds reproducible, then progressively work toward determinism. Every step toward hermeticity makes your software more reliable and secure!