Derivations and Nix Store

Derivations and Nix Store

Understand how Nix builds software reproducibly with free flashcards and spaced repetition practice. This lesson covers derivations (Nix's build instructions), the Nix store (immutable package storage), and content-addressable paths—essential concepts for mastering hermetic builds and the Nix mental model.

Welcome to the Nix Mental Model 💻

If you've ever wondered how Nix achieves perfect reproducibility where builds work identically on any machine, the answer lies in two fundamental concepts: derivations and the Nix store. These aren't just technical details—they're the philosophical foundation that makes Nix different from traditional package managers.

Think of a derivation as a recipe card that specifies exactly how to build something, including every ingredient (dependency) and every step. The Nix store is like a giant warehouse where the results of these recipes are stored in individually labeled boxes that can never be modified. Once something goes in the store, it's frozen in time forever.

🎯 Why This Matters: Understanding derivations and the Nix store is critical for:

• Debugging build failures
• Creating your own packages
• Understanding why Nix guarantees reproducibility
• Working with Nix in production environments

---

Core Concept 1: The Nix Store 📦

What is the Nix Store?

The Nix store is a directory (typically /nix/store) where Nix keeps all built packages, libraries, and artifacts. Every item in the store has a unique path based on a cryptographic hash of its inputs.

Structure of a Store Path:

/nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-107.0
          └─────────┬──────────┘ └────────┬──────────┘
                  hash                   name

• Hash (32 characters): Derived from all build inputs (source code, dependencies, compiler flags, build script)
• Name: Human-readable identifier (package-version)

Key Properties of the Nix Store 🔒

1. Immutability: Once a path exists in /nix/store, it can never be modified. If you change anything about a build, you get a different hash and a different path.

2. Content-Addressable: The hash acts like a fingerprint. If two builds have identical inputs, they produce the same hash and can share the same store path.

3. Isolation: Each package version lives in its own directory. You can have python-3.9, python-3.10, and python-3.11 all installed simultaneously without conflicts.

4. Garbage Collection: Store paths are only kept if something references them. Unreferenced paths can be cleaned up safely.

NIX STORE MENTAL MODEL

/nix/store/
  │
  ├─ abc123...xyz-glibc-2.35/
  │   ├─ lib/
  │   └─ include/
  │
  ├─ def456...uvw-openssl-3.0/
  │   ├─ lib/
  │   │   ├─ libssl.so → (immutable)
  │   │   └─ libcrypto.so
  │   └─ bin/
  │
  └─ ghi789...rst-nodejs-18.12/
      ├─ bin/node
      └─ lib/

  Each path is ISOLATED
  Each path is IMMUTABLE ✓
  Hash changes = new path ✓

💡 Tip: You can explore your store with ls /nix/store | head to see actual store paths on your system.

Why Hashes Matter 🔐

The hash isn't random—it's computed from:

• Source code (or download URL + checksum)
• All dependencies (their store paths)
• Build script content
• Environment variables used during build
• Compiler and tool versions

Change any of these? You get a different hash, a different path, and a completely separate build.

Example Scenario:

You build myapp that depends on openssl-1.1. Later, you update to openssl-3.0. Your new build gets a new hash because the dependency changed:

/nix/store/old-hash-myapp-1.0/  (depends on openssl-1.1)
/nix/store/new-hash-myapp-1.0/  (depends on openssl-3.0)

Both versions can coexist! The old one doesn't disappear until garbage collected.

---

Core Concept 2: Derivations (.drv files) 📝

What is a Derivation?

A derivation is Nix's blueprint for building something. It's a specification that describes:

• What to build (source code or inputs)
• How to build it (build commands)
• What dependencies are needed
• What environment variables to set

Derivations are stored as .drv files in the Nix store before they're built.

Conceptual Structure:

Derivation Component	Purpose	Example
outputs	What gets produced	"out", "dev", "doc"
inputDrvs	Dependencies (other derivations)	gcc, make, glibc
inputSrcs	Source files	tarball, patches
platform	Target system	x86_64-linux
builder	Program that executes build	/nix/store/...-bash/bin/bash
args	Arguments to builder	["-e", "build-script.sh"]
env	Environment variables	CC=gcc, PREFIX=$out

Derivation Lifecycle 🔄

DERIVATION WORKFLOW

1. Write Nix Expression          2. Instantiate
   ┌─────────────┐                  ┌──────────────┐
   │ default.nix │  nix-instantiate  │  .drv file   │
   │             │  ─────────────→   │              │
   │ { stdenv,   │                   │ /nix/store/  │
   │   fetchurl, │                   │ abc...xyz-   │
   │   ... }     │                   │ myapp.drv    │
   └─────────────┘                   └──────┬───────┘
                                             │
                                             │ nix-store --realise
                                             ↓
                              3. Build (realize)
                                 ┌──────────────┐
                                 │ Built output │
                                 │              │
                                 │ /nix/store/  │
                                 │ xyz...abc-   │
                                 │ myapp-1.0/   │
                                 └──────────────┘

Step-by-step:

1. Nix Expression → You write high-level Nix code (like default.nix)
2. Instantiation → Nix evaluates your expression and creates a .drv file
3. Realization → Nix executes the .drv instructions to produce the actual build output

💡 Key Insight: The .drv file is itself stored in /nix/store and has its own hash. It's the "recipe", while the final output is the "meal".

---

Core Concept 3: How Hashing Works 🔢

Computing Store Path Hashes

Nix uses a cryptographic hash function (SHA-256, truncated to 160 bits, base-32 encoded) to compute store paths. The hash input includes:

hash_input = (
  "output:out",
  "source" or derivation_path,
  all_dependency_paths,
  build_system,
  derivation_name
)

store_path = "/nix/store/" + base32(sha256(hash_input))[0:32] + "-" + name

Fixed-Output Derivations vs Regular Derivations

Regular Derivation:

• Hash computed from inputs (source, dependencies, build script)
• Two developers building the same inputs get the same hash before building
• Used for most software builds

Fixed-Output Derivation:

• Hash computed from expected output (usually a checksum)
• Used for downloading files from the internet
• Allows network access during build (normally prohibited)

Example: Fetching a tarball

fetchurl {
  url = "https://example.com/foo-1.0.tar.gz";
  sha256 = "0abc...xyz";  # Expected output hash
}

Nix knows the output hash before downloading. If the downloaded file's hash doesn't match, the build fails. This is how Nix ensures reproducibility even for network resources.

HASH COMPUTATION COMPARISON

┌──────────────────────────────────────────────┐
│        REGULAR DERIVATION                    │
├──────────────────────────────────────────────┤
│  Input Hash = f(source, deps, script)        │
│                    ↓                         │
│             Build Process                    │
│                    ↓                         │
│         Output (hash unknown until built)    │
└──────────────────────────────────────────────┘

┌──────────────────────────────────────────────┐
│      FIXED-OUTPUT DERIVATION                 │
├──────────────────────────────────────────────┤
│  Output Hash = sha256 (declared upfront)     │
│                    ↓                         │
│       Download/Build (network allowed)       │
│                    ↓                         │
│  Verify: actual_hash == declared_hash ✓      │
└──────────────────────────────────────────────┘

---

Example 1: Simple Derivation 🛠️

Let's build a minimal "Hello World" program using a derivation.

File: hello.nix

{ pkgs ? import <nixpkgs> {} }:

pkgs.stdenv.mkDerivation {
  name = "hello-custom";
  
  src = pkgs.writeText "hello.c" ''
    #include <stdio.h>
    int main() {
      printf("Hello from Nix!\n");
      return 0;
    }
  '';
  
  buildInputs = [ pkgs.gcc ];
  
  unpackPhase = "true";  # No archive to unpack
  
  buildPhase = ''
    gcc ${pkgs.writeText "hello.c" ''#include <stdio.h>
    int main() { printf("Hello from Nix!\n"); return 0; }''} -o hello
  '';
  
  installPhase = ''
    mkdir -p $out/bin
    cp hello $out/bin/
  '';
}

Build it:

nix-build hello.nix
## Output: /nix/store/xyz123...-hello-custom

./result/bin/hello
## Output: Hello from Nix!

What happened:

1. Nix evaluated hello.nix and created a .drv file
2. The .drv specifies: gcc as dependency, build commands, install location
3. Nix executed the build in an isolated environment
4. Output was placed in /nix/store/xyz123...-hello-custom/bin/hello
5. A symlink result points to the store path

💡 Note: $out is a special variable that points to the store path where output should go.

---

Example 2: Inspecting a Derivation 🔍

Let's peek inside an actual .drv file.

## Instantiate without building
nix-instantiate hello.nix
## Output: /nix/store/abc456...-hello-custom.drv

## Read the derivation
nix show-derivation /nix/store/abc456...-hello-custom.drv

Output (simplified JSON):

{
  "/nix/store/abc456...-hello-custom.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/xyz123...-hello-custom"
      }
    },
    "inputDrvs": {
      "/nix/store/def789...-gcc-11.3.0.drv": ["out"],
      "/nix/store/ghi012...-bash-5.1.drv": ["out"]
    },
    "inputSrcs": [
      "/nix/store/jkl345...-hello.c"
    ],
    "platform": "x86_64-linux",
    "builder": "/nix/store/ghi012...-bash-5.1/bin/bash",
    "args": ["-e", "/nix/store/mno678...-builder.sh"],
    "env": {
      "buildInputs": "/nix/store/def789...-gcc-11.3.0",
      "name": "hello-custom",
      "out": "/nix/store/xyz123...-hello-custom",
      "system": "x86_64-linux"
    }
  }
}

Key observations:

• The output path is predetermined (based on the hash)
• All dependencies are referenced by their full store paths
• The builder is bash (also from the store)
• Environment variables are explicitly listed

🔧 Try this: Run nix show-derivation $(nix-instantiate '<nixpkgs>' -A hello) to see a real-world package derivation.

---

Example 3: Understanding Dependencies 🔗

Derivations form a dependency graph. Let's visualize this.

Scenario: Building a simple web app

{ pkgs ? import <nixpkgs> {} }:

let
  myNodeApp = pkgs.stdenv.mkDerivation {
    name = "my-node-app";
    src = ./src;
    buildInputs = [ pkgs.nodejs pkgs.yarn ];
    buildPhase = "yarn install && yarn build";
    installPhase = "cp -r dist $out/";
  };
in
  myNodeApp

Dependency graph:

DEPENDENCY GRAPH

     ┌─────────────────────┐
     │  my-node-app.drv    │
     └──────────┬──────────┘
                │
       ┌────────┴────────┐
       ↓                 ↓
┌─────────────┐   ┌─────────────┐
│ nodejs.drv  │   │  yarn.drv   │
└──────┬──────┘   └──────┬──────┘
       │                 │
   ┌───┴───┐         ┌───┴───┐
   ↓       ↓         ↓       ↓
┌─────┐ ┌─────┐  ┌─────┐ ┌─────┐
│ gcc │ │ icu │  │node │ │ ... │
└─────┘ └─────┘  └─────┘ └─────┘

All dependencies are EXPLICIT
All are referenced by store paths
→ Perfect reproducibility

What makes this hermetic:

1. No implicit dependencies: Can't accidentally use system libraries
2. Exact versions: Not "nodejs" but "/nix/store/xyz-nodejs-18.12.0"
3. Transitive closure: All dependencies of dependencies are tracked
4. Reproducible: Same source + same dependencies = same hash = same build

💡 Mnemonic: Think "Derivations Define Dependencies Deterministically" (4 D's)

---

Example 4: Multiple Outputs 📤

Derivations can produce multiple outputs (binaries, libraries, documentation, etc.).

{ pkgs ? import <nixpkgs> {} }:

pkgs.stdenv.mkDerivation {
  name = "multi-output-example";
  
  outputs = [ "out" "dev" "doc" ];
  
  src = ./src;
  
  buildPhase = ''
    # Compile library
    gcc -shared -fPIC mylib.c -o libmylib.so
    
    # Create headers
    cp mylib.h mylib-headers/
    
    # Generate docs
    echo "Documentation" > README
  '';
  
  installPhase = ''
    # Binaries go to $out
    mkdir -p $out/lib
    cp libmylib.so $out/lib/
    
    # Development headers go to $dev
    mkdir -p $dev/include
    cp mylib.h $dev/include/
    
    # Documentation goes to $doc
    mkdir -p $doc/share/doc
    cp README $doc/share/doc/
  '';
}

Result in store:

/nix/store/aaa111...-multi-output-example/       # $out (runtime)
/nix/store/bbb222...-multi-output-example-dev/   # $dev (headers)
/nix/store/ccc333...-multi-output-example-doc/   # $doc (docs)

Why multiple outputs?

• Space efficiency: Don't need docs/headers in production deployments
• Dependency precision: Dev tools depend on $dev, users depend on $out
• Closure size reduction: Smaller Docker images, faster downloads

🤔 Did you know? The glibc package in nixpkgs has 9 different outputs (out, bin, dev, static, debug, ...) to optimize for different use cases!

---

Common Mistakes ⚠️

Mistake 1: Expecting Store Paths to Be Readable 👀

Wrong assumption:

ls /nix/store/firefox
## Error: No such file or directory

Why it fails: The directory name includes a hash. You can't predict the full path.

Correct approach:

## Let Nix tell you the path
nix-build '<nixpkgs>' -A firefox

## Or use nix-env to query
nix-env -qa --installed firefox

---

Mistake 2: Trying to Modify Store Paths 🚫

Wrong:

chmod +w /nix/store/xyz-myapp/bin/app
echo "hacked" >> /nix/store/xyz-myapp/bin/app
## Permission denied (store is read-only)

Why it fails: The Nix store is mounted read-only for non-root users (or protected by permissions).

Correct approach: If you need to modify something, create a new derivation that wraps or patches the original:

pkgs.runCommand "myapp-modified" {} ''
  mkdir -p $out/bin
  cp ${originalApp}/bin/app $out/bin/app
  echo "# My modifications" >> $out/bin/app
''

---

Mistake 3: Forgetting Impure Dependencies 🌐

Wrong:

stdenv.mkDerivation {
  name = "bad-example";
  buildPhase = ''
    # This will fail! No network access during build
    curl https://example.com/data.json > data.json
    gcc main.c -o app
  '';
}

Why it fails: Regular derivations run in a sandbox with no network access.

Correct approach: Use fetchurl or fetchFromGitHub (fixed-output derivations) to get external resources before the build:

let
  dataFile = pkgs.fetchurl {
    url = "https://example.com/data.json";
    sha256 = "0abc123...";  # Declare expected hash
  };
in
stdenv.mkDerivation {
  name = "good-example";
  buildPhase = ''
    cp ${dataFile} data.json
    gcc main.c -o app
  '';
}

---

Mistake 4: Not Understanding Hash Changes 🔄

Confusion: "I only changed a comment in my source code, why did the entire dependency tree rebuild?"

Explanation: Any change to inputs changes the hash, which changes the store path, which means all dependent packages see a different path and need rebuilding.

CHANGE PROPAGATION

Before:                After (comment added):
┌─────────────┐        ┌─────────────┐
│   app-v1    │        │   app-v2    │
│  /nix/...   │        │  /nix/...   │ ← New hash!
│   abc123    │        │   xyz789    │
└──────┬──────┘        └──────┬──────┘
       │                      │
       ↓                      ↓
┌─────────────┐        ┌─────────────┐
│   lib-v1    │        │   lib-v1    │ ← Same, but...
│  /nix/...   │        │  /nix/...   │
│   def456    │        │   def456    │
└─────────────┘        └─────────────┘

  Dependents of app-v1        Dependents of app-v2
  reference abc123            must reference xyz789
       ↓                            ↓
  EVERYTHING rebuilds that depends on changed path

Mitigation: Use Nix's binary cache (like cache.nixos.org) so you download pre-built binaries instead of rebuilding.

---

Key Takeaways 🎯

---

Mental Models 🧠

The Restaurant Analogy:

• Nix Expression = Menu item description ("Caesar Salad with dressing")
• Derivation (.drv) = Recipe card (exact ingredients, steps, tools)
• Nix Store = Walk-in freezer with labeled containers
• Store Path = Container label (includes date, source, chef ID)
• Building = Following the recipe to make the dish
• Result = Finished meal in a sealed container with hash label

The Version Control Analogy:

• Store Path Hash = Git commit SHA
• Derivation = Commit metadata (author, date, message, parent commits)
• Nix Store = Git object database (.git/objects)
• Building = Checking out a commit
• Dependencies = Parent commits in Git history

---

Further Study 📚

1. Nix Pills - Chapter 18 (Nix Store Paths): Deep dive into how Nix computes store path hashes
   🔗 https://nixos.org/guides/nix-pills/nix-store-paths.html

2. Nix Manual - Store Path Specification: Official technical specification for store path computation
   🔗 https://nixos.org/manual/nix/stable/store/store-path.html

3. Eelco Dolstra's PhD Thesis: The foundational research behind Nix (Chapter 4 covers the Nix store model)
   🔗 https://edolstra.github.io/pubs/phd-thesis.pdf

---

You now understand the foundation of Nix's hermetic build system! 🎉 The store and derivations are the "secret sauce" that makes reproducible builds possible. Everything else in Nix builds on these concepts. Next, explore how to write your own derivations and leverage the massive nixpkgs ecosystem.