Smart Chunking

The release build that came back with opt-level 0

A developer asks your docs assistant: "What optimization level do release builds use by default?" The answer sits in plain sight in the Cargo reference, as two lines under a heading called release:

[profile.release]
opt-level = 3

The assistant retrieves five chunks, and the only opt-level line under a release header among them says 0. Before you blame the embedding model or the vector search, look at what happened at indexing time, when a splitter decided where one chunk ends and the next begins.

This lesson works on a real corpus: four pages of the Cargo reference (Profiles, Specifying Dependencies, Features, Workspaces), pinned to one commit so the text never changes under you. It has what real documentation has: nested headings, TOML blocks, lists, a table, and code comments that start with #. The embedding model is the course's small baseline, all-MiniLM-L6-v2, which reads at most 256 word pieces. The first index cuts every page into windows of 128 of those word pieces (tokens), a round number well under the limit.

import re, urllib.request
import numpy as np
from sentence_transformers import SentenceTransformer

COMMIT = "07b80494920f3824e2515e536b76ff282e23993e"     # pinned: the same text on every run
BASE = f"https://raw.githubusercontent.com/rust-lang/cargo/{COMMIT}/doc/book/src/reference/"
DOCS = {}
for page in ["profiles", "specifying-dependencies", "features", "workspaces"]:
    md = urllib.request.urlopen(BASE + page + ".md").read().decode()
    DOCS[page] = re.sub(r"^\[[^\]\n]+\]:[^\n]*\n?", "", md, flags=re.M)   # drop link definitions

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
tok = model.tokenizer
LIMIT = model.max_seq_length                     # 256 word pieces, [CLS] and [SEP] included

def n_tokens(text):
    return len(tok(text, verbose=False)["input_ids"])

print(LIMIT, {page: n_tokens(text) for page, text in DOCS.items()})
# 256 {'profiles': 4342, 'specifying-dependencies': 7102, 'features': 4880, 'workspaces': 2954}

The regular expression removes Markdown link definitions such as [config]: config.md, which are addresses, not content. Cleaning belongs to Document Processing. Here is the naive index and the question:

def fixed_windows(size, overlap=0, tokenizer=tok):
    """Windows of `size` tokens, cut out of the original text by character offsets."""
    assert 0 <= overlap < size
    chunks = []
    for doc, text in DOCS.items():
        spans = tokenizer(text, add_special_tokens=False, return_offsets_mapping=True,
                          verbose=False)["offset_mapping"]
        start = 0
        while True:
            end = min(start + size, len(spans))
            a, b = spans[start][0], spans[end - 1][1]
            chunks.append({"doc": doc, "start": a, "end": b, "text": text[a:b]})
            if end == len(spans):
                break                                   # this window reached the end: stop
            start += size - overlap
    return chunks

def index(chunks):
    return model.encode([c["text"] for c in chunks], normalize_embeddings=True)

def search(query, chunks, vectors, k=5):
    scores = vectors @ model.encode(query, normalize_embeddings=True)
    return [(chunks[i], float(scores[i])) for i in np.argsort(-scores)[:k]]

windows = fixed_windows(128)
W = index(windows)
question = "What optimization level do release builds use by default?"
for r, (c, score) in enumerate(search(question, windows, W), 1):
    print(r, f"{score:.3f}", repr(c["text"][:60]))

Predict first: the index holds 153 windows. Does any of the top five contain opt-level = 3?

1 0.534 'settings\n\nThe following is a list of settings that can be co'
2 0.522 'split into. More\ncode generation units allows more of a crat'
3 0.490 'default settings for build overrides are:\n\n```toml\n[profile.'
4 0.476 'profile is used for creating binaries to run with a debugger'
5 0.454 'Profile selection](#profile-selection).\n\n## Custom profiles\n'

### release

The `release` profile is intended for optimized artifacts used for releases
and in production. This profile is used when the `--release` flag is used, and
is the default for [`cargo install`].

The default settings for

ranked = search(question, windows, W, k=len(windows))
for r, (c, score) in enumerate(ranked, 1):
    if "[profile.release]\nopt-level = 3" in c["text"]:
        print(r, f"{score:.3f}", repr(c["text"][:60]))
# 10 0.405 'the `release` profile are:\n\n```toml\n[profile.release]\nopt-le'

Tenth place. The window with the heading has the question's words ("release", "optimized", "default") but not the value. The window with the value starts with "the release profile are:" and then looks like every other TOML block on the page. One cut separated the words people search with from the fact they want.

That is the whole lesson in one trace. A chunk is the unit your system can find and the unit it can quote. Where you cut decides what each vector can mean. This lesson teaches seven moves, each triggered by a symptom you can see in results or metrics:

This lesson assumes you know what an embedding is (Vector Embeddings) and the basic pipeline of Classic RAG Pipeline, which already chunks a small help center by section. Here you learn to choose between strategies and to prove the choice.

Chunking happens at index time

Chunking runs once per document version, during ingestion. At query time the index is searched the same way whether its chunks came from a fixed-size splitter or an LLM. A claim such as "semantic chunking adds 200 ms per query" confuses indexing with search. Put the costs where they actually land:

Search time depends on how many vectors you have and how the index is built (ANN Algorithms), not on how the boundaries were chosen.

A ruler before any move

You will compare seven ideas. Without a ruler, every one of them sounds right. The ruler has to measure the thing chunking changes, and the obvious one does not.

Document-level labels can't see chunking. Suppose you label each question with its page (profiles.md) and count a hit when any chunk of that page comes back. Every strategy keeps every page, so they all look alike. On this corpus that ruler gives 1.00 at k = 5 to fixed windows, to structure-aware chunks and to semantic chunks. The failure in the hook, where the right page came back without the right sentence, is invisible to it.

Label the answer text instead. For each question, store the exact text that answers it (the gold span) as a substring of the page. A question counts as found when every character of that span lies inside the chunks you would send to the model. Two retrieved chunks that together cover the span count, because the model sees both. Spaces are ignored, since splitters often drop the whitespace between chunks. Because each question here has exactly one gold span, recall@k (relevant items in the top k divided by all relevant items, averaged over questions) equals the share of questions found. The metric definitions live in Retrieval Metrics.

The labeled set has 53 questions across the four pages, written the way a developer would ask them, not copied from the text:

QUESTIONS = [  # (page, question, the exact text that answers it)
    ("profiles", "What optimization level do release builds use by default?",
     "[profile.release]\nopt-level = 3"),
    ("profiles", "How many codegen units does a crate get by default?",
     "The default is 256 for [incremental](#incremental) builds, and 16 for\nnon-incremental builds."),
    ("profiles", "How do I define a profile that is release plus LTO?",
     "[profile.release-lto]\ninherits = \"release\"\nlto = true"),
    ("profiles", "Which settings can a per-package profile override not change?",
     "Overrides cannot specify the `panic`, `lto`, or `rpath` settings."),
    ("profiles", "Does panic = abort apply when running tests?",
     "Tests, benchmarks, build scripts, and proc macros ignore the `panic` setting."),
    ("profiles", "What does strip = true do?",
     "`strip = true` is equivalent to `strip = \"symbols\"`."),
    ("profiles", "Are build scripts and proc macros compiled with optimizations by default?",
     "To compile quickly, all profiles, by default, do not optimize build\ndependencies (build scripts, proc macros, and their dependencies)"),
    ("profiles", "Is any LTO performed when opt-level is 0 and lto is false?",
     "No LTO is performed\n  if codegen units is 1 or [opt-level](#opt-level) is 0."),
    ("profiles", "Which profile does cargo test use?",
     "The `test` profile is the default profile used by [`cargo test`]."),
    ("profiles", "When a named package override and the \"*\" override both set opt-level, which one wins?",
     "The precedence for which value is used is done in the following order (first\nmatch wins):\n\n1. `[profile.dev.package.name]` --- A named package."),
    ("profiles", "Which debug info level does the dev profile use by default?",
     "* `2`, `true`, or `\"full\"`: full debug info, default for [`dev`](#dev)"),
    ("profiles", "How is thin LTO different from fat LTO?",
     "* `\"thin\"`: Performs [\"thin\" LTO]. This is similar to \"fat\", but takes\n  substantially less time to run while still achieving performance gains\n  similar to \"fat\"."),
    ("profiles", "Can I turn incremental compilation off with an environment variable?",
     "The incremental value can be overridden globally with the `CARGO_INCREMENTAL`\n[environment variable]"),
    ("profiles", "Which opt-level should I use for dependencies in development if I still want generics shared between crates?",
     "consider trying opt-level 1, which will apply some optimizations while still\nallowing monomorphized items to be shared."),
    ("profiles", "What is the --release flag short for?",
     "The `--release` flag is equivalent to `--profile=release`."),
    ("specifying-dependencies", "Which versions does the requirement ~1.2 accept?",
     "~1.2    := >=1.2.0, <1.3.0"),
    ("specifying-dependencies", "What version range does a plain \"0.2.3\" requirement allow?",
     "0.2.3  :=  >=0.2.3, <0.3.0"),
    ("specifying-dependencies", "Will foo = \"1.0\" match a published 1.0.0-alpha?",
     "then a requirement of `foo = \"1.0\"` will *not* match, and\nwill return an error."),
    ("specifying-dependencies", "How do I depend on a git repository at a specific tag?",
     "# a commit with a particular tag\nregex = { git = \"https://github.com/rust-lang/regex.git\", tag = \"1.10.3\" }"),
    ("specifying-dependencies", "Can I publish to crates.io if a dependency is given only by path?",
     "Crates that use dependencies specified with only a path are not\npermitted on [crates.io]."),
    ("specifying-dependencies", "What happens if I append #commit to a git dependency URL?",
     "# specifying the commit after # ignores the commit ID and generates a warning"),
    ("specifying-dependencies", "Can my build.rs use crates listed under [dependencies]?",
     "The build script **does not** have access to the dependencies listed\nin the `dependencies` or `dev-dependencies` section."),
    ("specifying-dependencies", "Do my dev-dependencies get passed on to crates that depend on mine?",
     "These dependencies are *not* propagated to other packages which depend on this\npackage."),
    ("specifying-dependencies", "Can I add a dependency only when debug_assertions is on?",
     "There is currently no way to add dependencies based on these configuration values."),
    ("specifying-dependencies", "Does Cargo clone the submodules of a git dependency?",
     "Cargo automatically fetches its submodules recursively"),
    ("specifying-dependencies", "Which keys may a dependency with workspace = true still set?",
     "Other than `optional`, `features`, and `default-features`, inherited dependencies\ncannot use any other dependency key (such as `version`)."),
    ("specifying-dependencies", "Does a git dependency on a branch update automatically when the branch moves?",
     "Cargo locks the commits of `git` dependencies in `Cargo.lock` file at the time of their addition\nand checks for updates only when you run `cargo update` command."),
    ("specifying-dependencies", "Can I use a local path during development and the crates.io version once published?",
     "The `git` or `path` dependency will be used locally (in which case\nthe `version` is checked against the local copy), and when published to a\nregistry like [crates.io], it will use the registry version."),
    ("specifying-dependencies", "Does crates.io accept a dependency with version \"*\"?",
     "[crates.io] does not allow bare `*` versions."),
    ("specifying-dependencies", "How do I see which cfg values another target platform has?",
     "run `rustc --print=cfg --target=x86_64-pc-windows-msvc`"),
    ("specifying-dependencies", "Does the version key decide which git commit gets used?",
     "The `version` key does _not_ affect which commit is used when Cargo retrieves the `git` dependency"),
    ("specifying-dependencies", "I renamed an optional dependency with the package key. What is its feature called?",
     "That is, names of features take after the\nname of the dependency, not the package name, when renamed."),
    ("specifying-dependencies", "Will a requirement of 1.0.0-alpha upgrade me to 1.0.1-beta?",
     "Note that this only works on the same release\nversion, `foo = \"1.0.0-alpha\"` will not allow updating to `foo = \"1.0.1-alpha\"`\nor `foo = \"1.0.1-beta\"`."),
    ("features", "How do I stop an optional dependency from creating a feature with its own name?",
     "If you specify the optional dependency with the `dep:` prefix anywhere\nin the `[features]` table, that disables the implicit feature."),
    ("features", "What does the question mark in \"rgb?/serde\" mean?",
     "You can add a `?` as in `\"package-name?/feature-name\"` which will only enable\nthe given feature if something else enables the optional dependency."),
    ("features", "Two crates in my tree enable different features of the same dependency. What gets built?",
     "When a dependency is used by multiple packages, Cargo will use the union of\nall features enabled on that dependency when building it."),
    ("features", "How can a build script find out which features are enabled?",
     "inspecting the `CARGO_FEATURE_<name>` environment variable"),
    ("features", "How many features may a new crate have on crates.io?",
     "a maximum of 300 features"),
    ("features", "Should I add a no_std feature to my library?",
     "**do\nnot** use a `no_std` feature. Instead, use a `std` feature that *enables*\n`std`."),
    ("features", "With resolver 2, do dev-dependencies turn on features in a normal cargo build?",
     "[Dev-dependencies] do not activate features unless building a [Cargo target][target] that\n  needs them (like tests or examples)."),
    ("features", "How do I build with every feature of the selected packages turned on?",
     "* `--all-features`: Activates all features of all packages selected on the command line."),
    ("features", "Is removing a feature fine in a minor release?",
     "* The following should usually **not** be done in a minor release:\n  * [Remove a feature][cargo-feature-remove] or [optional dependency][cargo-remove-opt-dep]."),
    ("features", "How can I find out which package turned on a feature?",
     "* `cargo tree -e features`: This will show features in the dependency graph.\n  Each feature will appear showing which package enabled it."),
    ("features", "With resolver 2, does --no-default-features affect every workspace member?",
     "With\nversion \"2\", it will disable the default features for all workspace members."),
    ("features", "Two of my features must never be enabled together. What should I do?",
     "If it is not possible, consider adding a compile\nerror to detect this scenario."),
    ("workspaces", "In a virtual workspace without default-members, which packages does cargo build build?",
     "In the case of a [virtual workspace](#virtual-workspace), all members will be used\n(as if `--workspace` were specified on the command-line)."),
    ("workspaces", "Can a member crate inherit rust-version from the workspace?",
     "Keys that are supported:\n\n|                |                 |\n|----------------|-----------------|\n| `authors`      | `categories`    |\n| `description`  | `documentation` |\n| `edition`      | `exclude`       |\n| `homepage`     | `include`       |\n| `keywords`     | `license`       |\n| `license-file` | `publish`       |\n| `readme`       | `repository`    |\n| `rust-version` | `version`       |"),
    ("workspaces", "Does a virtual workspace need to set resolver?",
     "[`resolver`](resolver.md#resolver-versions) must be\n  set explicitly in virtual workspaces"),
    ("workspaces", "Where does Cargo put the Cargo.lock file in a workspace?",
     "All packages share a common [`Cargo.lock`] file which resides in the\n  *workspace root*."),
    ("workspaces", "Can workspace members be listed with a glob pattern?",
     "The `members` list also supports [globs] to match multiple paths, using\ntypical filename glob patterns like `*` and `?`."),
    ("workspaces", "Will Cargo warn about keys I put under workspace.metadata?",
     "The `workspace.metadata` table is ignored by Cargo and will not be warned\nabout."),
    ("workspaces", "Do path dependencies inside the workspace folder become members automatically?",
     "All [`path` dependencies] residing in the workspace directory automatically\nbecome members."),
    ("workspaces", "Since which Rust version is workspace.lints respected?",
     "> **MSRV:** Respected as of 1.74"),
]

Measure at equal cost, too. Bigger chunks buy recall with prompt tokens: five 1,000-token chunks contain more of the corpus than five 100-token chunks, whatever their boundaries. So the ruler reports three numbers. Recall@1 and recall@5 are the usual ones. The third fills a fixed budget of 1,000 tokens with chunks in rank order and stops at the first chunk that does not fit, so every strategy gets the same prompt space. That third number is the gold-in-prompt rate of Context Augmentation, for the simplest possible assembler.

QV = model.encode([q for _, q, _ in QUESTIONS], normalize_embeddings=True)

def rank(chunks, vectors=None, qvecs=QV):
    """For each question, all chunks from best to worst."""
    vectors = index(chunks) if vectors is None else vectors
    return [[chunks[i] for i in np.argsort(-(vectors @ qv))] for qv in qvecs]

def evaluate(ranked, k=5, budget=None):
    """A question counts as found when every non-space character of its answer lies inside
    the chunks sent: the top k, or, with a token budget, chunks in rank order until the next
    one would not fit."""
    found, sent = [], []
    for (page, _, answer), results in zip(QUESTIONS, ranked):
        start = DOCS[page].index(answer)
        need = {i for i in range(start, start + len(answer)) if not DOCS[page][i].isspace()}
        used = 0
        for r, c in enumerate(results):
            size = n_tokens(c["text"])
            if (budget is None and r == k) or (budget is not None and used + size > budget):
                break
            used += size
            if c["doc"] == page:
                need -= set(range(c["start"], c["end"]))
        found.append(not need)
        sent.append(used)
    return np.array(found), round(float(np.mean(sent)))

def report(name, ranked):
    r1, _ = evaluate(ranked, k=1)
    r5, tokens5 = evaluate(ranked, k=5)
    rb, _ = evaluate(ranked, budget=1000)
    print(f"{name:22} R@1 {r1.mean():.2f}  R@5 {r5.mean():.2f} ({tokens5:,} tokens)  "
          f"R@1k-budget {rb.mean():.2f}")
    return r5

def flips(a, b):
    """Questions a finds and b misses, and the reverse."""
    return f"+{int((a & ~b).sum())} -{int((b & ~a).sum())}"

base = report("fixed 128", rank(windows, W))
# fixed 128              R@1 0.30  R@5 0.60 (645 tokens)  R@1k-budget 0.66

The baseline finds the answer text in its top 5 for 60% of the questions, sending 645 tokens on average. Every move below reports the same three numbers.

How much is a difference worth? With 53 questions, one question is 1.9 points. Compare two strategies question by question, not by their averages. flips counts the questions one finds and the other misses. If A wins 13 and loses 1, a two-sided sign test says a split that lopsided happens about 2 times in 1,000 when the two are equally good. If A wins 5 and loses 2, it happens 45% of the time: no evidence. Retrieval Metrics covers paired tests; on found/missed outcomes its paired randomization test gives the same p-value as this sign test. The habit is what matters here: never ship a 3-point gain measured on 50 questions without looking at the flips.

💡 Build your own set the same way. Take 50–100 real questions from search logs or support tickets, find the sentence or block that answers each one, and store it verbatim with the document version it came from. Pin the corpus version, or your gold spans will stop matching the text.

Move 1: Size chunks in the model's tokens, under every limit

Symptom: questions answered late in long chunks never match, or the chunk sizes in your config are in characters.

The splitter that read a quarter of every chunk

Most text splitters count characters unless you tell them otherwise. LangChain's RecursiveCharacterTextSplitter (in langchain-text-splitters 1.1.2, September 2026) defaults to chunk_size=4000, chunk_overlap=200 and length_function=len: four thousand characters. Here is what the defaults do to this corpus:

from langchain_text_splitters import RecursiveCharacterTextSplitter

def split_with(splitter):
    chunks = []
    for doc, text in DOCS.items():
        for d in splitter.create_documents([text]):
            a = d.metadata["start_index"]
            chunks.append({"doc": doc, "start": a, "end": a + len(d.page_content),
                           "text": d.page_content})
    return chunks

defaults = split_with(RecursiveCharacterTextSplitter(add_start_index=True))
sizes = [n_tokens(c["text"]) for c in defaults]
print(len(defaults), "chunks,", sum(s > LIMIT for s in sizes), "over the limit, largest", max(sizes))
report("splitter defaults", rank(defaults))
# 21 chunks, 21 over the limit, largest 1247
# splitter defaults      R@1 0.36  R@5 0.75 (4,862 tokens)  R@1k-budget 0.11

Every chunk is longer than the 256 word pieces MiniLM reads, so every vector describes roughly the first quarter of its chunk. Recall@5 still looks respectable, 0.75. That is the trap the ruler's third column exists for: those five chunks carry 4,862 tokens, a quarter of this whole 19,278-token corpus, into every prompt. Give the defaults the same 1,000-token budget as everyone else and recall collapses to 0.11: for 38 of the 53 questions, the top chunk alone is bigger than the budget.

LlamaIndex's SentenceSplitter (llama-index-core 0.14.25) defaults to 1,024 tokens with 200 overlap, counted with tiktoken's gpt-3.5-turbo encoding unless you pass a tokenizer: four times MiniLM's limit, in another model's tokens. Past the limit, sentence-transformers drops text without a warning; Vector Embeddings shows it on a real document.

Three limits, one tokenizer each

A chunk has to fit three budgets, each counted in a different model's tokens:

The same text is a different number of tokens for every tokenizer and language, so size with the tokenizer of the model that truncates, leave a few tokens of margin, and assert at ingestion that no chunk exceeds the limit. The check is a few lines, and it is the fix for the silent failure the Qdrant lesson found hiding behind hybrid search:

def check_limit(chunks, limit=LIMIT):
    worst = max(n_tokens(c["text"]) for c in chunks)
    assert worst <= limit, f"a chunk of {worst} tokens would be truncated at {limit}"

check_limit(windows)                      # passes: the largest window is 131 tokens
try:
    check_limit(defaults)
except AssertionError as e:
    print(e)
# a chunk of 1247 tokens would be truncated at 256

Run it in the ingestion job, before anything is embedded, and fail the job instead of logging a warning nobody reads.

Cut the source text, don't decode tokens. A tokenizer can turn ids back into text, but not necessarily into your text. MiniLM's tokenizer lowercases and splits punctuation:

ids = tok("[profile.release]\nopt-level = 3", add_special_tokens=False)["input_ids"]
print(repr(tok.decode(ids)))
# '[ profile. release ] opt - level = 3'

That string can't be quoted, cited or matched against a gold span. fixed_windows asks the tokenizer for each token's character offsets and slices the original text instead, so every chunk is an exact substring with a known start and end.

Your turn. Your pipeline embeds with a 512-token model and reranks with ms-marco-MiniLM-L6-v2 (512 tokens for query and passage together, which includes three special tokens: one at the start and one after each text). User queries run up to 40 tokens, and every chunk carries a heading path of up to 20 tokens. What is the largest chunk body you can allow, and which limit sets it?

Move 2: Fixed windows, and what overlap really buys

Symptom: the answer straddles a boundary, as in the hook.

Fixed windows are the baseline every other strategy has to beat: simple, fast, and blind to meaning. Two details in fixed_windows decide whether it is correct.

The tail. The loop stops as soon as a window reaches the end of the page. The common version, while start < len(tokens), keeps going: with 300 tokens, windows of 128 and an overlap of 32, it emits windows starting at 0, 96, 192 and 288. The last one, tokens 288–300, lies entirely inside the third window, so it is a pure duplicate. The assertion overlap < size matters too: with overlap == size the stride is zero and the loop never ends.

The stride. With overlap o, a window starts every size − o tokens, so the index holds about size / (size − o) times as many tokens.

Predict: a 10,000-token page, windows of 256, overlap 64. How many windows, and how many tokens do you embed?

Now measure what overlap buys on the Cargo pages, including how much of the top 5 repeats text the prompt already has:

def repeated(ranked, k=5):
    """Average share of the text in each top k that repeats text sent earlier in it."""
    shares = []
    for results in ranked:
        chars = [(c["doc"], i) for c in results[:k] for i in range(c["start"], c["end"])]
        shares.append(1 - len(set(chars)) / len(chars))
    return float(np.mean(shares))

for size, overlap in [(128, 0), (128, 32), (128, 64), (250, 0)]:
    chunks = fixed_windows(size, overlap)
    ranked = rank(chunks)
    print(f"{len(chunks)} chunks, {sum(n_tokens(c['text']) for c in chunks):,} tokens to embed, "
          f"{repeated(ranked):.0%} of top-5 text repeated")
    r5 = report(f"fixed {size}, overlap {overlap}", ranked)
    print("   vs fixed 128:", flips(r5, base))

153 chunks, 19,581 tokens to embed, 0% of top-5 text repeated
fixed 128, overlap 0   R@1 0.30  R@5 0.60 (645 tokens)  R@1k-budget 0.66
   vs fixed 128: +0 -0
201 chunks, 25,983 tokens to embed, 7% of top-5 text repeated
fixed 128, overlap 32  R@1 0.28  R@5 0.66 (649 tokens)  R@1k-budget 0.70
   vs fixed 128: +5 -2
299 chunks, 38,757 tokens to embed, 18% of top-5 text repeated
fixed 128, overlap 64  R@1 0.30  R@5 0.62 (649 tokens)  R@1k-budget 0.75
   vs fixed 128: +5 -4
79 chunks, 19,428 tokens to embed, 0% of top-5 text repeated
fixed 250, overlap 0   R@1 0.42  R@5 0.74 (1,242 tokens)  R@1k-budget 0.64
   vs fixed 128: +10 -3

Read it the way you would read an experiment you paid for:

• Overlap of 32 (25%) found 5 answers the plain windows missed and lost 2. That is the effect overlap is supposed to have, answers that straddle a cut now appear whole in some window, but +5 −2 is not evidence (the sign test above: 45%). It cost a third more vectors and embedding work, for every document, forever.
• Overlap of 64 (50%) doubled the index and did no better at k = 5. The copies crowd the results: on average 18% of the text in its top 5 repeats text the prompt already has. At the 1,000-token budget it did reach 0.75, the best fixed-window score, but against plain windows that is +8 −3, which happens by chance about 23% of the time, and it costs twice the vectors. Overlap trades index size for boundary insurance, and past a point it buys duplicates.
• Windows of 250 look much better at k = 5 (0.74) until you notice they send twice the tokens. At the same 1,000-token budget they score 0.64 against 0.66: bigger windows mostly bought recall with prompt space.

This agrees with Chroma's 2024 chunking report (Smith & Troynikov), which measured token-level recall and precision across five corpora: reducing overlap improved their precision-style metrics, and a 200-token recursive splitter with no overlap was, in their words, "consistently high performing across all evaluation metrics". Treat any overlap percentage as a hypothesis to test on your own questions, as Classic RAG Pipeline already says.

⚠️ If you keep overlap, merge overlapping spans from the same document before building the prompt, so the model doesn't read the same sentence twice (Context Augmentation shows the merge).

Move 3: Cut where the text already breaks

Symptom: chunks start mid-sentence ("the release profile are:"), lists lose their first items, code blocks are cut in half.

A window cuts wherever its token count runs out. A recursive splitter tries a list of separators from coarsest to finest: paragraph breaks ("\n\n"), then line breaks, then spaces, then anything. It packs whole paragraphs into a chunk while they fit. Only a paragraph that is too big on its own gets split at lines, and only a line that is too big gets split at spaces. Cuts land on the biggest natural boundary that the budget allows.

The same LangChain class does this properly once you give it the model's tokenizer:

recursive = split_with(RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
    tok, chunk_size=LIMIT - 8, chunk_overlap=0, add_start_index=True))
assert max(n_tokens(c["text"]) for c in recursive) <= LIMIT
print(len(recursive), "chunks")
report("recursive 248", rank(recursive))
# 94 chunks
# recursive 248          R@1 0.53  R@5 0.77 (1,076 tokens)  R@1k-budget 0.74

from_huggingface_tokenizer measures length with tokenizer.tokenize(), which does not count [CLS] and [SEP], so the budget is the limit minus 2 special tokens and a few tokens of margin. Joining pieces can shift a count by a token or two, which is why the assertion stays. (transformers prints a warning about a sequence longer than 256, 506 > 256, when the splitter measures a piece that is too big and has to be split further. That is the splitter counting, not the model truncating.)

Against fixed windows of about the same size (250), the recursive splitter moved recall@1 from 0.42 to 0.53 and sent fewer tokens (1,076 against 1,242), because a chunk now ends where a paragraph ends instead of filling every window to the brim. At the 1,000-token budget it scored 0.74 against 0.64. Suggestive, not proven: the flips are +10 −4 at k = 1 and +9 −4 at the budget, and at k = 5 only +8 −6.

Sentences. The separator list goes from line breaks straight to spaces. A paragraph too big for the budget is therefore cut at a line break, and in hard-wrapped Markdown like this source a line break usually sits in the middle of a sentence. Here it happened twice, both inside one 506-token warning box on the Specifying Dependencies page. Put a sentence boundary between paragraphs and lines:

text = DOCS["specifying-dependencies"]
box = text.index("> [!CAUTION]")                     # a 506-token warning box, hard-wrapped
box_end = text.index("\n\n", box)
SENTENCE_END = r"(?<=[.!?])\s+(?=>?\s*[A-Z])"         # . ! or ?, then a capital letter
by_sentence = split_with(RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
    tok, chunk_size=LIMIT - 8, chunk_overlap=0, add_start_index=True, is_separator_regex=True,
    separators=["\n\n", SENTENCE_END, "\n", " ", ""]))
for chunks in (recursive, by_sentence):
    print([c["text"][-24:] for c in chunks
           if c["doc"] == "specifying-dependencies" and box < c["end"] < box_end])
report("recursive + sentences", rank(by_sentence))
# ['ver incompatible version', 'cified `=1.2.3` might be']
# ['blic dependency" to use.', 'ble error (see [#9029]).']
# recursive + sentences  R@1 0.53  R@5 0.77 (1,073 tokens)  R@1k-budget 0.74

Both cuts now fall after a full stop. The lookbehind keeps the full stop with its sentence, and the lookahead asks for a capital letter (after an optional > quote marker), because the plain pattern (?<=[.!?])\s+ cut this same box twice after "e.g.". Recall didn't move: two chunks out of 94 are too few to show in 53 questions. On text whose paragraphs are longer than the budget, though, every paragraph gets cuts of this kind. LlamaIndex's SentenceSplitter packs whole sentences by default: it splits by paragraph, then by sentence with NLTK's Punkt tokenizer (which was built to recognise abbreviations), then at clause punctuation, then by word.

Two things the recursive splitter still does not know:

• It has no idea what a heading is. "\n\n" before ### release is just another paragraph break. 18 of its 94 chunks end with a heading whose content went into the next chunk.
• It splits code at blank lines. A TOML block with an empty line inside is two "paragraphs" to it.

LangChain also ships Markdown-flavoured separators (RecursiveCharacterTextSplitter.from_language(Language.MARKDOWN) starts with a heading pattern, "\n#{1,6} "), but that pattern can't tell a heading from a comment line inside a code block. That's the next move.

Move 4: Follow the document's structure

Symptom: a chunk doesn't say what it is about. The opt-level = 3 window in the hook is the textbook case: the value is there, the words that make it findable are not.

The author already divided the page into units: sections under headings. So split by section first, then run the recursive splitter inside a section only if the section is too long. Put the heading path (Profiles > Default profiles > release) on top of every chunk, so a piece from the middle of a long section still says where it lives.

One trap before the code. The Profiles page contains this block:

# Set the default for dependencies.
[profile.dev.package."*"]
opt-level = 2

A regular expression that treats every line starting with # as a heading turns that TOML comment into a top-level section called "Set the default for dependencies". Track code fences and skip lines inside them:

HEADING = re.compile(r"(#{1,6}) +(.+?)\s*$")

def sections(text):
    """(heading path, start, end) for every Markdown section. A line inside a ``` fence is
    code, so a TOML or shell comment such as '# Set the default' is not a heading."""
    marks, pos, in_code = [], 0, False
    for line in text.splitlines(keepends=True):
        if line.startswith("```"):
            in_code = not in_code
        m = None if in_code else HEADING.match(line)
        if m:
            marks.append((pos, len(m.group(1)), m.group(2)))
        pos += len(line)
    out, stack = [], []
    for i, (start, level, title) in enumerate(marks):
        end = marks[i + 1][0] if i + 1 < len(marks) else len(text)
        stack = [(l, t) for l, t in stack if l < level] + [(level, title)]
        out.append(([t for _, t in stack], start, end))
    return out

def structure_chunks(budget=LIMIT - 6, with_path=True):
    chunks = []
    for doc, text in DOCS.items():
        for path, s, e in sections(text):
            head = " > ".join(path) + "\n" if with_path else ""
            splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
                tok, chunk_size=budget - n_tokens(head), chunk_overlap=0, add_start_index=True)
            for d in splitter.create_documents([text[s:e]]):
                a = s + d.metadata["start_index"]
                b = a + len(d.page_content)
                chunks.append({"doc": doc, "start": a, "end": b, "path": " > ".join(path),
                               "text": head + text[a:b]})
    return chunks

structured = structure_chunks()
assert max(n_tokens(c["text"]) for c in structured) <= LIMIT
S = index(structured)
print(len(structured), "chunks")
c, score = search(question, structured, S, k=1)[0]
print(f"{score:.3f}", repr(c["text"][:90]))
best = report("structure + path", rank(structured, S))
report("structure, no path", rank(structure_chunks(with_path=False)))
print("structure vs fixed 128:", flips(best, base))

129 chunks
0.511 'Profiles > Default profiles > release\n### release\n\nThe `release` profile is intended for o'
structure + path       R@1 0.53  R@5 0.83 (812 tokens)  R@1k-budget 0.85
structure, no path     R@1 0.55  R@5 0.81 (786 tokens)  R@1k-budget 0.85
structure vs fixed 128: +13 -1

The hook question now gets the whole release section as its top result, 153 tokens including opt-level = 3. Across the set, structure-aware chunks found 13 answers the 128-token windows missed and lost 1 (sign test: about 2 in 1,000 by chance). They also beat the 250-token windows 11 to 0 at the 1,000-token budget. This is the first difference in this lesson that is clearly not noise.

Why it works

A section is a unit the author wrote to be read together, so its heading, its explanation and its values share one vector. Sections vary in length (here from 10 to 250 tokens after splitting, 161 on average), and short, single-topic chunks help twice. Dense retrieval gets one vector per topic instead of a blend. Lexical retrieval gets shorter documents, and BM25's length normalization stops penalizing the right chunk for sharing space with three other topics, the problem Sparse vs Dense Retrieval ran into with a long pump manual.

The heading path did not move recall here (0.83 with it, 0.81 without, +2 −1). Two reasons. Every section already begins with its own heading line, so most chunks name themselves. And few questions depend on the parent heading. Keep the path anyway, for reasons the ruler doesn't score: it disambiguates look-alike sections (this corpus has two debug headings, one a setting and one a profile), it tells the generator and the reader where a quote came from, and it is metadata you can filter on. In a corpus of look-alike sections, Classic RAG Pipeline measured section chunks that carry their heading landing clear of their look-alike neighbours (0.657 against 0.472; the heading was not tested on its own). Measure it on yours.

Tables and code need their own units

Tables. A row means nothing without its header, and here even the header is empty: the Workspaces page introduces its table with the sentence "Keys that are supported:" and a blank header row. Keep small tables whole. Split a large one into row groups, and repeat the lead sentence and header in every group:

def split_table(table, lead, budget):
    """Row groups that fit the budget; each repeats the lead text and the header rows."""
    header, rule, *rows = table.strip().splitlines()
    pieces, rows_in = [], []
    for row in rows:
        trial = "\n".join([lead, header, rule, *rows_in, row])
        if rows_in and n_tokens(trial) > budget:
            pieces.append("\n".join([lead, header, rule, *rows_in]))
            rows_in = []
        rows_in.append(row)
    pieces.append("\n".join([lead, header, rule, *rows_in]))
    return pieces

text = DOCS["workspaces"]
start = text.index("|  ")
table = text[start:text.index("\n\n", start)]
lead = "Workspaces > The `package` table\nKeys that are supported:"
for piece in split_table(table, lead, budget=90):
    print(n_tokens(piece), piece.splitlines()[-1])
# 82 | `edition`      | `exclude`       |
# 86 | `license-file` | `publish`       |
# 76 | `rust-version` | `version`       |

Splitting keeps a table readable. It doesn't make it findable. The question "Can a member crate inherit rust-version from the workspace?" is the hardest in the set: under structure-aware chunking the section holding the whole table ranks 19th, only small-to-big (Move 6) gets it into a top five (at rank 5), and no strategy in this lesson gets it inside the 1,000-token budget. A grid of key names embeds poorly against a question in prose. The fix is upstream: render rows as sentences, or add a line that says what the table lists (Document Processing owns tables to text).

Code files. For source code the structure is the syntax tree. A chunk per function or class, with its decorators and docstring, named by its path, is the code version of a section with a heading path:

import ast

def code_chunks(source, module):
    """One chunk per top-level function or class, decorators included, named by its path."""
    lines = source.splitlines(keepends=True)
    for node in ast.parse(source).body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            first = min([node.lineno] + [d.lineno for d in node.decorator_list]) - 1
            yield f"{module} > {node.name}", "".join(lines[first:node.end_lineno])

SOURCE = '''import functools

@functools.cache
def parse_version(req: str) -> tuple[int, ...]:
    """Parse '1.2.3' into (1, 2, 3)."""
    return tuple(int(p) for p in req.split("."))


class Resolver:
    """Picks the newest version that satisfies every requirement."""

    def __init__(self, index):
        self.index = index

    def pick(self, name, req):
        ok = [v for v in self.index[name] if v >= parse_version(req)]
        return max(ok) if ok else None
'''
for path, chunk in code_chunks(SOURCE, "resolver.py"):
    print(path, "|", chunk.splitlines()[0])

from langchain_text_splitters import Language
by_separators = RecursiveCharacterTextSplitter.from_language(
    Language.PYTHON, chunk_size=120, chunk_overlap=0)      # 120 characters, to make cuts visible
for piece in by_separators.split_text(SOURCE):
    print(repr(piece[:40]))

resolver.py > parse_version | @functools.cache
resolver.py > Resolver | class Resolver:
'import functools\n\n@functools.cache'
'def parse_version(req: str) -> tuple[int'
'return tuple(int(p) for p in req.split("'
'class Resolver:\n    """Picks the newest '
'def __init__(self, index):\n        self.'
'def pick(self, name, req):\n        ok = '
'return max(ok) if ok else None'

code_chunks skips module-level lines. In a real pipeline, put imports and constants into a chunk of their own, and split a class that is too big for the budget per method, with a path such as resolver.py > Resolver > pick. Now compare LangChain's Language.PYTHON separators ("\nclass ", "\ndef ", "\n\tdef ", then blank lines): with a 120-character budget they left @functools.cache at the end of the import chunk, cut two return lines away from their def, and stripped the leading indentation of each method. At realistic sizes whole functions usually fit, but the separators only know where a definition starts. For other languages, parsers such as tree-sitter give you a syntax tree to split on.

What each chunk should carry

A chunk is more than its text. Store, next to the vector: the document id, the heading path, a section anchor for citation links, the character range, the token count and the embedding model's name. The document id stays the same across edits (the source system's id or the page's path). The version and a content hash go in the payload, never into the document id.

One property keeps everything that stores chunk ids working: after an edit, an id never names different text. Two schemes have it, as long as every re-ingest writes the new chunks first and then deletes the document's points that are not in the new set:

• Position plus version, f"{doc_id}:{version}:{chunk_index}". Every edit writes a complete new set of ids, and the delete removes the old set.
• Content-based, f"{doc_id}:{hash of the chunk text}". A chunk whose text did not change keeps its id and its vector, so an edit re-embeds only what changed. Two identical chunks in one document need a tie-breaker, such as a count of earlier copies.

Qdrant accepts only unsigned integers and UUIDs as point ids, so turn either string into a UUID with uuid.uuid5. Position without a version gives that property up: insert a section at the top of a page, and every later id now names different text, which a plain upsert overwrites without complaint. The index still ends up right if every update deletes the leftovers (Data Pipeline & Indexing and Embedding Pipeline do exactly that), but anything that stored those ids, such as evaluation labels, citation logs or caches keyed by point id, now points at other text (Challenge 3).

How much content-based ids save depends on where your boundaries fall. Insert a two-line section near the top of the Profiles page and re-chunk it both ways:

import hashlib, uuid

def content_id(doc_id, chunk_text):
    """The same text in the same document always gets the same UUID."""
    digest = hashlib.sha256(chunk_text.encode()).hexdigest()
    return str(uuid.uuid5(uuid.NAMESPACE_URL, f"{doc_id}:{digest}"))

page = DOCS["profiles"]
new_section = ("## Profiles at a glance\n\nUse `dev` while you work, `release` for what "
               "you ship, and a custom profile when neither fits.\n\n")
edited = page.replace("## Profile settings", new_section + "## Profile settings", 1)

for name, chunker in [("fixed 128", lambda: fixed_windows(128)), ("structure", structure_chunks)]:
    before = [c["text"] for c in chunker() if c["doc"] == "profiles"]
    DOCS["profiles"] = edited
    after = [c["text"] for c in chunker() if c["doc"] == "profiles"]
    DOCS["profiles"] = page                                  # back to the pinned text
    moved = sum(a != b for a, b in zip(before, after))
    old_ids = {content_id("profiles", t) for t in before}
    new_ids = {content_id("profiles", t) for t in after}
    print(f"{name}: {len(before)} -> {len(after)} chunks | position ids now on other text: {moved}"
          f" | content ids to embed: {len(new_ids - old_ids)}, to delete: {len(old_ids - new_ids)}")

Predict: for each chunker, how many chunks of the edited page need a new embedding under content-based ids?

fixed 128: 34 -> 35 chunks | position ids now on other text: 33 | content ids to embed: 34, to delete: 33
structure: 32 -> 33 chunks | position ids now on other text: 30 | content ids to embed: 1, to delete: 0

Ids built from a section anchor, as the help center in Classic RAG Pipeline does (returns#used-gear), also hold up when leftovers are deleted, on two conditions. The anchor must be unique: this page has two debug headings, so build it from the heading path, not the heading alone. And a renamed heading gives a new id, so the old one goes out with the leftovers. Such an id keeps its value when the section's text changes, so on its own it can't tell you what to re-embed; a content hash in the payload can (Embedding Pipeline). The update procedure is in Classic RAG Pipeline, and running it at scale is Data Pipeline & Indexing.

Your turn. An API reference has one section per endpoint. Most sections are 150 tokens, but POST /orders is 1,900 tokens: a paragraph, a 60-row parameter table and two JSON examples. Your embedding limit is 512. What chunks do you make for that section?

Move 5: Semantic chunking, and when it doesn't pay

Symptom: long prose with no headings (transcripts, interviews, some reports) where topics change mid-page.

When the author left no structure, you can try to find it. Semantic chunking embeds each sentence, measures how far each sentence's vector is from the next one's, and cuts where that distance is unusually large. "Unusually" needs a threshold. The common default is a percentile of the distances in the document: LangChain's experimental SemanticChunker and LlamaIndex's SemanticSplitterNodeParser both default to the 95th percentile (checked in langchain-experimental 0.4.2 and llama-index-core 0.14.25; the former now warns that the package is being sunset). Both also embed each sentence together with its neighbours; the sketch below skips that.

UNIT = re.compile(r"```.*?```|[^\n]+?(?:[.!?](?=\s)|$)", re.S | re.M)

def semantic_chunks(percentile=95, budget=LIMIT - 6):
    """Cut where the next sentence's embedding is unusually far from this one's."""
    fit = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
        tok, chunk_size=budget - 2, chunk_overlap=0, add_start_index=True)
    chunks = []
    for doc, text in DOCS.items():
        units = [m.span() for m in UNIT.finditer(text) if m.group().strip()]
        V = model.encode([text[a:b] for a, b in units], normalize_embeddings=True)
        dist = 1 - (V[1:] * V[:-1]).sum(axis=1)      # cosine distance to the next unit
        cut = np.percentile(dist, percentile)
        groups, a0 = [], units[0][0]
        for (a, b), (_, prev_b), d in zip(units[1:], units[:-1], dist):
            if d > cut or n_tokens(text[a0:b]) > budget:
                groups.append((a0, prev_b))
                a0 = a
        groups.append((a0, units[-1][1]))
        for a, b in groups:                            # a single huge unit still has to fit
            for d in fit.create_documents([text[a:b]]):
                s = a + d.metadata["start_index"]
                chunks.append({"doc": doc, "start": s, "end": s + len(d.page_content),
                               "text": d.page_content})
    return chunks

semantic = semantic_chunks()
print(len(semantic), "chunks")
sem = report("semantic p95", rank(semantic))
print("structure vs semantic:", flips(best, sem))
# 123 chunks
# semantic p95           R@1 0.34  R@5 0.70 (874 tokens)  R@1k-budget 0.70
# structure vs semantic: +9 -2

The units are sentences, list lines, and whole fenced code blocks. To produce 123 chunks, the chunker first embedded 1,238 units: one extra embedding pass over the entire corpus at ingestion, repeated whenever a page changes.

It lost. Recall@5 0.70 against 0.83 for structure-aware chunks, and recall@1 0.34 against 0.53 (13 wins to 3 for structure at k = 1). On documentation this is what you should expect. The headings already mark the topic shifts, exactly, for free. The vector of a whole TOML block, or of a single bullet line, is a noisy neighbour for the prose around it, so the percentile cut lands in odd places. The chunk that holds ### release begins with the dev profile's TOML block (opt-level = 0) and does not contain the release values at all.

Published evaluations point the same way:

• Chroma's 2024 report (Smith & Troynikov) found differences of up to 9% in recall between chunkers. With text-embedding-3-large and 5 retrieved chunks, the popular Kamradt-style semantic chunker at default settings recalled 83.6% of the evidence tokens against 88.1% for a 200-token recursive splitter with no overlap. Their own embedding-based ClusterSemanticChunker scored best on precision and IoU, so "semantic" is not one method with one result.
• Qu, Tu & Bao (arXiv, 2024) compared semantic and fixed-size chunking on document retrieval, evidence retrieval and answer generation, and concluded that "the computational costs associated with semantic chunking are not justified by consistent performance gains."

Where semantic cuts can still earn their cost: long unstructured text with real topic changes. A support-call transcript that moves from billing to a broken device is the classic case. Even there, try the structure the text does have first (speaker turns, timestamps, paragraphs), and compare on labeled questions.

LLM-made units. A step further, an LLM rewrites each passage into small self-contained statements. Chen et al. (Dense X Retrieval, EMNLP 2024) indexed such propositions and report that indexing by these fine-grained units "significantly outperforms passage-level units in retrieval tasks" on their benchmarks. The cost is a model pass over every passage at ingestion, and a second copy of your corpus in generated text that can contain generation errors. Keep the source passage as the unit you quote, and measure the gain on your own set before you pay it.

Move 6: Search small, send big

Symptom: small chunks match questions precisely, but the answer needs the sentences around them. Or big chunks carry the context, but their vectors blur.

Look at the tension in the numbers so far. Small units match well, because one vector describes one thing. Big units are what the generator needs, because a setting's value means little without its explanation. Parent–child retrieval (also called small-to-big) stops making one chunk serve both jobs. Index small children for matching. When a child matches, send its parent section.

parents = structured
children = []
for p_id, p in enumerate(parents):
    head = p["path"] + "\n"
    splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
        tok, chunk_size=64 - n_tokens(head), chunk_overlap=0, add_start_index=True)
    for d in splitter.create_documents([DOCS[p["doc"]][p["start"]:p["end"]]]):
        a = p["start"] + d.metadata["start_index"]
        children.append({"doc": p["doc"], "start": a, "end": a + len(d.page_content),
                         "parent": p_id, "text": head + d.page_content})
C = index(children)

def small_to_big(qv):
    """Rank the small children, return their parents, each parent once."""
    seen, out = set(), []
    for i in np.argsort(-(C @ qv)):
        if children[i]["parent"] not in seen:
            seen.add(children[i]["parent"])
            out.append(parents[children[i]["parent"]])
    return out

print(len(children), "children for", len(parents), "parents")
s2b = report("small-to-big", [small_to_big(qv) for qv in QV])
print("small-to-big vs structure:", flips(s2b, best))
# 586 children for 129 parents
# small-to-big           R@1 0.62  R@5 0.89 (909 tokens)  R@1k-budget 0.87
# small-to-big vs structure: +5 -2

Children of at most 64 tokens, each with its parent's heading path, rank the same 129 parent sections. Recall@1 rose from 0.53 to 0.62 and recall@5 from 0.83 to 0.89, for a similar number of prompt tokens (909 against 812). The flips at k = 5 are +5 −2, so on 53 questions this is promising, not proven. At k = 1 it is 8 wins to 3.

What it costs:

• 4.5 times the vectors (586 against 129), plus parent text stored somewhere you can fetch by id. Vectors dominate the memory bill, so size it before you commit.
• Parents can outgrow the embedding limit. Parents are never embedded, so they only need to fit the prompt budget (and the reranker's limit, if you rerank parents rather than children). Using whole, unsplit sections (up to 692 tokens) as parents lifted recall@5 to 0.96 in a separate run, but at 1,596 prompt tokens, and at the 1,000-token budget it scored 0.89 against 0.87: that is the budget trade again.
• Deduplicate by parent. Three children of the same section should produce one parent, not three copies.

In Qdrant, grouping does the deduplication. Store the parent id in each child's payload, keep parents in a second collection (no vectors needed), and let a grouped query return one hit per parent with the parent attached:

from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")                     # or your server URL
client.create_collection("parents", vectors_config={})
client.create_collection("children", vectors_config=models.VectorParams(
    size=384, distance=models.Distance.COSINE))
client.upsert("parents", [models.PointStruct(id=i, vector={}, payload={
    "path": p["path"], "text": p["text"]}) for i, p in enumerate(parents)])
client.upsert("children", [models.PointStruct(id=i, vector=C[i].tolist(), payload={
    "parent_id": c["parent"]}) for i, c in enumerate(children)])

groups = client.query_points_groups(
    "children", query=model.encode(question).tolist(), group_by="parent_id",
    limit=3, group_size=1,
    with_lookup=models.WithLookup(collection="parents", with_payload=True))
for g in groups.groups:
    print(g.id, round(g.hits[0].score, 3), g.lookup.payload["path"])
# 20 0.589 Profiles > Default profiles > release
# 24 0.527 Profiles > Custom profiles
# 23 0.518 Profiles > Default profiles > Build Dependencies

Pass with_payload=True explicitly: with qdrant-client 1.19.1 in local mode, the plain string form with_lookup="parents" returned the parents without their payload, while the 1.19.1 server returned it. The same code gave the same three groups against the server. Grouping and the rest of the client API are in Qdrant. Frameworks package the pattern too: LlamaIndex's HierarchicalNodeParser (default chunk sizes 2,048, 512 and 128 tokens) with its AutoMergingRetriever, for example. Check which tokenizer those sizes are counted in.

This move decides what is indexed. What happens at prompt time (expanding hits to their neighbours, merging overlapping spans, reranking before you expand) belongs to Context Augmentation.

Your turn. You plan to go live with this design on a corpus of 1,000,000 parent sections, with the same ratio of children to parents and 384-dimensional float32 vectors. How much memory do the vectors need, compared with indexing the parents directly?

Move 7: Put the page's context into the vector

Symptom: a chunk only makes sense with its page. The Profiles page lists eleven settings, and three of their sections contain the same two lines: "true: enabled" and "false: disabled". Cut out of the page, that list could belong to any of them.

The heading path from Move 4 is the cheapest fix. Two stronger ones change what the vector itself knows.

Late chunking

An ordinary chunk vector is computed from the chunk alone. Late chunking (Günther et al., arXiv 2024) runs a long-context embedding model over the whole page first, so every token's vector has attended to the entire page, and only then averages the token vectors inside each chunk: chunking is applied "after the transformer model and just before mean pooling". It needs a model that reads the whole page (or long windows of it) and that uses mean pooling. It needs no training and no LLM calls. The index-time cost is one long forward pass per page instead of many short ones. Every chunk vector now depends on the whole page, so an edit anywhere on the page means re-embedding all of its chunks, even those whose text did not change: content-based ids can't skip them.

Here it is with nomic-ai/modernbert-embed-base (Apache 2.0, 8,192 tokens, mean pooling, about a 0.6 GB download; its card requires the search_document: and search_query: prefixes), on windows of 64 and 128 of its own tokens:

long_model = SentenceTransformer("nomic-ai/modernbert-embed-base")   # 8,192 tokens, mean pooling
ltok = long_model.tokenizer
DOC_PREFIX, QUERY_PREFIX = "search_document: ", "search_query: "    # from the model card

def late_chunk(doc, chunks):
    """Run the model over the whole page once, then average the token vectors inside each chunk."""
    text = DOC_PREFIX + DOCS[doc]
    states = long_model.encode(text, output_value="token_embeddings").float().cpu().numpy()
    starts = np.array([a for a, b in ltok(text, return_offsets_mapping=True)["offset_mapping"]])
    starts -= len(DOC_PREFIX)                        # offsets into DOCS[doc]
    special = np.array([t in ltok.all_special_ids for t in ltok(text)["input_ids"]])
    vecs = []
    for c in chunks:
        inside = (starts >= c["start"]) & (starts < c["end"]) & ~special
        v = states[inside].mean(axis=0)
        vecs.append(v / np.linalg.norm(v))
    return vecs

LQ = long_model.encode([QUERY_PREFIX + q for _, q, _ in QUESTIONS], normalize_embeddings=True)
for size in (64, 128):
    chunks = fixed_windows(size, tokenizer=ltok)
    naive = long_model.encode([DOC_PREFIX + c["text"] for c in chunks], normalize_embeddings=True)
    late = np.vstack([late_chunk(doc, [c for c in chunks if c["doc"] == doc]) for doc in DOCS])
    a = report(f"{size} naive", rank(chunks, naive, LQ))
    b = report(f"{size} late", rank(chunks, late, LQ))
    print("late vs naive:", flips(b, a))

64 naive               R@1 0.23  R@5 0.53 (326 tokens)  R@1k-budget 0.70
64 late                R@1 0.21  R@5 0.66 (323 tokens)  R@1k-budget 0.87
late vs naive: +9 -2
128 naive              R@1 0.32  R@5 0.70 (645 tokens)  R@1k-budget 0.75
128 late               R@1 0.23  R@5 0.70 (632 tokens)  R@1k-budget 0.83
late vs naive: +5 -5

(Token counts in this table are still MiniLM's, so the budget column means the same thing as before.) The windows are identical in each pair; only the vectors differ. Late chunking helped the smallest windows most. For 64-token windows, recall@5 went from 0.53 to 0.66, but +9 −2 is suggestive, not proven (about 6.5% by chance). At the 1,000-token budget, filled with 64-token pieces, it went from 0.70 to 0.87, and +10 −1 is stronger evidence (about 1% by chance). At 128 tokens, recall@5 did not move (+5 −5). And recall@1 fell in both pairs, from 0.23 to 0.21 and from 0.32 to 0.23. A reasonable reading: page context rescues chunks too short to describe themselves, and adds little to chunks that already can. Try it where you want small units, and measure. The library chonkie (1.7.0) ships a LateChunker, if you'd rather not write the pooling yourself.

Look also at the naive rows. The larger model with plain 128-token windows (in its own tokens) scored 0.70 at k = 5, where MiniLM's 128-token windows scored 0.60. Changing the embedding model moved recall@5 about as much as some splitter changes did (250-token windows reached 0.74), though less than following the structure (0.83), and you test a model the same way (Vector Embeddings). A long-context model is not a license for long chunks, though: one vector for a whole page is a blend of every topic on it, which Vector Embeddings measured as dilution.

Contextual enrichment at index time

The other option asks an LLM to write, for every chunk, a short context that situates it in its document, and indexes that context together with the chunk. Anthropic's Contextual Retrieval is the best-known recipe. The prompt, the reported gains and the prompt-time alternative are covered in Context Augmentation. What this lesson adds is the index-time bill, because chunking decides it:

• One LLM call per chunk, and each call carries the whole page. For our 129 structure-aware chunks that is 129 calls and about 636,000 input tokens (counted with o200k_base as a stand-in tokenizer) to enrich a corpus of about 17,000 tokens: 37 times the corpus. 96% of it is the same four pages sent again and again, which is why prompt caching on the page prefix matters.
• Smaller chunks, more calls. Halving chunk size roughly doubles the bill.
• Every edit re-runs it for every chunk of the edited page, since each context was written from the old page.

Late chunking changes the vectors with no generated text. Contextual enrichment adds words that lexical search can also match, at LLM prices. Both are index-time moves, and both are worth exactly what they gain on your labeled questions.

The scoreboard

Every number below came from the code in this lesson: 53 questions, four Cargo pages, all-MiniLM-L6-v2, recall measured on exact answer spans.

What this table supports, and what it doesn't:

• Size in the model's tokens is not optional: the defaults hide a collapse behind a decent-looking recall@5.
• Follow the structure was the one large, clearly real gain on this corpus (13 to 1 against fixed windows).
• Overlap, the heading path and small-to-big moved numbers by a few questions. Some of that may be real, and 53 questions can't tell. Add questions before you decide.
• Semantic chunking lost on documentation that already has headings.
• It says nothing about your corpus. Transcripts, contracts or source code could rank these rows differently. The ruler is the transferable part: gold spans, recall at k, recall at a fixed budget, flips.

Re-run it whenever the embedding model changes. A new model with a 512-token limit invites bigger chunks, and bigger chunks change every column.

Final round: no label on the problem

Real symptoms don't say which move they need. Read the trace, then pick.

Challenge 1: the crowded top five

A team indexes its runbooks in 200-token windows with 100 tokens of overlap "to be safe". Users complain that answers repeat themselves, and the prompt log shows the same paragraph two or three times per question. Recall@5 is fine. What do you change, and what do you measure?

Challenge 2: the invisible limits table

Questions about rate limits ("How many requests per minute on the Pro plan?") fail, even though a table in the API docs answers them. The table is chunked whole, 180 tokens, under its heading "Limits". It ranks 15th.

Challenge 3: the ids that moved

Chunk ids are f"{doc_id}-{position}". Your evaluation set marks relevant chunks by id, and the answer log stores the ids each answer cited. An editor inserts a new first section into a 40-chunk page, and re-ingestion upserts the page. Overnight, recall on that page's questions drops to almost zero, although searching by hand looks fine, and old citation links open the wrong text.

Cheat sheet: symptom → move → cost

Before moving on, pick one document set you know and write down: its structure (headings, tables, code, speaker turns), the three token limits of your models, the chunk budget that follows from them, and 20 questions with their exact answer text. Then run two chunkers through the ruler and read the flips. If you can explain every flip by pointing at a chunk boundary, you understand chunking.

How would you measure it on your own data? Collect 50–100 real questions and store each answer as an exact span of a pinned document version. For every candidate chunker, report recall@1, recall@5, prompt tokens at k = 5, and recall at your real prompt budget. Compare question by question and count flips. Re-run when the corpus, the embedding model or the reranker changes, and keep the questions that failed as a regression set.

Next: Document Processing for getting clean text, tables and headings out of PDFs and HTML in the first place, Embedding Pipeline for embedding the chunks at scale, Context Augmentation for turning retrieved chunks into a prompt, and Retrieval Metrics for the statistics behind the ruler.