Foundations of Modern AI Search

Two queries, one search box

You run search for an e-bike maker's help center: twelve short help pages and one search box. Two customers show up.

The first types "is my battery still under guarantee after 18 months". Keyword search returns three pages: "Error E-227: the battery voltage is too low", "Error E-217: the battery is above 60 °C" and a page about riding in rain. All three matched on one word: is. The warranty page, which says batteries are covered for 2 years, is not in the results at all. It shares no word with the question. The customer wrote guarantee, the page says warranty; battery against batteries; months against years.

The second customer types what their display shows: "E-217". This time you use embedding search, the meaning-based kind. It ranks "Error E-227" first with a cosine similarity of 0.330, just ahead of the correct page at 0.326. To the embedding model, the code E-217 looks almost exactly like E-227, both pages begin "Error E-2…7: the battery…", and the one character that matters barely moves the vector.

Two failures in opposite directions. Keyword search can't see that different words mean the same thing. Embedding search can't see that one character makes a different thing. Modern AI search is the engineering of handling both on purpose, and proving it with measurements instead of a demo.

That is this course in one paragraph. This lesson is its map: what each kind of retrieval is good at, what it costs, the path a query takes through a modern system, and which later lesson owns each stage. Every number above comes from code you will run on the same twelve pages.

You need Python and the idea of a dot product. The code uses Python 3.12 with rank_bm25, numpy and sentence-transformers; run the blocks in order in one session, because later blocks reuse earlier names.

Before any move: write down what "found it" means

Two retrievers can both return "reasonable-looking" results and be very different systems. So before comparing anything, decide what the judge will observe. First, four words this course uses precisely:

• Document: the source file or record (a help page, a PDF, a ticket).
• Chunk: the unit you embed and index. Here each help page is one chunk.
• Candidate pool: what first-stage retrieval hands to later stages, often 50 to 200 chunks.
• Context set: the few chunks (often 3 to 10) that finally go into an LLM prompt.

The judge is a labeled query set: real queries, each with the chunk or chunks that actually answer it. Two metrics are enough for this lesson (the Retrieval Metrics lesson owns the full toolbox):

• recall@k: for one query, the fraction of its relevant chunks that appear in the top k; then average over queries. With exactly one relevant chunk per query, it simply asks "was it in the top k?"
• MRR (mean reciprocal rank): for each query, 1 divided by the rank of the first relevant chunk (0 if it never appears), averaged over queries.

Predict first: query 1 has one relevant chunk, charge-time, and a system returns [storage, charge-time, warranty]. Query 2 has two relevant chunks, storage and charge-time, and the system returns [storage, warranty, rain]. What are the mean recall@3 and the MRR?

Two habits make the set useful:

• Size it for the change you want to see. With 10 queries, one query moves recall by 0.1, so a toy set only shows big effects. With 50 real queries from your logs, one query is 0.02: a change of five queries (0.1) is worth a closer look, a change of one is noise.
• Slice it by query type. Identifier lookups and descriptive questions fail in opposite directions, as you just saw. An average over both can hide a retriever that is great at one and useless at the other.

⚠️ Two different recalls. ANN recall@k measures how many of the exact nearest neighbors an approximate index returns. Retrieval recall@k (the one above) measures whether the user's answer is found. An index with perfect ANN recall still misses every answer if the embedding model is wrong for your data. Always say which one you mean.

Lexical search: fast, exact, literal

A keyword engine builds an inverted index: for each term, the list of chunks that contain it. Here are four entries from our twelve pages, using the tokenizer in the code below:

term        chunks containing it
battery  -> err-217, err-227, bt-4471, bt-4417, charge-time, storage
e-217    -> err-217
is       -> err-217, err-227, rain
warranty -> warranty

At query time the engine looks up only the query's terms and scores the chunks on those lists, so it never scans the whole corpus. The usual scoring function, BM25, rewards a term that is rare across the corpus and frequent in the chunk, with diminishing returns for repeats and an adjustment for chunk length. Its formula and its k1 and b knobs are taught in Sparse vs Dense Retrieval.

Predict first: for the query E-217, how many chunks will come back?

import re
from rank_bm25 import BM25Okapi

docs = {
    "end-plan":     "Ending your Ride+ subscription: open Settings, choose Billing, then End plan. You keep access until the billing period ends.",
    "cancel-order": "Cancel an order: orders can be cancelled from the Orders page within 30 minutes of purchase.",
    "err-217":      "Error E-217: the battery is above 60 °C. Stop riding and let it cool before charging.",
    "err-227":      "Error E-227: the battery voltage is too low. Charge it before riding.",
    "err-271":      "Error E-271: the motor controller lost contact with the torque sensor. Check the sensor cable.",
    "err-712":      "Error E-712: a firmware update failed. Restart the bike and retry the update.",
    "bt-4471":      "Battery pack BT-4471 (625 Wh) fits Trail models from 2024 and 2025.",
    "bt-4417":      "Battery pack BT-4417 (500 Wh) fits City models from 2022 and 2023.",
    "charge-time":  "A flat battery takes about 4 hours to charge with the standard 4 A charger.",
    "warranty":     "Warranty: frames are covered for 5 years; batteries and motors for 2 years from delivery.",
    "rain":         "Riding in rain is fine. Do not submerge the bike or clean it with a pressure washer.",
    "storage":      "Storing the bike for winter: keep the battery at 30-60 % charge in a dry room.",
}
ids = list(docs)

def tokenize(text):
    # lowercase words and numbers; keep codes such as "e-217" and "bt-4471" as one token
    return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())

bm25 = BM25Okapi([tokenize(docs[i]) for i in ids])   # rank_bm25 defaults: k1=1.5, b=0.75

def search_bm25(query, k=3):
    scores = bm25.get_scores(tokenize(query))
    hits = [(ids[i], float(s)) for i, s in enumerate(scores) if s > 0]   # a zero score is not a hit
    return sorted(hits, key=lambda h: -h[1])[:k]

for q in ["is my battery still under guarantee after 18 months", "E-217"]:
    print(f"{q!r:55} ->", [(d, round(s, 2)) for d, s in search_bm25(q)])

# 'is my battery still under guarantee after 18 months'   -> [('err-227', 1.09), ('err-217', 0.96), ('rain', 0.93)]
# 'E-217'                                                 -> [('err-217', 1.96)]

What lexical search is good at. Exact identifiers, error strings, names and rare terms: the rarer the term, the more it counts. There is no model to train or host, and a new page is searchable as soon as it is indexed. You can show the user which terms matched. It is also a strong baseline: the BEIR benchmark (Thakur et al., 2021) found BM25 robust across 18 retrieval datasets, where several dense retrievers of the time did worse than BM25 outside the domain they were trained on (rerankers and late-interaction models did better, at a higher compute cost).

Where it fails: vocabulary mismatch. Authors and searchers pick their words independently. Furnas et al. (1987) found that two people chose the same term for the same object with probability below 0.20 in every domain they tested. A synonym list patches the pairs you think of and nothing else, and a careless entry can do harm: mapping "cardiac arrest" to "heart attack" in a clinical search would merge two different diagnoses.

⚠️ The analyzer decides what "the same word" means. This toy tokenizer keeps e-217 whole and does no stemming, so battery and batteries never match. An English analyzer with a stemmer (Elasticsearch's english analyzer, for example) would reduce both to batteri, and would still never match guarantee to warranty. A user who types e217 without the hyphen matches nothing at all. Stop words cause the opposite problem: the rain page matched on is because it says "Riding in rain is fine". Analyzer choices are covered in Sparse vs Dense Retrieval.

Meaning-based search: close in meaning, close in space

An embedding model turns a text into a fixed-length list of numbers, a vector, trained so that texts with similar meaning land close together. all-MiniLM-L6-v2 outputs 384 numbers of unit length; its model card lists English and says input longer than 256 word pieces is truncated. You compare vectors with cosine similarity, cos(a, b) = a·b / (‖a‖ ‖b‖), which for unit vectors is just the dot product. You embed every chunk once at indexing time and every query at query time, with the same model. This is called dense retrieval, and a model that encodes query and chunk separately like this is a bi-encoder.

Predict first: for "how do I cancel my membership", which page does dense retrieval rank first: "Ending your Ride+ subscription…" or "Cancel an order…"?

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")   # English, 384-d
doc_vecs = model.encode([docs[i] for i in ids], normalize_embeddings=True)
print(doc_vecs.shape)

def search_dense(query, k=3):
    q = model.encode([query], normalize_embeddings=True)[0]
    scores = doc_vecs @ q                  # unit vectors, so dot product = cosine
    best = scores.argsort()[::-1][:k]
    return [(ids[i], float(scores[i])) for i in best]

for q in ["is my battery still under guarantee after 18 months", "E-217", "how do I cancel my membership"]:
    print(f"{q!r:55} ->", [(d, round(s, 3)) for d, s in search_dense(q)])

# (12, 384)
# 'is my battery still under guarantee after 18 months'   -> [('warranty', 0.43), ('storage', 0.295), ('err-217', 0.288)]
# 'E-217'                                                 -> [('err-227', 0.33), ('err-217', 0.326), ('err-712', 0.23)]
# 'how do I cancel my membership'                         -> [('cancel-order', 0.508), ('end-plan', 0.439), ('err-227', 0.169)]

What meaning-based retrieval adds. Paraphrase and synonyms without anyone writing a list: guarantee found warranty at 0.43 while keyword search never returned it. With a multilingual model, a question in one language can find a page in another (you will try that below).

What it costs. Every row here is a real line item:

Why do the two error codes collide? To MiniLM the query E-217 is three word pieces, e, - and 217. The E-227 page contains the same first two pieces and a different three-digit number, and nothing in the query mentions a hot battery or a low voltage, the content that actually separates the two pages. With that little to go on, look-alike codes land within a hair of each other: 0.330 against 0.326. Negation, numbers and "similar is not true" get a full treatment in Semantic Search Principles; how models are trained and chosen is in Vector Embeddings, and the metrics in Cosine Similarity & Distance Metrics.

⚠️ No universal threshold. The correct warranty page scored 0.43; run search_dense("can I ride when it is raining") and the correct rain page scores 0.741. A rule such as "cosine above 0.5 means relevant" would have thrown away the warranty answer. Cut-offs are model-specific and must be tuned on labeled data.

Your turn: multilingual customers. Forty percent of customers now write in German or Turkish; the pages stay in English. Predict what BM25 and MiniLM return for "Wie lange dauert es, bis der Akku voll ist?" (how long until the battery is full) and "Aboneliğimi nasıl iptal ederim?" (how do I cancel my subscription). What would you change?

multi = SentenceTransformer("intfloat/multilingual-e5-small")   # ~100 languages, also 384-d
multi_vecs = multi.encode(["passage: " + docs[i] for i in ids], normalize_embeddings=True)

for q in ["Wie lange dauert es, bis der Akku voll ist?", "Aboneliğimi nasıl iptal ederim?"]:
    en = doc_vecs @ model.encode([q], normalize_embeddings=True)[0]
    ml = multi_vecs @ multi.encode(["query: " + q], normalize_embeddings=True)[0]
    print(q)
    print("  bm25       ", search_bm25(q))
    print("  MiniLM     ", [(ids[i], round(float(en[i]), 3)) for i in en.argsort()[::-1][:2]])
    print("  multilingual", [(ids[i], round(float(ml[i]), 3)) for i in ml.argsort()[::-1][:2]])

# Wie lange dauert es, bis der Akku voll ist?
#   bm25        []
#   MiniLM      [('err-217', 0.055), ('bt-4471', 0.034)]
#   multilingual [('charge-time', 0.832), ('warranty', 0.816)]
# Aboneliğimi nasıl iptal ederim?
#   bm25        []
#   MiniLM      [('cancel-order', 0.054), ('err-271', 0.049)]
#   multilingual [('end-plan', 0.769), ('err-227', 0.762)]

The scoreboard: ten labeled queries, three retrievers

Time to stop arguing from single queries. Here is a labeled set of ten: half descriptive, half identifier lookups. You will score BM25, dense, and a hybrid that runs both and fuses their rankings.

The fusion is Reciprocal Rank Fusion (Cormack, Clarke & Büttcher, SIGIR 2009): each chunk gets RRF(d) = Σ 1 / (k + rank) summed over the lists it appears in, with 1-based ranks and k = 60. A chunk missing from a list gets nothing from that list. RRF uses ranks, not scores, because the scales disagree: BM25 scores in this corpus run up to 6.01 while cosines stay below 0.75. Weighted fusion and tuning belong to Hybrid Retrieval Systems.

Predict first: which retriever gets the highest recall@1? Does the hybrid beat both?

labeled = {   # query -> the one chunk that answers it
    "is my battery still under guarantee after 18 months": "warranty",
    "how long until the battery is full":                  "charge-time",
    "how do I cancel my membership":                       "end-plan",
    "can I ride when it is raining":                       "rain",
    "which battery fits a 2023 City bike":                 "bt-4417",
    "E-217":                                               "err-217",
    "BT-4417":                                             "bt-4417",
    "what does error E-271 mean":                          "err-271",
    "my display says E-712":                               "err-712",
    "the bike shows E-227 what do I do":                   "err-227",
}

def rrf(rankings, k=60):
    fused = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):   # ranks are 1-based
            fused[doc_id] = fused.get(doc_id, 0.0) + 1 / (k + rank)
    return sorted(fused, key=lambda d: -fused[d])

def bm25_ids(q):   return [d for d, _ in search_bm25(q, k=len(ids))]
def dense_ids(q):  return [d for d, _ in search_dense(q, k=len(ids))]
def hybrid_ids(q): return rrf([bm25_ids(q), dense_ids(q)])

def evaluate(search):
    hit1 = hit3 = rr = 0.0
    for query, relevant in labeled.items():
        ranking = search(query)
        if relevant in ranking:              # a miss contributes 0 to every metric
            rank = ranking.index(relevant) + 1
            hit1 += rank <= 1
            hit3 += rank <= 3
            rr += 1 / rank
    n = len(labeled)
    return hit1 / n, hit3 / n, rr / n

for name, search in [("bm25", bm25_ids), ("dense", dense_ids), ("hybrid", hybrid_ids)]:
    r1, r3, mrr = evaluate(search)
    print(f"{name:7} recall@1 {r1:.1f}   recall@3 {r3:.1f}   MRR {mrr:.2f}")

# bm25    recall@1 0.5   recall@3 0.6   MRR 0.59
# dense   recall@1 0.8   recall@3 1.0   MRR 0.90
# hybrid  recall@1 0.6   recall@3 0.8   MRR 0.73

With twelve chunks, dense retrieval returns its whole ranking and BM25 every chunk with a positive score (a chunk whose only shared word is battery scores 0, so it is not returned). In a real system each would return its top 50 to 200.

Fusing two lists that disagree

Dense wins, and the hybrid lands between the two. Fusion fixed E-217 (dense rank 2, fused rank 1) but broke three descriptive queries. Trace the warranty query:

RRF rewards agreement. A chunk that both lists rank moderately beats a chunk that only one list ranks first. And RRF ignores scores: BM25's first place here is a weak match on the word is (1.09), yet it earns the same 1/61 as a perfect match would, and those equal votes drag the right answer down to fourth.

So hybrid is not automatically better. It helps when each retriever's failures are the other's strengths and neither fills the top of its list with weak matches. This toy is also kind to dense retrieval on identifiers: there are only four error codes and two battery packs, and even so dense lost E-217 by 0.330 to 0.326 and won BT-4417 by 0.367 to 0.299. A real catalog with thousands of near-identical codes makes those coin flips much worse. Do not choose an architecture from twelve chunks; choose it from your own labeled set, sliced by query type. The usual moves when plain fusion hurts are weighting the lists (tuned on labels), routing identifier-shaped queries to lexical search, or reranking the fused pool, which is next.

A second opinion: rerank the pool

A bi-encoder squeezes each text into one vector before it has seen the query, so its similarity is a coarse guess at relevance. A cross-encoder reads the query and one chunk together and outputs a relevance score for that pair. It judges more precisely but must run once per pair at query time, so you give it a small candidate pool instead of the whole corpus. The pattern is called retrieve-then-rerank.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def hybrid_rerank_ids(query, pool=5):
    candidates = hybrid_ids(query)[:pool]
    scores = reranker.predict([(query, docs[d]) for d in candidates])
    return [d for _, d in sorted(zip(scores, candidates), reverse=True)]

r1, r3, mrr = evaluate(hybrid_rerank_ids)
print(f"rerank  recall@1 {r1:.1f}   recall@3 {r3:.1f}   MRR {mrr:.2f}")

q = "is my battery still under guarantee after 18 months"
pool = hybrid_ids(q)[:5]
print(pool)
print([(d, round(float(s), 1)) for d, s in zip(pool, reranker.predict([(q, docs[d]) for d in pool]))])

# rerank  recall@1 0.9   recall@3 1.0   MRR 0.95
# ['err-227', 'err-217', 'rain', 'warranty', 'storage']
# [('err-227', -11.2), ('err-217', -11.1), ('rain', -11.3), ('warranty', -8.9), ('storage', -10.7)]

Every relevant chunk was somewhere in the fused top 5, so the reranker only had to put them in order, and it lifted recall@1 from 0.6 to 0.9. Two things to notice:

• The scores are logits, not probabilities. The winning chunk scored −8.9. Negative does not mean irrelevant here. Another reranker, or the same kind of model in another library, may squash its scores into 0 to 1, and that is still not a calibrated probability. Use the scores to order the pool, and tune any cut-off per model on labeled data.
• A reranker reorders; it cannot find. If the answer had been sixth after fusion, a pool of 5 would never show it to the reranker. Missing answers are a recall problem, fixed upstream with a bigger pool or better first-stage retrieval. Model choices, pool sizes and costs are in Reranking Models.

The query nobody got right

"How do I cancel my membership" fails everywhere. BM25 returns "Cancel an order" (and the rain page, which matched on do) and never returns the subscription page. Dense ranks it second, the fusion third, and the reranker keeps "Cancel an order" on top, −2.2 against −9.0. Every stage agrees, because the right page never says cancel or membership.

Your turn: which fix do you try first?

The path of a query

You now have every piece. Here is how they fit in a production system:

OFFLINE, whenever content changes
  documents -> parse -> chunk --+--> embed with model M -----> vector index  --+  both store text
                                +--> tokenize (analyzer) ---> lexical index --+  and metadata

ONLINE, per query
                                 +-> lexical search, top 100, filter inside -+
  query -> understand -----------+                                           +-> fuse -> rerank -> context set -> generate
           normalize, rewrite,   +-> dense search, top 100, filter inside ---+   RRF    100 -> 5   5 chunks      (RAG only)
           extract filters           (query embedded with the same model M)

• Understand. Normalize the text, spot identifiers, pull out constraints ("a 2023 City bike" becomes year = 2023, line = City), and turn a follow-up such as "what about the Trail one?" into a standalone query. Owned by Query Understanding & Intent; questions that need several searches go to Query Decomposition.
• Retrieve. Lexical and dense search run in parallel, each returning a candidate list. The vector side is an exact scan or an ANN index: Vector Database Architecture, ANN Algorithms, Vector DB Selection, and a hands-on engine in Qdrant.
• Filter, inside each search (below).
• Fuse the two lists (up to 200 ids) into one candidate pool, here the top 100: Hybrid Retrieval Systems.
• Rerank the pool and keep the context set: Reranking Models.
• Generate, only in retrieval-augmented generation (RAG): an LLM answers from the context set with citations and says so when the answer is not there. Pure search stops after reranking and shows the list. See RAG Architecture & Implementation, Classic RAG Pipeline and Context Augmentation.

The offline half (parsing, chunking, embedding at scale, keeping the index fresh) has its own branch, starting at Data Pipeline & Indexing.

Why filters go inside the search. Suppose a shared index returns the top 10 chunks and the application then drops the ones the user may not see. If the user's tenant owns 2% of the chunks, and ownership has nothing to do with the ranking, the expected number of survivors is 10 × 0.02 = 0.2, and only 1 − 0.98¹⁰ ≈ 18% of queries return anything at all. Real users may search for things their own tenant wrote about, which helps, but nothing guarantees a single survivor. Forget that application-side step once and you leak other tenants' text instead. So the filter goes into the retrieval call itself, enforced on the server, deny by default: Metadata & Filtering.

Trace one query

The warranty question through the pipeline, with the numbers your code printed:

Which stage broke?

When an answer is wrong, find the first stage where the right chunk went missing:

💡 Read the context before blaming the model. A retrieval miss and a generation mistake look identical to the user: a fluent, confident, wrong answer. Logging the context set for each answer is what lets you tell them apart. RAG reduces unsupported answers when retrieval finds the evidence; the model can still ignore or misread it.

The latency budget

Every knob above is set by a time budget. Say the help center grows to 400,000 chunks and becomes a chat assistant. Product wants the answer to start appearing within 2 seconds at p95, and you give everything before the LLM call 300 ms at p95.

The rerank budget buys 225 ms ÷ time per pair candidates. On the laptop CPU used for this lesson (an Apple M2 Max, lightly loaded), the MiniLM cross-encoder measured about 0.8 ms per short pair (31 tokens) and about 7 ms per 278-token pair. Those are throughput figures, 50 pairs scored in batches of 32 and divided by 50, with the model loaded with device="cpu"; without that argument, sentence-transformers uses a GPU or Apple Silicon's accelerator when one is present. While other jobs loaded the machine, the same runs took five to seven times longer, which is exactly what a p95 has to absorb. At 7 ms, 225 ms buys 32 candidates, so the pool of 100 (about 700 ms) does not fit on this CPU. The plan therefore assumes a GPU that brings a pair down to 1.5 ms (an assumption to check on your hardware): 100 candidates then cost 150 ms and fit, with room for up to 150. Pair length matters as much as pool size: nine times the tokens cost about nine times the time here. Measure with your real chunk length and your actual reranker before you trust any of these numbers.

Two rules of thumb for the arithmetic. For one request, parallel stages cost the maximum and sequential stages the sum. Adding up per-stage p95s is a reasonable first plan, but it is not a guarantee: rare stalls in several stages can push the end-to-end p95 above the sum, and two parallel stages together cost more than the slower one's p95, a little more with everyday spread and much more when both have rare stalls. Measure the end-to-end p95 as well.

Where exact search stops fitting. A flat scan of float32 vectors reads n × dim × 4 bytes per query. Time it:

import time
import numpy as np

rng = np.random.default_rng(0)
for n in [10_000, 100_000, 1_000_000]:
    X = rng.standard_normal((n, 384), dtype=np.float32)
    X /= np.linalg.norm(X, axis=1, keepdims=True)       # unit vectors, like the model's
    q = X[0].copy()
    timings = []
    for _ in range(20):
        t0 = time.perf_counter()
        top10 = np.argpartition(-(X @ q), 10)[:10]       # exact: scores every vector
        timings.append(time.perf_counter() - t0)
    print(f"n = {n:>9,}   {X.nbytes / 1e9:.3f} GB   median {np.median(timings) * 1e3:6.1f} ms")

# n =    10,000   0.015 GB   median    0.2 ms
# n =   100,000   0.154 GB   median    2.9 ms
# n = 1,000,000   1.536 GB   median   36.0 ms

Your numbers will differ; the shape will not. Time grows roughly linearly with the number of vectors. At ten thousand chunks an exact scan costs well under a millisecond, so a prototype needs no vector database. At a million, the scan alone takes 36 ms of a 40 ms search budget, and an approximate index that visits a small fraction of the vectors earns its keep. Vector Database Architecture works out where that line falls for your data.

Your turn: the budget before the LLM call drops to 150 ms, with the same plan for the other stages. What do you cut, and how do you know it was the right cut?

Why not paste everything into the prompt?

Context windows are huge now. Meta announced Llama 4 Scout in April 2025 with a 10-million-token context window. So why retrieve at all?

Sometimes you shouldn't. The twelve help pages above are about 250 tokens in total. Put them all in the prompt and skip retrieval.

Scale it up and the arithmetic turns. The 400,000-chunk help center at about 300 tokens a chunk is 120 million tokens, twelve times that 10-million-token window. A middle case, a 2,000-page manual set of about 1.2 million tokens, fits in some windows, but every query then pays for 1.2 million input tokens instead of about 2,000 (five 300-token chunks plus 500 tokens of instructions and question), which is about 600 times more per query at any per-token price. The time before the first token also grows with prompt length. What moves this trade-off:

• Prompt caching. Some providers discount a repeated prompt prefix. For example, Anthropic's prompt caching bills cache reads at 0.1 times the base input price for most of its models (checked September 2026). A small, stable corpus used for many queries gets much cheaper.
• Access control. One shared prompt shows everything in it to everyone. Per-user permissions need per-user context, which is retrieval with a filter.
• Using what is in the window. Liu et al. (2023) found that models used information in the middle of long inputs less reliably than at the start or end; the size of the effect varies by model and task.

Long context tends to win when the corpus is small and stable, when the question needs whole documents ("compare these two contracts"), or when there are few queries. Retrieval tends to win when the corpus is large, changes often, or has permissions. RAG Architecture & Implementation makes this decision in depth, together with fine-tuning.

What 2026 practice adds, and where each piece lives

The stages themselves are not new. What changed by 2026 is what runs inside them:

Retrieval without embeddings, and retrieval over knowledge graphs, have their own lessons: Vectorless RAG and Graph Rag.

Traps that don't throw errors

The dangerous failures in search return plausible results instead of errors. Each has a fix, and each fix has a price:

Final round: no label on the problem

Real incidents don't say which stage broke. For each one, name the stage, the first move and how you would confirm it.

1. A hardware store's dense-only search: "M8 x 40 hex bolt" returns "M8 x 45 hex bolt" and "M6 x 40 hex bolt" above the exact item.

2. After re-chunking the manuals into longer 600-word-piece chunks, questions about the last steps of long troubleshooting guides stopped finding the right chunk; questions about the first steps still work.

3. A reranker was added. Recall@5 on 50 labeled queries rose by three queries, and p95 latency went from 110 ms to 1.4 s.

4. The support bot says returns are accepted within 14 days. The logged context set holds two chunks: the 2023 policy (14 days) and the 2025 policy (30 days).

Cheat sheet: symptom → move → cost

Before moving on

Explain the path of one query aloud, from a blank page: what the understand step does to it, what each retriever returns and why, how fusion and reranking change the order, where the filter runs, and what the latency budget allows at each stage.

Then answer the question every later lesson will ask: how would you measure it on your own data? Take 50 real queries from your search logs, label the chunk or chunks that answer each, and slice them by query type. Score BM25, dense and hybrid on recall@10 and MRR; record p95 latency per stage and the cost per 1,000 queries. Every change you make from here on, whether a new model, chunk size, reranker or index setting, gets rerun against that set before it ships. Evaluation & Quality Metrics turns this habit into a full practice.

Next: Semantic Search Principles for what embedding similarity does and does not capture, Vector Database Architecture for storing and searching vectors at scale, Hybrid Retrieval Systems for combining lexical and dense retrieval properly, and Query Understanding & Intent for fixing the query before you search.