Sparse vs Dense Retrieval

The part number that vanished

You run search for a pump maker's support portal. The parts catalog covers 400 pump models, XK-4400 to XK-4799, with five spare parts each: 2,000 short chunks. The help center holds troubleshooting articles. Last quarter the team swapped keyword search for embeddings (all-MiniLM-L6-v2, cosine similarity, exact search), because "semantic search understands what people mean".

On Monday a technician types seal kit XK-4471 and orders the wrong kit. Here is what the embedding search returned (a real run; the code is in Move 3):

Twenty-two chunks with the wrong number sit above the right one, all within 0.008 of each other. The old keyword engine, BM25, puts the XK-4471 seal kit first with a score of 11.35, well ahead of the next chunk (the XK-4471 impeller, 8.44). Over 100 random part lookups, the embeddings put the right chunk first 34 times. BM25 did it 100 times.

So you switch back. On Tuesday a customer types into the help center: loud scraping sound when I switch it on. BM25's top hit is the XK-4471 impeller page, which matched on "when" and "it". The article that answers the question, "Bearing wear: if the motor rattles or grinds at startup, the bearings are worn", shares no word with the query and scores exactly 0. The embedding model ranks it first.

Neither system is broken. Each ranks with what its representation can see. Sparse retrieval sees which tokens a text contains and how much each is worth. Dense retrieval sees one learned vector per text. This lesson opens both up, so that on a system you have not seen you can predict which queries each will miss, fix what is fixable, and measure the rest.

You should already know what an embedding is and how cosine similarity ranks results (Vector Embeddings, Cosine Similarity & Distance Metrics). Merging a sparse and a dense result list into one ranking is the job of Hybrid Retrieval Systems; this lesson builds the two parts it fuses. The code is Python 3.12 with numpy, sentence-transformers, fastembed and rank_bm25; models download from Hugging Face on first use.

Move 1: Index the words, not the documents

Symptom: you picture search as comparing the query with every document. That works at 2,000 chunks, not at 50 million.

A keyword engine turns the problem around. At index time it builds an inverted index: for every term, a postings list of the chunks that contain it and how often. At query time it reads only the postings lists of the query's terms.

import re
from collections import Counter, defaultdict

def tokenize(text):
    return re.findall(r"\w+", text.lower())

chunks = {
    "d04": "Bearing wear: if the motor rattles or grinds at startup, the bearings are worn.",
    "d05": "Leaking shaft seal: water dripping under the pump housing means the shaft seal has failed.",
    "d11": "Error code E07: the motor overheated. Let the pump cool for 30 minutes.",
    "d13": "Frost protection: drain the pump before freezing temperatures.",
}

index = defaultdict(dict)                 # term -> {chunk id: term frequency}
for chunk_id, text in chunks.items():
    for term, tf in Counter(tokenize(text)).items():
        index[term][chunk_id] = tf

for term in ["the", "pump", "motor", "seal", "cavitation"]:
    print(f"{term:10s} -> {index.get(term, {})}")
print(len(index), "distinct terms")

# the        -> {'d04': 2, 'd05': 2, 'd11': 2, 'd13': 1}
# pump       -> {'d05': 1, 'd11': 1, 'd13': 1}
# motor      -> {'d04': 1, 'd11': 1}
# seal       -> {'d05': 2}
# cavitation -> {}
# 39 distinct terms

Predict: for the query motor seal, which chunks does the engine look at, and which one does it never touch?

The same index, drawn as vectors

Picture a matrix with one row per chunk and one column per term in the vocabulary. Each chunk's row is almost all zeros: d05 has non-zero entries in 12 of the 39 columns here, and a real vocabulary has hundreds of thousands of columns. That is why these are called sparse vectors. The inverted index is that matrix stored column by column, with the zeros left out.

Scoring a chunk is then a dot product between a sparse query vector and a sparse chunk vector: multiply the weights of the terms both contain and add them up. Every sparse method in this lesson fits that picture; they differ in which entries are non-zero and who sets the weights. Dense is the contrast:

That is also why vector databases score sparse vectors with a dot product (Qdrant supports only dot product for them).

What it costs. A query's work grows with the total length of the postings lists it reads. Rare terms have short lists; "the" has a list as long as the corpus. Engines skip blocks of postings that cannot reach the current top k (Lucene added block-max WAND in version 8.0, and Elastic reported large speed-ups). Adding a chunk means appending to a few lists, and no model runs anywhere.

Measure it: log p95 latency split by number of query terms, and look at which queries are slowest.

Move 2: Weight the words, then read BM25 term by term

Symptom: a chunk that obviously matches ranks below one that doesn't, and nobody on the team can say why.

Counting matches is not enough. "The" matches everything, a chunk that repeats a word ten times is not ten times as relevant, and a long manual mentions every word once. BM25 answers each problem with one factor.

Rare words are stronger evidence

A term that occurs in few chunks tells you more than one that occurs everywhere. Inverse document frequency (IDF) turns that into a weight. Lucene (and so Elasticsearch, OpenSearch and Solr) uses:

idf(t) = ln(1 + (N − df + 0.5) / (df + 0.5))
N  = number of chunks in the index
df = number of chunks that contain t

In the 14-chunk help center used below: "the" (in all 14) gets 0.034, "pump" (8) gets 0.568, "seal" (5) gets 1.003, and "leaking" (1) gets 2.303. At a million chunks, a term in one chunk gets 13.41, and a term in 900,000 chunks gets 0.105. The 1 + inside the logarithm keeps the value above zero. The older Robertson–Spärck Jones form, ln((N − df + 0.5) / (df + 0.5)), goes negative for terms in more than half the chunks, which will bite you in a moment.

Repetition helps, with diminishing returns

BM25 does not use the raw count tf. For a chunk of average length it uses tf·(k1 + 1) / (tf + k1), which starts at 1 and can never pass k1 + 1:

import math

k1, b = 1.2, 0.75
print("tf   BM25 k1=1.2   log(1+tf)")
for tf in [1, 2, 3, 5, 10, 20]:
    print(f"{tf:2d}   {tf * (k1 + 1) / (tf + k1):11.3f}   {math.log(1 + tf):9.3f}")

print("dl/avgdl   tf=2, b=0.75   tf=2, b=0")
for ratio in [0.25, 0.5, 1, 2, 4]:
    norm = 1 - b + b * ratio
    print(f"{ratio:8.2f}   {2 * (k1 + 1) / (2 + k1 * norm):12.3f}   {2 * (k1 + 1) / (2 + k1):9.3f}")

# tf   BM25 k1=1.2   log(1+tf)
#  1         1.000       0.693
#  2         1.375       1.099
#  3         1.571       1.386
#  5         1.774       1.792
# 10         1.964       2.398
# 20         2.075       3.045
# dl/avgdl   tf=2, b=0.75   tf=2, b=0
#     0.25          1.743       1.375
#     0.50          1.600       1.375
#     1.00          1.375       1.375
#     2.00          1.073       1.375
#     4.00          0.746       1.375

With k1 = 1.2, twenty occurrences are worth 2.075, barely twice one occurrence, and the ceiling is 2.2. A log-scaled TF-IDF keeps climbing (3.045 at 20), so keyword stuffing keeps paying there. A smaller k1 saturates sooner; a larger one lets repetition count for longer.

Long chunks mention everything once

The second table is length normalization. BM25 replaces k1 with k1·(1 − b + b·dl/avgdl), where dl is the chunk's length in tokens and avgdl is the average. Two occurrences in a chunk a quarter of the average length are worth 1.743. The same two in a chunk four times the average are worth 0.746. With b = 0 length is ignored; with b = 1 it is fully normalized.

The whole formula

score(q, d) = Σ over query terms t that occur in d:
              idf(t) · tf·(k1 + 1) / (tf + k1·(1 − b + b·dl/avgdl))

Defaults in Lucene, Elasticsearch and OpenSearch: k1 = 1.2, b = 0.75

BM25 goes back to the Okapi system at City University London; Lucene's BM25Similarity documentation cites Robertson et al., "Okapi at TREC-3" (1994). Lucene made it the default in version 6, and Elasticsearch followed in 5.0 (2016); before that the default was classic TF-IDF, which you will meet after the first exercise.

Here it is on the help-center corpus used for the rest of this lesson. Save the first block as helpcenter.py and the second as bm25.py. Chunk d09 is an installation manual of 256 tokens (as bm25.py counts them), kept as one chunk on purpose.

# helpcenter.py: the pump maker's help-center articles, one chunk each
MANUAL = " ".join([
    "VX-300 installation manual.",
    "Mount the pump with the motor shaft horizontal and leave 30 cm of free space around the motor for cooling air.",
    "Flush the pipework before connecting the pump so that welding beads and sand cannot reach the impeller.",
    "Fit isolation valves on both sides of the pump so it can be removed without draining the whole system.",
    "Check that the arrow on the housing points in the direction of flow.",
    "Connect the supply cable to the terminal box with a cable gland rated for the ambient temperature.",
    "The pump must be earthed, and a residual current device must be fitted where local regulations require one.",
    "Fill the system and vent the pump through the venting screw until water flows without air bubbles.",
    "Never run the pump dry, not even for a few seconds, because the mechanical seal is lubricated by the pumped liquid.",
    "Set the operating mode on the control panel. Auto-adapt mode is the factory setting and suits most heating systems.",
    "In constant pressure mode the pump keeps the differential pressure fixed regardless of the flow.",
    "In constant curve mode the pump runs at the fixed speed selected on the panel.",
    "Record the settings in the service log and hand this manual to the operator after commissioning.",
    "Inspect the installation after one week of operation and again after one month.",
    "During each inspection, check the terminal box for moisture and the pipe connections for drips.",
    "Flange connections: tighten the flange bolts crosswise to 25 Nm with a calibrated torque wrench.",
])
DOCS = {
    "d01": "XK-4471 seal kit: shaft seal and O-rings for the XK-4471 circulation pump. Fits housings made after 2019.",
    "d02": "XK-4417 seal kit: shaft seal and O-rings for the XK-4417 circulation pump. Fits all housings.",
    "d03": "XK-4471 impeller: bronze impeller for the XK-4471 circulation pump. Replace it when the flow drops below 60 percent of the rated flow.",
    "d04": "Bearing wear: if the motor rattles or grinds at startup, the bearings are worn. Replace both bearings before the shaft is damaged.",
    "d05": "Leaking shaft seal: water dripping under the pump housing means the shaft seal has failed. Order the seal kit for your model.",
    "d06": "Cavitation: a crackling noise like gravel in the pipes means the inlet pressure is too low and vapour bubbles collapse on the impeller.",
    "d07": "VX-300 controller: the VX-300 requires the external CT-9 controller. Connect the controller before the first start.",
    "d08": "VX-100 controller: the VX-100 has a built-in controller, so no separate control unit is needed.",
    "d09": MANUAL,
    "d10": "Warranty: pumps are covered for 24 months from the date of purchase. Seal kits and impellers are wear parts and are not covered.",
    "d11": "Error code E07: the motor overheated. Let the pump cool for 30 minutes and clean the ventilation slots.",
    "d12": "Error code E17: the pressure sensor failed. Replace the sensor board.",
    "d13": "Frost protection: drain the pump before freezing temperatures so the housing does not crack.",
    "d14": "Flange gasket: replace the flange gasket every time the pump is removed from the pipework.",
}

# bm25.py: Okapi BM25 with Lucene's IDF (k1 = 1.2, b = 0.75 are Lucene's defaults)
import math
import re
from collections import Counter

def tokenize(text):
    return re.findall(r"\w+", text.lower())      # lowercase, split on non-word chars

class BM25:
    def __init__(self, docs, k1=1.2, b=0.75, analyzer=tokenize):
        self.k1, self.b, self.analyze = k1, b, analyzer
        self.tf = {i: Counter(analyzer(text)) for i, text in docs.items()}
        self.dl = {i: sum(c.values()) for i, c in self.tf.items()}
        self.N = len(docs)
        self.avgdl = sum(self.dl.values()) / self.N
        self.df = Counter(term for c in self.tf.values() for term in c)

    def idf(self, term):
        n = self.df.get(term, 0)
        return math.log(1 + (self.N - n + 0.5) / (n + 0.5))

    def term_score(self, term, doc_id):
        f = self.tf[doc_id].get(term, 0)
        norm = 1 - self.b + self.b * self.dl[doc_id] / self.avgdl
        return self.idf(term) * f * (self.k1 + 1) / (f + self.k1 * norm)

    def scores(self, query):
        # A real engine walks the postings of the query terms; a loop is fine at this size.
        terms = set(self.analyze(query))
        return {i: sum(self.term_score(t, i) for t in terms) for i in self.tf}

Predict first: for the query shaft seal leaking, which chunk ranks first? And d01 and d02 both contain "shaft" once and "seal" twice. Which of them ranks higher, and why?

from helpcenter import DOCS
from bm25 import BM25, tokenize

bm = BM25(DOCS)
query = "shaft seal leaking"
print(f"N={bm.N}  avgdl={bm.avgdl:.2f}")
for t in tokenize(query):
    print(f"idf({t}) = {bm.idf(t):.3f}   (df={bm.df[t]})")

scores = bm.scores(query)
for doc_id in sorted(scores, key=scores.get, reverse=True)[:4] + ["d09"]:
    parts = "  ".join(f"{t}: tf={bm.tf[doc_id][t]} -> {bm.term_score(t, doc_id):.3f}"
                      for t in tokenize(query))
    print(f"{doc_id} dl={bm.dl[doc_id]:3d}  {parts}  total={scores[doc_id]:.3f}")

Why it works. Each term's contribution is capped at idf·(k1 + 1) and shrinks as the chunk gets longer, so a chunk climbs by matching more distinct rare terms, not by repeating one. It is also an explanation you can show an auditor: every score is a sum of per-term contributions, and engines print them (Elasticsearch's explain API gives the breakdown above).

Your turn. Query sensor. Chunk d12 ("Error code E17: the pressure sensor failed. Replace the sensor board.") has 11 tokens and contains "sensor" twice, and "sensor" occurs in no other chunk (so its idf is 2.303, like "leaking"). Compute d12's score. Then a writer "optimizes" the article by pasting both sentences twice, so tf = 4 and dl = 22. Keeping avgdl at 35.93, by what percentage does the score rise?

What BM25 replaced: TF-IDF

TF-IDF also multiplies a term's count by its IDF, but its tf part has no ceiling and there is no b: it handles length by normalizing. Two forms you will meet:

scikit-learn TfidfVectorizer, defaults:
  weight(t, d) = tf · idf(t)      idf(t) = ln((1 + N) / (1 + df)) + 1
  each chunk's vector of weights is scaled to length 1; score = cosine with the query's vector
  (sublinear_tf=True uses 1 + ln(tf) in place of tf: a log scaling like the log(1+tf) column above)

Lucene ClassicSimilarity (named DefaultSimilarity until BM25 replaced it in Lucene 6):
  score(q, d) = Σ over query terms t in d:  √tf · idf(t)² · 1/√dl      idf(t) = ln((N + 1) / (df + 1)) + 1

For "sensor", which only d12 contains, scikit-learn's idf is ln(15/2) + 1 = 3.015. Now replay the exercise's edit, plus a plain keyword-stuffing one, under both scorers:

from sklearn.feature_extraction.text import TfidfVectorizer
from helpcenter import DOCS
from bm25 import BM25, tokenize

d12 = DOCS["d12"]
for label, text in [("original", d12),
                    ("article pasted twice", d12 + " " + d12),
                    ("'sensor' added 4 times", d12 + " sensor sensor sensor sensor")]:
    docs = {**DOCS, "d12": text}
    tfidf = TfidfVectorizer(analyzer=tokenize)    # defaults: raw tf, smoothed idf, rows scaled to length 1
    X = tfidf.fit_transform(docs.values())
    cosine = (X @ tfidf.transform(["sensor"]).T).toarray().ravel()[list(docs).index("d12")]
    print(f"{label:22s}  TF-IDF cosine {cosine:.3f}   BM25 {BM25(docs).scores('sensor')['d12']:.3f}")

# original                TF-IDF cosine 0.640   BM25 3.934
# article pasted twice    TF-IDF cosine 0.640   BM25 4.187
# 'sensor' added 4 times  TF-IDF cosine 0.928   BM25 4.555

Pasting the whole article twice changes nothing under TF-IDF: every count doubles, and scaling the vector to length 1 cancels it (in ClassicSimilarity, √tf / √dl cancels the same way). Stuffing the one keyword is another story: TF-IDF rewards it with 45% more, BM25 with 16%, because k1 caps what repetition can earn. That is the whole difference: TF-IDF normalizes length away and lets repetition keep paying; BM25 caps repetition and lets you choose, with b, how much length counts.

Why your numbers don't match another library

• Lucene 8.0 removed the constant (k1 + 1) factor (LUCENE-8563), because multiplying every score by 2.2 cannot change a ranking. Elasticsearch still reports scores that include it, while MongoDB Atlas Search, also built on Lucene, returns the same order with scores 2.2 times lower, as Franck Pachot showed.
• rank_bm25's BM25Okapi defaults to k1 = 1.5, b = 0.75, epsilon = 0.25. It uses the older IDF ln((N − df + 0.5)/(df + 0.5)) and replaces negative values with epsilon × the average IDF.

What does this print?

from rank_bm25 import BM25Okapi

bm = BM25Okapi([["pump", "seal", "leak"], ["pump", "impeller"]])
print(bm.get_scores(["seal"]), bm.get_scores(["pump"]))

Tuning k1 and b, and when not to bother

• b only matters when lengths differ. If every chunk is about the same length, dl/avgdl ≈ 1 and b changes almost nothing. RAG pipelines usually index chunks of similar size, so b matters less there than when you mix titles with bodies, or short answers with whole manuals.
• k1 sets how much repetition counts. Lower it for short fields such as titles and product names, where one mention is the signal; raise it for long, focused articles. Because length enters the formula only through k1·(1 − b + b·dl/avgdl), a smaller k1 also softens the length penalty.
• Scores have no fixed range. A three-term query scores higher than a one-term query, and one rare term can add up to idf·(k1 + 1), which is 29.5 at a million chunks. A threshold such as "BM25 above 10 means relevant" does not carry across queries, and raw BM25 scores cannot be added to cosine scores without normalization (Hybrid Retrieval Systems handles that).

Your turn. The manual (d09) is the only chunk that answers flange bolt torque, yet BM25 ranks the short "Flange gasket" article (d14) first: 2.946 against 1.562. Which parameter could you change to put the manual first? What does that change cost you on other queries? And what is the better fix?

Measure it: tune k1 and b with a small grid (say k1 from 0.6 to 2.0, b from 0.3 to 0.9) on a labeled query set, and keep the defaults unless the gain holds on a second sample of queries. Fix tokenization first; it is usually the bigger lever.

Move 3: Tokens decide what "exact" means

Symptom: a code that is definitely in the index doesn't match, a typo returns nothing, or "leak" misses "leaking".

BM25 never sees your strings. It sees the tokens an analyzer produces: a tokenizer plus filters such as lowercasing, stopword removal and stemming. Elasticsearch's standard tokenizer turns "Brown-Foxes" into Brown and Foxes; the tokenizer in bm25.py likewise turns XK-4471 into xk and 4471. So BM25 did not win the opening lookup by string matching. It won because 4471 is a rare token (5 of 2,000 catalog chunks) with a high IDF.

Predict: a technician types seal kit XK4471, without the hyphen. What does BM25 return?

Here is the catalog experiment from the opening: 100 random part lookups, in both formats, against MiniLM embeddings, plain BM25, and BM25 with an identifier-aware analyzer that also indexes each part number without its separator.

import random
import re
from sentence_transformers import SentenceTransformer
from bm25 import BM25, tokenize

PARTS = {"seal kit": "shaft seal and O-rings", "impeller": "bronze impeller",
         "bearing set": "two motor bearings", "capacitor": "motor start capacitor",
         "sensor board": "pressure sensor board"}
catalog = {f"XK-{n}/{part}": f"XK-{n} {part}: {what} for the XK-{n} circulation pump."
           for n in range(4400, 4800) for part, what in PARTS.items()}      # 2,000 chunks

def id_aware(text):
    """Also index each part number without its separator: XK-4471 -> xk, 4471, xk4471."""
    joined = [m.replace("-", "").lower() for m in re.findall(r"\b[a-z]{2}-?\d{4}\b", text, re.I)]
    return tokenize(text) + joined

def rank_of(scores, target):
    """1-based rank of the target; ties count against the retriever."""
    return 1 + sum(s >= scores[target] for i, s in scores.items() if i != target)

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ids = list(catalog)
vecs = model.encode(list(catalog.values()), normalize_embeddings=True)

def dense(q):
    return dict(zip(ids, vecs @ model.encode(q, normalize_embeddings=True)))

retrievers = {"dense": dense,
              "BM25": BM25(catalog).scores,
              "BM25 + id-aware": BM25(catalog, analyzer=id_aware).scores}

s = dense("seal kit XK-4471")
print([(i, round(float(s[i]), 3)) for i in sorted(s, key=s.get, reverse=True)[:4]],
      "target rank:", rank_of(s, "XK-4471/seal kit"))

random.seed(1)
tests = [(random.randrange(4400, 4800), random.choice(list(PARTS))) for _ in range(100)]
for fmt in ["{part} XK-{n}", "{part} XK{n}"]:
    for name, retrieve in retrievers.items():
        ranks = [rank_of(retrieve(fmt.format(part=p, n=n)), f"XK-{n}/{p}") for n, p in tests]
        print(f"{fmt:15s} {name:16s} top-1 {sum(r == 1 for r in ranks):3d}/100"
              f"   top-10 {sum(r <= 10 for r in ranks):3d}/100")

# [('XK-4428/seal kit', 0.684), ('XK-4432/seal kit', 0.681), ('XK-4628/seal kit', 0.681), ('XK-4416/seal kit', 0.681)] target rank: 23
# {part} XK-{n}   dense            top-1  34/100   top-10  76/100
# {part} XK-{n}   BM25             top-1 100/100   top-10 100/100
# {part} XK-{n}   BM25 + id-aware  top-1 100/100   top-10 100/100
# {part} XK{n}    dense            top-1  21/100   top-10  56/100
# {part} XK{n}    BM25             top-1   0/100   top-10   0/100
# {part} XK{n}    BM25 + id-aware  top-1 100/100   top-10 100/100

The fix lives in the analyzer, not the ranking formula. In Elasticsearch, a word_delimiter_graph filter with catenate_all emits a joined token like xk4471 next to the parts. Another common approach copies identifiers into a keyword field whose normalizer strips separators, and queries that field too. Either way, the query must produce the same tokens the documents were indexed with, which usually means the same analyzer; a separate search-time analyzer is a common source of silent mismatches.

Stemming and stopwords: merging words on purpose

A stemmer maps word forms to one token. The Snowball English stemmer (used by fastembed's Qdrant/bm25 model) turns "leaking", "leaks" and "leaked" into leak, which is how "leak" finds "Leaking shaft seal". It also merges words you wanted apart: "university" and "universe" both become univers, "organization" becomes organ, "bearings" becomes bear. Stemming buys recall with some precision; measure whether the trade pays.

Stopwords are subtler. In the opening paraphrase failure, BM25's top hit matched only "when" and "it". In a 14-chunk corpus "when" occurs in one chunk, so its IDF (2.303) is higher than "seal"'s. Removing stopwords doesn't rescue the query. Lucene's default English stop list drops "it" and "on" but not "when", so the same impeller page still wins, on "when" alone; a longer list that also drops "when" leaves nothing to match, and every chunk scores 0. Either way the bearing article, which shares no content word with the query, stays at 0. Stopword lists also break real queries, such as the band "The Who" or a clause that hinges on "not".

Typos

warrenty on impelers contains two words that occur nowhere in the index, so the warranty article (d10) scores 0. Fuzzy matching helps: Elasticsearch's fuzziness: "AUTO" allows one edit for terms of 3–5 characters and two above that, and its match query counts swapping two adjacent characters as one edit by default. That fixes "warrenty". It also lets the 4-character token 4471 match 4417, the very confusion you kept BM25 to avoid. So apply fuzziness to words, keep identifiers in an exact field, or correct spelling before retrieval (Query Understanding & Intent). Character n-grams are another option, at the price of a much larger index and false matches on short tokens.

Languages

Analyzers are language-specific (stemmers, stopword lists, compound splitting). Chinese, Japanese and Thai, written without spaces, need a word segmenter or character n-grams before BM25 works at all. Across languages, BM25 matches only strings the two texts share: names, numbers, codes, some borrowed words. The bake-off's German query matches no token in the English help center.

Measure it: keep a slice of real identifier queries in your test set, including the variants users actually type (with and without hyphens or leading zeros, lower case), and report its recall separately. One broken variant hides easily inside a healthy average.

Move 4: Let a model choose the terms

Symptom: paraphrases miss, but you want to keep what an inverted index gives you: exact tokens, weights you can inspect, cheap queries.

Learned sparse retrieval keeps the sparse vector and lets a trained model set the weights. Two families:

• Re-weighting only. The model weights the tokens that occur in the text and nothing else. BGE-M3's sparse output works this way (a weight only for tokens present in the text, says its model card). That fixes "which words matter here", not vocabulary mismatch.
• Expansion. The model also adds terms that are not in the text. doc2query (Nogueira et al., 2019) appended generated questions to each document and ran plain BM25. SPLADE (Formal, Piwowarski & Clinchant, SIGIR 2021) does it inside the model: BERT's masked-language-model head scores all 30,522 wordpieces of the vocabulary, the weights are log-saturated, and a sparsity penalty in training drives most of them to exactly zero. What survives is the text's informative terms plus the terms the model thinks belong there; low-information pieces of the text itself, such as "the", "and" or "is", usually get zero. Elastic's ELSER and OpenSearch's neural sparse models are in this family.

Here it is on the paraphrase that beat BM25, with prithivida/Splade_PP_en_v1, an Apache-2.0 SPLADE++ model that fastembed runs on CPU. A * marks a term that is not one of the text's own wordpieces.

from fastembed import SparseTextEmbedding
from transformers import AutoTokenizer
from helpcenter import DOCS

splade = SparseTextEmbedding("prithivida/Splade_PP_en_v1")      # about 0.5 GB download
vocab = AutoTokenizer.from_pretrained("bert-base-uncased")       # same 30,522-piece vocabulary

def as_dict(emb):
    return {vocab.convert_ids_to_tokens(int(i)): float(w) for i, w in zip(emb.indices, emb.values)}

def show(label, text, vec, n):
    own = set(vocab.tokenize(text))                # the text's own wordpieces
    top = sorted(vec.items(), key=lambda x: -x[1])[:n]
    print(f"{label} ({len(vec)} non-zeros):", ", ".join(
        f"{t}{'' if t in own else '*'} {w:.2f}" for t, w in top))

q_text = "loud scraping sound when I switch it on"
q = as_dict(next(splade.query_embed(q_text)))
show("query", q_text, q, 8)
for doc_id in ["d04", "d06"]:
    d = as_dict(next(splade.embed([DOCS[doc_id]])))
    show(doc_id, DOCS[doc_id], d, 10)
    shared = {t: q[t] * d[t] for t in q.keys() & d.keys()}
    top = {t: round(v, 2) for t, v in sorted(shared.items(), key=lambda x: -x[1])[:4]}
    print(f"  dot product with query = {sum(shared.values()):.2f}; biggest terms {top}")

# query (28 non-zeros): scraping 2.27, switch 2.17, scrape* 1.81, sound 1.68, loud 1.60, on 1.59, sounds* 1.11, when 1.03
# d04 (110 non-zeros): bearing 2.10, grind 1.88, wear 1.88, rattle 1.87, startup 1.81, bearings 1.75, motor 1.69, grinding* 1.59, worn 1.55, shaft 1.45
#   dot product with query = 1.66; biggest terms {'when': 0.95, 'noise': 0.55, 'sign': 0.05, 'sound': 0.04}
# d06 (124 non-zeros): ##vita 2.88, inlet 2.15, ca 2.11, ##eller 2.01, gravel 2.00, crack 1.83, noise 1.82, ##tion 1.79, bubbles 1.75, pressure 1.72
#   dot product with query = 7.32; biggest terms {'sound': 2.69, 'noise': 1.7, 'sounds': 1.28, 'loud': 1.12}

Predict before reading on: across the whole help center, which article does SPLADE rank first for this query, and where does the bearing article land?

Note that "cavitation" is not in BERT's vocabulary, so SPLADE weights its wordpieces (ca, ##vita, ##tion). And expansion guesses in both directions: for the catalog chunk "XK-4471 seal kit…" the same model adds 45, 43 and 42 (neighbours of the 44 wordpiece), brake, and artery (from "circulation"). On the 100 catalog lookups from Move 3, SPLADE++ put the right part first 95 times with the hyphen and 51 times without it (100 and 82 in the top 10): between BM25 and MiniLM on the first format, better than both unmodified retrievers on the second.

What it costs

• Index time. Every chunk goes through a transformer. On one laptop CPU (Apple M2 Max, all threads, September 2026) this SPLADE++ model encoded about 10 short chunks per second, against about 415 for MiniLM: roughly 27 hours per million chunks against 40 minutes. Bulk indexing wants a GPU.
• Query time. The query runs the model too: about 36 ms on one CPU thread here, against about 5 ms for MiniLM. Expanded queries have more terms, so they read more postings.
• Truncation. In fastembed, Splade_PP_en_v1 stops at 128 tokens: the full manual and the manual with its last sentence deleted get identical sparse vectors. ELSER encodes the first 512 tokens of a field. BM25 has no such window.
• Language. An English-trained model expands into English.

Inference-free variants move all the model work to index time. OpenSearch's doc-only models such as opensearch-neural-sparse-encoding-doc-v3-gte expand documents with a model but encode queries with only a tokenizer and a weight lookup table (about 0.02 ms per query here). The index pays instead: 309 non-zeros per help-center chunk on average, against 87 for SPLADE++.

Options as of September 2026

Model-card benchmark numbers are self-reported: a reason to test, not a result. Learned sparse vectors fit any engine with sparse fields or sparse vectors; Qdrant stores them next to dense vectors and, since version 1.10, can compute IDF on the server (see Qdrant). Cross-language learned sparse models exist in research, such as BLADE (Nair et al., SIGIR 2023), but the everyday routes to cross-language search remain a multilingual dense model or query translation.

Measure it: compare BM25 and a learned sparse model on your paraphrase slice and your identifier slice, since expansion can help one and hurt the other, and record index size and encoding time per 1,000 chunks.

Move 5: What the bi-encoder can and cannot see

Symptom: the embedding search is great on questions but confuses near-identical codes, misses facts deep inside long documents, or ignores "not".

A recap, with the demos left to Semantic Search Principles and Vector Embeddings. A dense retriever is a bi-encoder: an embedding model encodes every chunk once, offline, into one vector, and encodes each query at search time; chunks are ranked by cosine similarity, exactly or through an approximate index (ANN Algorithms). Because the vector encodes meaning learned in training, "scraping sound when I switch it on" lands near "rattles or grinds at startup". Four blind spots matter for the bake-off below:

1. A fixed window. all-MiniLM-L6-v2 reads 256 wordpieces and silently drops the rest, with no error (Vector Embeddings). In this help center, the manual's torque sentence starts after wordpiece 271, so it never shapes the manual's vector. Longer windows exist (512 tokens for multilingual-e5-small, 8,192 for BGE-M3), but one vector for a long text is still an average of many topics.
2. Identifiers become wordpieces. XK-4471 becomes x ##k - 44 ##7 ##1 and XK-4428 becomes x ##k - 44 ##28: shared pieces plus one or two short ones with no numeric meaning. With 400 look-alike codes, the right one came 23rd in the opening.
3. Language is whatever the model learned. MiniLM is English. A multilingual model such as intfloat/multilingual-e5-small (384 dimensions; its model card says every input should start with query: or passage: ) maps a German question and an English article close together (Semantic Search Principles).
4. Negation barely moves the vector. Test it with a minimal pair, the same query with and without the negation, and turn constraints such as "without" into structured filters (Semantic Search Principles, Metadata & Filtering).

⚠️ Scores are not evidence of relevance. In the opening table 23 chunks sat within 0.008 cosine of each other and one was right. Cosine scores are not probabilities and not comparable across models (Cosine Similarity & Distance Metrics).

Measure it: minimal pairs for negation and look-alike codes, plus a "needle at the end" test for long chunks, take minutes and tell you more about a model than its leaderboard rank.

The bake-off: eight queries, four retrievers

Now side by side on the help center: eight queries, one failure type each, and four retrievers: BM25 (Move 2), SPLADE++ (Move 4), MiniLM (English, dense) and multilingual-e5-small (dense). Each cell is the rank of the one relevant chunk among 14; "no match" means a score of 0 (no shared token).

from fastembed import SparseTextEmbedding
from sentence_transformers import SentenceTransformer
from helpcenter import DOCS
from bm25 import BM25

QUERIES = [  # (slice, query, relevant chunk)
    ("identifier",     "seal kit XK-4471",                                "d01"),
    ("identifier",     "what does E17 mean",                              "d12"),
    ("rare term",      "cavitation",                                      "d06"),
    ("typo",           "warrenty on impelers",                            "d10"),
    ("paraphrase",     "loud scraping sound when I switch it on",         "d04"),
    ("other language", "Wasser tropft unter der Pumpe",                   "d05"),
    ("negation",       "pump that works without an external controller",  "d08"),
    ("long document",  "flange bolt torque",                              "d09"),
]
ids, texts = list(DOCS), list(DOCS.values())

def dense_retriever(name, q_prefix="", p_prefix=""):
    model = SentenceTransformer(name)
    vecs = model.encode([p_prefix + t for t in texts], normalize_embeddings=True)
    return lambda q: dict(zip(ids, vecs @ model.encode(q_prefix + q, normalize_embeddings=True)))

def splade_retriever():
    model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
    docs = [dict(zip(e.indices, e.values)) for e in model.embed(texts)]
    def search(q):
        qv = next(model.query_embed(q))
        return {i: sum(w * d.get(t, 0.0) for t, w in zip(qv.indices, qv.values))
                for i, d in zip(ids, docs)}
    return search

retrievers = {
    "BM25": BM25(DOCS).scores,
    "SPLADE++": splade_retriever(),
    "MiniLM": dense_retriever("sentence-transformers/all-MiniLM-L6-v2"),
    "mE5-small": dense_retriever("intfloat/multilingual-e5-small", "query: ", "passage: "),
}

def rank_of(scores, target):
    if scores[target] <= 0:
        return "no match"
    return 1 + sum(s >= scores[target] for i, s in scores.items() if i != target)

ranks = {}
print(f"{'slice':15s}" + "".join(f"{name:>11s}" for name in retrievers))
for slice_, q, rel in QUERIES:
    ranks[q] = {name: rank_of(r(q), rel) for name, r in retrievers.items()}
    print(f"{slice_:15s}" + "".join(f"{str(v):>11s}" for v in ranks[q].values()))

Predict first: for each row, which retrievers put the right chunk first? Write your guesses down, then open the result.

Every row has a mechanism behind it:

• Identifiers and rare terms: everyone wins at 14 chunks. The dense failure from the opening needed 400 look-alike part numbers, which is why you test at realistic scale.
• Typo: BM25 has no token to match. The wordpieces of "warrenty" (warren ##ty) and "impelers" (imp ##ele ##rs) still land near the right meaning for both dense models. SPLADE++ matched only the wordpiece imp (shared with "impellers"), so three impeller-related chunks outscored the warranty article; it expanded "warrenty" towards the name Warren.
• Paraphrase: dense is best, expansion helps, BM25 cannot match. The multilingual model is weaker here (rank 4) than English MiniLM: no model is better on every slice.
• Other language: only the multilingual model works.
• Negation: mE5 ranked d08 first, but by a hair. On a minimal pair (the same query with "needs" instead of "works without") it flipped to d07, and both margins were about 0.01. MiniLM preferred d08 for both wordings, so its choice had nothing to do with "without". That is luck, not a capability.
• Long document: MiniLM never sees the torque sentence (rank 12) and SPLADE++ stops at 128 tokens (rank 4). BM25 sees every token, but length normalization drops the manual (256 tokens, seven times the average) below the short gasket article; mE5, with 512 tokens, also ranks it second. The cure for all four is smaller chunks, not another retriever.

⚠️ Eight queries show mechanisms; they do not measure quality, and one reworded query can flip a cell. Decisions need a few hundred labeled queries (see "Decide by slice" below).

What the benchmarks say, and what they don't

The standard reference is BEIR (Thakur et al., NeurIPS 2021 Datasets and Benchmarks), which tested retrievers zero-shot, with no training on the target domain, on 18 datasets. From the paper's Table 2 (nDCG@10):

• Training-domain wins don't transfer. On MS MARCO, where the neural models were trained, BM25 scored 0.228 against 0.408 for TAS-B. In the paper's average relative to BM25 across the zero-shot datasets, TAS-B came out 2.8% below BM25, ANCE 7.4% and DPR 47.7%. In the authors' words, "BM25 is a robust baseline."
• Domain and task shift hurt dense models most: Touché-2020 (argument retrieval), BM25 0.367 against at most 0.240 for the dense models; BioASQ (biomedical), 0.465 against at most 0.398.
• Learned term weighting failed; document expansion helped (the paper's framing). DeepCT and SPARTA fell below BM25 on nearly all datasets; docT5query, which appends generated queries to documents, beat it on 11 of 18.
• Quality costs latency. Reranking BM25's top 100 with a cross-encoder was best on average (11% above BM25), at about 450 ms per query on a GPU.
• The authors' own caveat: many BEIR datasets were labeled from candidates that lexical search found, which can favour lexical systems.

Embedding models have moved on since. E5 (Wang et al., 2022) was "the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data", and the SPLADE-v3 paper reports it as significantly more effective than BM25 over more than 40 query sets. What has not changed: an average hides the datasets where BM25 still wins, and your corpus is exactly one dataset. BM25 costs almost nothing to run, so it is the baseline every new retriever must beat on your queries.

Costs and updates

Memory and latency

BEIR also measured one million DBPedia documents on 2021 hardware: BM25 used a 0.4 GB index and about 20 ms per query on CPU; a 768-dimensional dense model with exact search used 3 GB and 125 ms on CPU (14 ms on a GPU); SPARTA, a sparse model with 2,000 non-zeros per document, needed 12 GB. Redo the arithmetic for your setup:

• Dense: 1,000,000 chunks × 384 dimensions × 4 bytes = 1.536 GB of float32; at 768 dimensions, 3.072 GB (2.86 GiB). Graph indexes and quantization change this (Vector Database Architecture).
• Learned sparse: 87 non-zeros per chunk (SPLADE++ on the help center), stored as a 4-byte term id plus a 4-byte weight: 1,000,000 × 87 × 8 bytes = 0.696 GB before index structures. The inference-free model's 309 non-zeros come to 2.47 GB.
• BM25: compressed postings for the tokens each chunk contains; usually the smallest, as in BEIR.

And query latency is more than index search. BM25 runs no model. A dense query first runs an embedding model (about 5 ms for MiniLM on one CPU thread here, or a network round trip to an API); a SPLADE query runs a bigger one (about 36 ms) and then reads more postings.

What happens when the data changes

That first cell surprises people. Watch scores move without anyone touching the chunks:

from helpcenter import DOCS
from bm25 import BM25

query = "shaft seal leaking"
before = BM25(DOCS)

# Import 40 seal-kit pages from the catalog into the same index. d05 itself is untouched.
imported = {f"kit-{n}": f"XK-{n} seal kit: shaft seal and O-rings for the XK-{n} circulation pump."
            for n in range(4500, 4540)}
after = BM25({**DOCS, **imported})

for name, bm in [("before", before), ("after", after)]:
    s = bm.scores(query)
    print(f"{name:6s} N={bm.N:2d} idf(seal)={bm.idf('seal'):.3f} idf(leaking)={bm.idf('leaking'):.3f}"
          f"  d05={s['d05']:.3f}  d01={s['d01']:.3f}")

# before N=14 idf(seal)=1.003 idf(leaking)=2.303  d05=6.004  d01=2.802
# after  N=54 idf(seal)=0.190 idf(leaking)=3.602  d05=4.040  d01=0.454

"seal" became common, so its IDF fell from 1.003 to 0.190 and d01 dropped from 2.802 to 0.454. "leaking" became relatively rarer and its IDF rose, which is why d05 still leads. Rankings mostly survive this; a score threshold tuned last month does not. Two production versions of the same effect:

• Sharding. Elasticsearch's default query_then_fetch scores each shard with its own term statistics, so a document's score depends on where it lives, most visibly on small indexes. dfs_query_then_fetch gathers global statistics first, at the cost of an extra round trip (Elastic).
• Deletes. Lucene's docFreq counts deleted documents until their segments are merged, so IDF lags behind a mass delete.

For dense and learned sparse retrieval the expensive event is a model change: re-encode every chunk before queries switch over, because old and new vectors are not comparable. A dense model with a different dimension at least fails loudly (the database or library rejects the query vector). One with the same dimension, or a new learned sparse model, where nothing checks the vocabulary, keeps returning results that are quietly worse: some still right, some wrong, which is why nobody notices (Vector Embeddings). Keeping indexes fresh is the subject of Data Freshness & Lifecycle.

Decide by slice, not by fashion

The question is never "sparse or dense?" in general. It is "which kinds of queries does my traffic contain, and which retriever handles each?" A procedure that fits in a week:

1. Sample and slice. Pull a few hundred real queries from logs and tag each: identifier or code, natural-language question, keyword phrase, other language, misspelled.
2. Label 50–200 of them with the chunks that answer them. Tedious, and the most valuable data you will own (Retrieval Metrics).
3. Run the baselines: BM25 with a sensible analyzer plus one dense model, and compute retrieval recall@10 (the share of each query's relevant chunks that reach the top 10) per slice, not just overall.
4. Choose from the slices, using the table below.
5. Re-measure after every change, watching the slices you weren't trying to improve.

Continuing from the bake-off block: with one relevant chunk per query, recall@3 is the share of queries whose chunk made the top 3.

def found(rank, k):
    return rank != "no match" and rank <= k

k = 3
for name in retrievers:
    hits = [q for q in ranks if found(ranks[q][name], k)]
    print(f"{name:10s} recall@{k} = {len(hits)}/{len(ranks)}")
either = [q for q in ranks if found(ranks[q]["BM25"], k) or found(ranks[q]["MiniLM"], k)]
print(f"BM25 or MiniLM in top {k}: {len(either)}/{len(ranks)}")

# BM25       recall@3 = 5/8
# SPLADE++   recall@3 = 4/8
# MiniLM     recall@3 = 5/8
# mE5-small  recall@3 = 7/8
# BM25 or MiniLM in top 3: 7/8

BM25 and MiniLM tie at 5 of 8 but fail on different queries, so together their top-3 lists cover 7 of 8. Only a per-query or per-slice view shows that, and it is the argument for Hybrid Retrieval Systems.

Explaining a ranking. When auditors ask why a document was retrieved, BM25 and learned sparse models give per-term contributions. If a hybrid system ranked it because of the dense score, report each retriever's rank or score; don't present the BM25 term matches as the reason.

Your turn: change one requirement. The pump maker opens sales in Germany and Poland. The manuals stay in English, and a third of the help-center queries will arrive in German or Polish. Your current system is BM25 plus MiniLM. What do you change, and what do you measure?

Final round: no label on the problem

Real tickets don't say which move they need. For each: what failed, why, and what is the first fix?

1. After a migration from Elasticsearch to a vector-only database, searches for Oracle error ORA-01722 return general articles about Oracle errors, and the article about ORA-01722 is on page two.

2. An Elasticsearch index of 3,000 FAQ entries has two shards. Two entries with identical text (a duplicate nobody cleaned up) score 8.1 and 9.6 for the same query. A threshold of 9.0 decides which answers are shown.

3. A SPLADE index over insurance policies misses questions about exclusions, even though every policy has an exclusions section. That section is always near the end of a 3,000-word policy.

Cheat sheet: symptom → move → cost

Before moving on, take a search system you know and, for each main type of query, say which tokens or vector the retriever sees, which bake-off failure could hit it, how a minimal pair or a small slice would prove it, and what the fix costs. Then fusion, filtering and reranking are just the next layer.

Next: Hybrid Retrieval Systems to fuse the two lists, Metadata & Filtering for the constraints no retriever reads reliably, and Reranking Models for reordering the fused candidates.