Cosine Similarity & Distance Metrics

The threshold that let the Lisbon office in

A support bot has ten help-center chunks and one rule: keep every chunk whose cosine similarity with the question is at least 0.5, and if nothing passes, answer "I don't have that information." With the small open model all-MiniLM-L6-v2 it looked sensible. For "I forgot my login" exactly one chunk passed: the password-reset article, at 0.609.

Then the team switched to intfloat/e5-small-v2, because it reads up to 512 tokens instead of 256. They even added the query: and passage: prefixes its model card requires. Same question, same rule, and all ten chunks passed. The lowest of them was "Our office in Lisbon is closed on public holidays," at 0.719. That is higher than the 0.609 the correct chunk got from the old model.

Nothing crashed. Both models put the password article first. What broke was the meaning the team had given to a number: "0.5" was a fact about one model on one corpus, not a fact about similarity.

That is this lesson in one sentence: a similarity score is produced by one model, under one metric, in one engine's convention, and it ranks candidates for one query. Anything more — which metric, which direction is better, what counts as "good enough" — you have to check or measure.

Each move below starts with a symptom you will actually see:

This lesson assumes you know what an embedding is and why similar texts land near each other (Semantic Search Principles, Vector Embeddings). Notation: n vectors in the index, dim dimensions each, k results returned.

Before any metric: five questions

A score only means something together with its context. Before you trust a number, or a threshold built on one, answer these:

💡 Habit worth building: write the model name, prefix, dimension, normalization and metric next to every stored vector's collection or table. When you change any of them, everything downstream of the score is suspect.

Move 1: Three rulers for "close"

Symptom: two libraries, or two teammates, disagree about which chunk is closest to the query.

Predict first. A query q = (2, 1) and three documents: A = (2, 0), B = (4, 2), C = (6, 0). Which one is closest?

The three measures:

• Dot product: a·b = Σ aᵢbᵢ. Range: any real number. Higher means more similar.
• Cosine similarity: cos(a, b) = a·b / (‖a‖ ‖b‖). Range: −1 to 1. Higher means more similar. Cosine distance is 1 − cos(a, b), range 0 to 2, where lower is better.
• Euclidean (L2) distance: ‖a − b‖ = √(Σ (aᵢ − bᵢ)²). Range: 0 upward. Lower means more similar.

import numpy as np

q = np.array([2.0, 1.0])
docs = {"A": np.array([2.0, 0.0]), "B": np.array([4.0, 2.0]), "C": np.array([6.0, 0.0])}

def dot(a, b):
    return float(a @ b)

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def l2(a, b):
    return float(np.linalg.norm(a - b))

print("doc  length  dot  cosine     L2")
for name, d in docs.items():
    print(f"{name:>3} {np.linalg.norm(d):7.3f} {dot(q, d):4.0f} {cosine(q, d):7.3f} {l2(q, d):6.3f}")

# doc  length  dot  cosine     L2
#   A   2.000    4   0.894  1.000
#   B   4.472   10   1.000  2.236
#   C   6.000   12   0.894  4.123

What each ruler rewards

The geometric form of the dot product explains the whole table:

a·b = ‖a‖ × ‖b‖ × cos(θ)        θ = angle between a and b

• Cosine sees only direction. B = 2q, so it points exactly where q points: cosine 1.000. A and C point the same way as each other, so they tie at 0.894. Cosine cannot tell them apart, because they differ only in length.
• Dot product multiplies by length. C has the same direction as A and three times the length, so it gets exactly three times the score (12 vs 4). Length is a multiplier on alignment.
• L2 measures the gap between the tips. It punishes being too long and too short. A's tip is 1.0 away from q's tip. B is perfectly aligned but twice as long as q, so its tip is 2.236 away.

L2 is tied to the other two by one identity:

‖a − b‖² = ‖a‖² + ‖b‖² − 2 a·b

Check it on C: 5 + 36 − 2 × 12 = 17, and √17 = 4.123. ✓ So L2 mixes three things: both lengths and the dot product.

🧠 Shadows. Draw a and b as arrows from the same origin. The dot product is the length of a's shadow on b's line, times the length of b. Cosine keeps only the angle between the arrows. L2 is the distance between the arrowheads. The picture stops working in two places: a shadow can point backwards (a negative dot product), and in 384 dimensions you cannot draw it. The algebra, though, is exactly the same.

Your turn: add D = (−2, −1) and E = (−1, 2). Predict all three scores for each. Which of them does L2 rank above C?

Move 2: Normalize, and the rulers agree

Symptom: a teammate proposes rebuilding the index with dot product "because it's faster", or with Euclidean "because it's more accurate".

Divide every vector by its length (L2-normalize it). With ‖a‖ = ‖b‖ = 1, the identities from Move 1 collapse:

a·b         = cos(a, b)
‖a − b‖²    = 1 + 1 − 2 a·b  =  2 − 2·cos(a, b)

A higher cosine means a higher dot product and a smaller L2 distance, for every candidate. So on unit vectors, sorting by cosine (descending), dot (descending) or L2 (ascending) gives the same ranking. Only the numbers differ.

Apply it to Move 1. After normalizing, A and C both become (1, 0), and B and q both become the same unit vector, (0.894, 0.447). B now scores dot = cosine = 1 with L2 = 0. A and C both score dot = cosine = 0.894 with L2 = 0.459. That is one ranking under all three rulers. Notice what it cost: A and C are now indistinguishable. Normalizing throws length away. That is fine when length was noise and a loss when it was signal (Move 3).

Proof on real vectors

all-MiniLM-L6-v2 ends its pipeline with a Normalize module, so its vectors are already unit length. Put them in two FAISS indexes, one inner product and one L2:

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

chunks = [   # five of the opening story's ten help-center chunks
    "To reset your password, click 'Forgot password' on the sign-in page and follow the link we email you.",
    "Two-factor authentication can be turned on under Settings > Security.",
    "Deleted accounts are kept for 14 days and then permanently erased.",
    "Invoices are emailed on the first day of each billing cycle and can be downloaded as PDF.",
    "Our office in Lisbon is closed on public holidays.",
]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
X = model.encode(chunks)                    # float32, shape (5, 384)
q = model.encode(["I forgot my login"])     # shape (1, 384)
print("norms:", np.round(np.linalg.norm(X, axis=1), 4))

ip = faiss.IndexFlatIP(X.shape[1]); ip.add(X)
l2 = faiss.IndexFlatL2(X.shape[1]); l2.add(X)
D_ip, I_ip = ip.search(q, 5)
D_l2, I_l2 = l2.search(q, 5)
print("IP order:", I_ip[0], " scores:", np.round(D_ip[0], 3))
print("L2 order:", I_l2[0], " values:", np.round(D_l2[0], 3))
print("2 - 2*IP:          ", np.round(2 - 2 * D_ip[0], 3))

X128 = X[:, :128]    # keep the first 128 dims (MiniLM is not trained for this; it only shows the length)
print("norms after truncation:", np.round(np.linalg.norm(X128, axis=1), 3))

# norms: [1. 1. 1. 1. 1.]
# IP order: [0 2 1 4 3]  scores: [ 0.609  0.338  0.284  0.071 -0.017]
# L2 order: [0 2 1 4 3]  values: [0.781 1.325 1.433 1.859 2.034]
# 2 - 2*IP:           [0.781 1.325 1.433 1.859 2.034]
# norms after truncation: [0.569 0.556 0.548 0.587 0.583]

Same order, and the L2 index's values are exactly 2 − 2·IP. They are squared distances: IndexFlatL2 returns squared L2, which is enough for ranking and saves a square root. The same holds at scale. On the BEIR SciFact benchmark (5,183 paper abstracts, 300 test queries, exact search, run for this lesson), all-MiniLM-L6-v2 scored retrieval recall@10 = 0.783 and nDCG@10 = 0.645 under dot, cosine and L2 alike, to the last digit. Every SciFact number in this lesson embeds each abstract as its title, a space and the abstract text, the format the BEIR toolkit uses, and was computed on a CPU. (Retrieval recall@10 is the share of the judged-relevant abstracts that appear in the top 10.)

OpenAI states the same for its API models: its embeddings "are normalized to length 1", so "cosine similarity and Euclidean distance will result in the identical rankings" (OpenAI embeddings guide).

"But dot product is faster"

Only if your cosine recomputes norms on every query. In numpy, on 200,000 random 384-dimensional float32 vectors on one laptop, one query took about 5 ms as a plain dot product and about 5 ms as cosine with the norms computed once in advance, but about 27 ms when the cosine code recomputed every document's norm per query. Engines do the precomputation for you. Qdrant's documentation describes normalizing each vector once, "when adding it to the collection", after which comparing two vectors is a dot product (Qdrant search docs). In two quick runs of FAISS flat search on the same data, which of IP and L2 was faster changed from one run to the next. On normalized vectors, choose the metric by what the model expects, never by speed.

Traps that break unit length

• ⚠️ Truncation. Cutting a vector to fewer dimensions (Matryoshka-style shortening, covered in Vector Embeddings) leaves it shorter than 1: 0.548 to 0.587 in the run above. Dot and L2 now mix leftover lengths back into the score. OpenAI's guide says it plainly: "When you change the dimension manually, you need to be sure to normalize the dimensions of the embedding." Re-normalize after truncating. Doing it twice is harmless.
• ⚠️ Zero vectors. A failed embedding call stored as a zero placeholder has no direction. In numpy, cosine becomes nan (0/0). pgvector 0.8.6 returns NaN for <=>, which sorts after every real number. Qdrant 1.19 scores it 0 in a Cosine collection. None of these raises an error. Reject zero-norm vectors at ingest and alert on them.
• ⚠️ In-place and float32. faiss.normalize_L2(x) modifies x and accepts only float32; a float64 array raises TypeError.

Your turn: your model outputs unit vectors, and a teammate wants the Qdrant collection switched from Cosine to Euclid "because distances are easier to explain". What changes: the ranking, the scores, the thresholds, the collection?

Move 3: When length is the signal

Symptom: long, popular or generic items keep beating short, specific ones. Or the model card says "dot product".

Take a query q of length 1 and two catalogue pages. The specific blender page has vector length 0.6 and cosine 0.94 with the query. The appliance bundle page has length 1.4 and cosine 0.71. Using a·b = ‖a‖‖b‖cos θ:

Cosine ranks the specific page first. Dot product ranks the bundle first.

Predict: the query vector doubles in length. Does the dot-product ranking change?

Is the bundle winning a bug? That depends on what vector length means for this model:

• Trained with cosine similarity: cosine ignores length, so training never shaped it. Length is a by-product, and scoring with dot product adds noise. When the card is silent, measure: paraphrase-MiniLM-L6-v2 has unnormalized output and names no score function, and on SciFact it reached recall@10 0.662 with cosine and 0.633 with dot.
• Trained with dot product: length is part of what the model learned. The card of multi-qa-MiniLM-L6-dot-v1 says "Produces normalized embeddings: No" and "Suitable score functions: dot-product". On SciFact it reached 0.669 with dot and 0.633 with cosine. Normalizing it deletes learned signal.
• Recommenders: in a matrix-factorization model the score is user·item, so an item's length works like a learned per-item boost. Finding the items with the largest dot product for a query is called maximum inner product search (MIPS).
• Sparse vectors (BM25-style weights, SPLADE) are scored by dot product too, and Qdrant's sparse vectors support nothing else. The mechanics are in Sparse vs Dense Retrieval.

Length means what training made it mean

The folk story says "long documents get long vectors, so dot product favors long documents." Sometimes. The BEIR authors (Thakur et al., 2021, Appendix H) trained two otherwise identical models, one with cosine and one with dot product. On every dataset the cosine model preferred shorter documents; on TREC-COVID it pulled up papers that had a title and no abstract. The dot-product model gained 15.3 points of nDCG@10 there (0.482 to 0.635), but it was worse than the cosine model on most other datasets.

Now the opposite case. On SciFact, multi-qa-MiniLM-L6-dot-v1 gave vectors with lengths from 7.0 to 10.6, and the length fell as abstracts got longer (Spearman correlation −0.68 between length and word count). Dot product retrieved shorter abstracts than cosine did (median 190 vs 244 words in the top 10), and it was still the more accurate metric for this model. Do not reason from folklore. Plot vector length against document length on your own corpus, and compare the documents each metric puts in the top 10.

MIPS is not nearest-neighbor search

Predict: three items (1, 0), (3, 1) and (0.5, 0.5). Query with item 0's own vector. Which item has the largest dot product with it?

import numpy as np

items = np.array([[1.0, 0.0],     # item 0
                  [3.0, 1.0],     # item 1: longer, 18 degrees off
                  [0.5, 0.5]])    # item 2
q = items[0]                      # query with item 0's own vector
print("dot:", items @ q, "-> best:", int(np.argmax(items @ q)))

# Bachrach et al. (2014): add one coordinate so that L2 order == dot order
norms = np.linalg.norm(items, axis=1)
M = norms.max()
items_aug = np.column_stack([items, np.sqrt(M**2 - norms**2)])
q_aug = np.append(q, 0.0)
print("augmented item norms:", np.round(np.linalg.norm(items_aug, axis=1), 3))
print("order by dot:          ", np.argsort(-(items @ q)))
print("order by L2, augmented:", np.argsort(np.linalg.norm(items_aug - q_aug, axis=1)))

# dot: [1.  3.  0.5] -> best: 1
# augmented item norms: [3.162 3.162 3.162]
# order by dot:           [1 0 2]
# order by L2, augmented: [1 0 2]

Item 1 beats item 0's own vector. Under dot product a vector need not be its own best match. The hnswlib README says so directly: "inner product is not an actual metric. An element can be closer to some other element than to itself" (hnswlib). Nearest-neighbor indexes were designed around distances that behave like distances, so MIPS needed a bridge. Bachrach et al. (RecSys 2014) gave one for the Xbox recommender: append √(M² − ‖x‖²) to every item (M is the largest item length) and 0 to the query. Every augmented item then has length M, and ‖q̃ − x̃‖² = ‖q‖² + M² − 2 q·x, so the L2 order is exactly the dot order, as the run shows.

Today you rarely need the trick by hand. Qdrant (Distance.DOT), FAISS (METRIC_INNER_PRODUCT, also for HNSW), pgvector (vector_ip_ops) and hnswlib ('ip') all index inner product directly. Whichever you use, measure ANN recall on your own vectors, because what a given efSearch buys depends on the data (ANN Algorithms).

And recommenders do not always keep the lengths. YouTube's two-tower retrieval model (Yi et al., RecSys 2019) found that "adding embedding normalization ... improves model trainability and thus leads to better retrieval quality", and it indexes the normalized vectors. "Recommenders need MIPS" is a design choice, not a law.

Your turn: your catalogue search uses a dot-trained model. Product now requires that "niche items with a strong match must be able to reach page one". Your teammate says to switch the collection to cosine. Decide, and say what you would measure.

Move 4: Use the metric the model was trained with

Symptom: recall on your labeled queries is a few points lower than it should be, and nothing raised an error.

Contrastive training shows the model a query, a matching passage and many non-matching ones, and pushes the match's score above the others. "Score" here means one specific similarity function: often cosine divided by a temperature, sometimes the raw dot product. The card of multi-qa-MiniLM-L6-dot-v1 says it was trained with "MultipleNegativesRankingLoss using CLS-pooling, dot-product as similarity function". The vector geometry is shaped for that function, and scoring it with another function measures something the model was never trained on.

Here is what the mismatch cost on SciFact (5,183 abstracts, 300 queries, exact search; each text is the title, a space and the abstract, cut at the model's maximum length):

nDCG@10 moved the same way (0.539 / 0.517 / 0.455 for the dot model; 0.472 / 0.526 / 0.524 for the paraphrase model). These are one run's numbers, and the size of a gap is fragile: joining title and abstract with ". " instead of a space moved the dot model's gap from 3.6 to 5.7 points. The direction held both times. The table has two halves. For a normalized model the metric question is settled: all three rulers agree. For an unnormalized model it moves recall by a few points (ten for L2 on the dot model), and the right answer is whatever the model was trained with.

How to find out

1. Read the model card. Look for "normalized embeddings" and "suitable score functions" (or "similarity"). Vendors of API models document this too, as OpenAI does.
2. Ask the library, but know its defaults. sentence-transformers exposes model.similarity_fn_name. When the saved config does not declare one, version 6.1.0 falls back to cosine. So "cosine" may just be a default rather than a statement from the model's authors.
3. Measure the norms on a thousand of your own chunks. If they are all 1.0, the metric choice is free.

import numpy as np
from sentence_transformers import SentenceTransformer

texts = ["refund policy",
         "How do I restore a single file from last week's backup snapshot?"]
for name in ["sentence-transformers/all-MiniLM-L6-v2",
             "sentence-transformers/paraphrase-MiniLM-L6-v2",
             "sentence-transformers/multi-qa-MiniLM-L6-dot-v1"]:
    model = SentenceTransformer(name)
    last = type(model[len(model) - 1]).__name__        # last module in the pipeline
    norms = np.linalg.norm(model.encode(texts), axis=1)
    print(f"{name.split('/')[1]:27} last={last:9} "
          f"similarity_fn_name={model.similarity_fn_name:6} norms={np.round(norms, 2)}")

# all-MiniLM-L6-v2            last=Normalize similarity_fn_name=cosine norms=[1. 1.]
# paraphrase-MiniLM-L6-v2     last=Pooling   similarity_fn_name=cosine norms=[8.64 5.58]
# multi-qa-MiniLM-L6-dot-v1   last=Pooling   similarity_fn_name=dot    norms=[7.02 6.99]

Only the third model's config actually declares a function ("dot"). The first two print "cosine" because of the fallback. For all-MiniLM-L6-v2 that is harmless, since its vectors are unit length. For paraphrase-MiniLM-L6-v2 it is an unverified default, which is why the SciFact run above was worth doing.

Where mismatches actually happen

A query cannot quietly use a different metric from its collection. In Qdrant the distance is part of the collection's vector configuration, and it is required: VectorParams(size=4) without distance fails validation, so there is no default. Updating a collection can change HNSW, quantization and storage settings, but not its distance. The real failures come from somewhere else:

• The collection was created with the wrong metric for the model. With Cosine, Qdrant stores unit-length copies, so the lengths a dot-trained model produced are gone for good. Switching to Dot later means re-uploading the original vectors, and if you did not keep them, re-embedding the corpus.
• Client-side rescoring with a different function, for example rescoring an index's candidates with a hand-written cosine while the index ranked by dot product.
• A migration that picks up a default. FAISS's IndexFlat and index_factory(dim, "Flat") default to L2. hnswlib makes you name a space, and each space returns a different kind of number (Move 5).
• A threshold carried from one score scale to another (Moves 5 and 6).

Your turn: you are upgrading to a new model whose card says "normalized: yes". The current Qdrant collection uses Dot. Must you change the metric? What must you change?

Move 5: Read the number your engine hands you

Symptom: results come back upside down, a threshold keeps the worst matches, or latency jumps after a harmless-looking query edit.

Every engine answers "how close?" with its own number. Here is what each one returns, verified against the versions named:

pgvector returns a negative inner product because, in its README's words, "Postgres only supports ASC order index scans on operators". hnswlib does the same kind of thing: it turns every space into a distance so that "smaller is better" holds everywhere.

The same four points in each Qdrant distance, with an in-memory client:

from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")
points = {1: [3, 4, 0, 0], 2: [1, 0, 0, 0], 3: [0.6, 0.8, 0, 0], 4: [6, 8, 0, 0]}
query = [1, 1, 0, 0]

for dist in [models.Distance.COSINE, models.Distance.DOT,
             models.Distance.EUCLID, models.Distance.MANHATTAN]:
    name = f"demo_{dist.value.lower()}"
    client.create_collection(name, vectors_config=models.VectorParams(size=4, distance=dist))
    client.upsert(name, points=[models.PointStruct(id=i, vector=v) for i, v in points.items()])
    hits = client.query_points(name, query=query, limit=4).points
    print(f"{dist.value:9}", [(h.id, round(h.score, 3)) for h in hits])

stored = client.retrieve("demo_cosine", ids=[1], with_vectors=True)[0].vector
print("point 1 as stored in the Cosine collection:", [round(x, 3) for x in stored])

# Cosine    [(4, 0.99), (3, 0.99), (1, 0.99), (2, 0.707)]
# Dot       [(4, 14.0), (1, 7.0), (3, 1.4), (2, 1.0)]
# Euclid    [(3, 0.447), (2, 1.0), (1, 3.606), (4, 8.602)]
# Manhattan [(3, 0.6), (2, 1.0), (1, 5.0), (4, 12.0)]
# point 1 as stored in the Cosine collection: [0.6, 0.8, 0.0, 0.0]

Points 1, 3 and 4 share a direction, so Cosine ties them (their order among themselves is arbitrary). Dot ranks the longest first, and Euclid and Manhattan rank the nearest tip first. Qdrant always returns the best hit first, but "best" is the highest score for Cosine and Dot and the lowest for Euclid and Manhattan. score_threshold follows the same rule: in the docs' words, "higher scores of Euclidean metric are considered more distant and, therefore, will be excluded." With score_threshold=1.0, the Euclid collection returned only point 3 (0.447).

Predict: your code re-sorts Qdrant hits with sorted(hits, key=lambda h: h.score, reverse=True) before building the prompt. It worked for months on a Cosine collection. The data moves to a Euclid collection. What does the prompt get first?

pgvector: the query shape decides whether the index runs

In pgvector an HNSW index serves one operator. The README's rule is: "The query needs to have an ORDER BY and LIMIT, and the ORDER BY must be the result of a distance operator (not an expression) in ascending order." Checked with EXPLAIN on pgvector 0.8.6:

CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);

-- Uses the HNSW index (approximate, fast)
SELECT id, 1 - (embedding <=> '[1,1,0,0]') AS cosine_similarity
FROM items
ORDER BY embedding <=> '[1,1,0,0]'
LIMIT 10;

-- Exact answer, but no index: sequential scan + sort of the whole table
SELECT id, 1 - (embedding <=> '[1,1,0,0]') AS cosine_similarity
FROM items
ORDER BY cosine_similarity DESC
LIMIT 10;

-- Also a sequential scan: the L2 operator cannot use a cosine index
SELECT id FROM items ORDER BY embedding <-> '[1,1,0,0]' LIMIT 10;

All three return sensible rows, so quality tests pass. Only your latency percentiles will show the difference: the "tidier" second query reads every row. This is the real form of the "metric mismatch" you will hear about. The engine doesn't return wrong neighbors. It stops using the index.

Your turn: you tuned "keep chunks with Qdrant Cosine score ≥ 0.72". The same unit vectors now also live in (a) pgvector, queried with <=>, (b) a FAISS IndexFlatL2, and (c) a Qdrant Euclid collection. Write the cutoff for each that keeps exactly the same chunks.

Move 6: A score ranks; it does not certify

Symptom: a fixed cutoff suddenly returns nothing, or everything. Or someone proposes "cosine above 0.8 means relevant".

Back to the Lisbon office. On SciFact, with each query's top 20 abstracts inspected (exact search, the prefixes each model card asks for), three small models gave these scores:

SciFact judges only a few abstracts per query, so "other" includes some unjudged relevant ones, and the precision behind the F1 row is an underestimate. The gaps between the columns are what matter: the same cutoff means "almost nothing" for one model and "everything" for another. The model cards say so. E5's card calls scores bunched between 0.7 and 1.0 "a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss" (e5-small-v2). BGE's card explains the same effect for its earlier models (a temperature of 0.01 put their similarities "about in the interval [0.6, 1]"), recommends v1.5 because it "alleviates the issue of the similarity distribution", and adds that "what matters is the relative order of the scores, not the absolute value" (bge-small-en-v1.5). Alleviated is not gone: in the table, v1.5's median over all pairs is 0.563, against 0.097 for MiniLM.

Why the scores bunch. The training loss sees cos / τ, where τ is the temperature. With τ = 0.01, a cosine gap of 0.05 is already a logit gap of 5, which is plenty to rank the right passage first. So the model never needs to push unrelated texts toward 0. For contrast, random directions in 384 dimensions have cosines around 0 with a spread of about 1/√384 ≈ 0.051. Trained embeddings are not random, and many models put unrelated texts well above 0.

Three things a score is not

1. Not a probability. Cosine 0.82 does not mean "82% relevant". It means the angle is about 35 degrees.
2. Not comparable across models, prefixes or metrics. You just saw it.
3. Not stable across queries, even within one model. On SciFact, one query's correct top result scored 0.334 with all-MiniLM-L6-v2, while another query's top result, which the judges did not mark relevant, scored 0.844. No single cutoff separates those two. Score levels shift from query to query with wording and topic, so a global cutoff is always a compromise.

What to do instead

• Rank, then decide with something that reads the text. Take a candidate pool by similarity, and let a reranker and the generator's grounding instruction handle "not enough evidence" (Classic RAG Pipeline). Reranker scores are not calibrated either, and their scale depends on the model and library (some return raw logits, some a 0–1 sigmoid), so any cutoff on them needs the same tuning.
• If you need a cutoff (deduplication, routing to "I don't know", cost control), tune it on labeled pairs from your own data, and choose it by what each kind of error costs you. Re-tune after any change to the model, prefix, metric or corpus.
• Watch the distribution. Log the top-1 score per query. A sudden shift in its median can be the first visible sign that someone changed the model or dropped a prefix.

A threshold sweep is a few lines of code. This one reproduces the opening story, with six labeled questions over the ten help-center chunks:

import numpy as np
from sentence_transformers import SentenceTransformer

chunks = [
    "Refunds: you can request a full refund within 30 days of purchase from the Billing page.",
    "After 30 days, annual plans are refunded pro rata for the unused months; monthly plans are not refundable.",
    "To reset your password, click 'Forgot password' on the sign-in page and follow the link we email you.",
    "Two-factor authentication can be turned on under Settings > Security.",
    "Backups run every night at 02:00 in your account's time zone.",
    "You can restore a single file from any backup snapshot taken in the last 90 days.",
    "Our office in Lisbon is closed on public holidays.",
    "Invoices are emailed on the first day of each billing cycle and can be downloaded as PDF.",
    "The desktop app supports Windows 11 and macOS 14 or later.",
    "Deleted accounts are kept for 14 days and then permanently erased.",
]
labeled = {  # question -> index of the chunk that answers it
    "I forgot my login": 2,
    "can I get my money back after two months?": 1,
    "how do I get back a file I deleted yesterday": 5,
    "which operating systems are supported": 8,
    "when do backups happen": 4,
    "where can I find my receipt": 7,
}
models = {  # name -> (query prefix, passage prefix) from each model card
    "sentence-transformers/all-MiniLM-L6-v2": ("", ""),
    "intfloat/e5-small-v2": ("query: ", "passage: "),
}

for name, (qp, pp) in models.items():
    model = SentenceTransformer(name)
    C = model.encode([pp + c for c in chunks], normalize_embeddings=True)
    Q = model.encode([qp + q for q in labeled], normalize_embeddings=True)
    S = Q @ C.T                                   # cosine: 6 questions x 10 chunks
    relevant = np.zeros_like(S, dtype=bool)
    relevant[np.arange(len(labeled)), list(labeled.values())] = True
    print(name.split("/")[1], "| lowest score:", round(float(S.min()), 3))
    for t in [0.4, 0.5, 0.6, 0.7, 0.8, 0.85]:
        kept = S >= t
        precision = relevant[kept].mean() if kept.any() else float("nan")
        recall = relevant[kept].sum() / relevant.sum()
        empty = int((~kept.any(axis=1)).sum())
        print(f"  t={t:<4}: kept {int(kept.sum()):2d} pairs  precision {precision:.2f}"
              f"  recall {recall:.2f}  questions with nothing kept {empty}")

# all-MiniLM-L6-v2 | lowest score: -0.071
#   t=0.4 : kept  8 pairs  precision 0.75  recall 1.00  questions with nothing kept 0
#   t=0.5 : kept  5 pairs  precision 0.80  recall 0.67  questions with nothing kept 1
#   t=0.6 : kept  2 pairs  precision 1.00  recall 0.33  questions with nothing kept 4
#   t=0.7 : kept  0 pairs  precision nan  recall 0.00  questions with nothing kept 6
#   t=0.8 : kept  0 pairs  precision nan  recall 0.00  questions with nothing kept 6
#   t=0.85: kept  0 pairs  precision nan  recall 0.00  questions with nothing kept 6
# e5-small-v2 | lowest score: 0.691
#   t=0.4 : kept 60 pairs  precision 0.10  recall 1.00  questions with nothing kept 0
#   t=0.5 : kept 60 pairs  precision 0.10  recall 1.00  questions with nothing kept 0
#   t=0.6 : kept 60 pairs  precision 0.10  recall 1.00  questions with nothing kept 0
#   t=0.7 : kept 59 pairs  precision 0.10  recall 1.00  questions with nothing kept 0
#   t=0.8 : kept 19 pairs  precision 0.32  recall 1.00  questions with nothing kept 0
#   t=0.85: kept  2 pairs  precision 0.50  recall 0.17  questions with nothing kept 4

Six questions are far too few to choose a threshold, but they are enough to see the shape. Each model has a narrow band where the cutoff does something useful, and the two bands don't overlap. Notice one more thing: at 0.5, MiniLM's single wrong keep is the 30-day refund chunk for "money back after two months". The answer sits in the pro-rata chunk. Similarity is not relevance, and no metric fixes that (Semantic Search Principles).

Your turn: legal requires that the bot says "I don't know" rather than answer from an irrelevant chunk. Precision matters more than recall, but refusing half of the answerable questions is also unacceptable. How do you set the cutoff?

In brief: L1 and Hamming

L1 (Manhattan) distance is Σ |aᵢ − bᵢ|. Qdrant (Distance.MANHATTAN), pgvector (<+>) and FAISS (METRIC_L1) offer it. Unlike L2, it does not agree with cosine even on unit vectors, because it charges more when a difference is spread over many dimensions than when it is concentrated in a few:

import numpy as np

def unit(v):
    v = np.asarray(v, dtype=float)
    return v / np.linalg.norm(v)

q = unit([1, 0, 0, 0])
a = unit([0.8, 0.6, 0, 0])            # its off-axis part sits in one dimension
b = unit([0.85, 0.3, 0.3, 0.3])       # closer to q, off-axis part spread over three
for name, v in [("a", a), ("b", b)]:
    print(name, "cosine", round(float(q @ v), 3),
          " L2", round(float(np.linalg.norm(q - v)), 3),
          " L1", round(float(np.abs(q - v).sum()), 3))

# a cosine 0.8  L2 0.632  L1 0.8
# b cosine 0.853  L2 0.542  L1 1.05

Cosine and L2 prefer b, and L1 prefers a. Text embedding models are normally trained with cosine or dot product, so use L1 only when a model card asks for it.

Hamming distance counts the positions where two bit strings differ: the popcount of their XOR. It appears when vectors are binary-quantized to 1 bit per dimension, which makes them 32× smaller than float32. pgvector's binary_quantize keeps each coordinate's sign, so [0.3,-0.2,0.1,-0.5] becomes 1010.

import numpy as np

a = np.array([0.31, -0.20, 0.12, -0.52, 0.70, 0.05, -0.11, 0.40])
b = np.array([0.22, 0.14, 0.35, -0.41, 0.61, -0.18, -0.30, -0.09])
code_a, code_b = np.packbits(a > 0), np.packbits(b > 0)   # 8 dims -> 1 byte each
print(f"codes: {code_a[0]:08b} {code_b[0]:08b}")
hamming = int(np.unpackbits(code_a ^ code_b).sum())        # popcount of XOR
print("Hamming:", hamming, "of", a.size, "bits")
print("cosine of the float vectors:", round(float(a @ b / np.linalg.norm(a) / np.linalg.norm(b)), 3))

# codes: 10101101 11101000
# Hamming: 3 of 8 bits
# cosine of the float vectors: 0.735

Why it works at all: for bits taken from random hyperplanes, the chance that two vectors get different bits is θ/π, where θ is the angle between them (Charikar, STOC 2002). So the Hamming distance estimates the angle. Sign-of-coordinate codes only approximate this, and 8 bits is a very coarse estimate. Here θ/π = 0.237, which predicts about 1.9 differing bits, and we got 3. At full size the estimate is still rough. For SciFact's 384-bit all-MiniLM-L6-v2 codes, the Hamming top 10 contained 56.6% of the exact cosine top 10 (an ANN recall@10 of 0.566, which measures agreement with exact search, not relevance). Taking the Hamming top 100 and rescoring them with the full vectors raised that to 0.945. That rescoring pattern, and where the full vectors live, is covered in ANN Algorithms and Vector Database Architecture.

Final round: no label on the problem

Real incidents don't say which move they need. Read the setup, name the move, then check.

1. A prototype moves from a Qdrant Cosine collection to a FAISS IndexFlatL2 over the same unit vectors. The relevance filter D >= 0.75 was copied along, and now the prompt fills with the least related chunks.

2. After an index rebuild, recall@10 on your 200 labeled queries falls from 0.71 to 0.66. The model and queries are unchanged. The new collection says Distance.COSINE, the model card says "normalized: No, dot-product", and vectors read back with length 1.0.

3. p95 search latency on a 3-million-row pgvector table jumps from single-digit milliseconds to most of a second after a refactor. The results look fine. The new query ends with ORDER BY similarity DESC LIMIT 10, where similarity is 1 - (embedding <=> ...).

4. A nightly job deletes near-duplicate chunks: for any pair with cosine ≥ 0.9, it keeps the newer one. It ran for a year. After a model switch it deletes 30% of the corpus, including chunks about different products. The new vectors are unit length and correctly prefixed.

Cheat sheet: symptom → move → cost

How would you measure it on your own data?

• Labels first. Collect real queries with the ids of their relevant chunks. 50–200 are enough to compare two metrics. Tuning a refusal cutoff needs a few hundred, including queries the corpus cannot answer, because each error rate is measured on only part of the set.
• Norms. Look at the vector lengths of 1,000 stored chunks. All 1.0 means the metric question is settled; anything else means compare metrics.
• Metric comparison. For an unnormalized model, compute retrieval recall@10 and nDCG@10 under the card's metric and the alternative, using exact search so that the index's approximation doesn't blur the comparison.
• Thresholds. Sweep the cutoff. Record precision, recall and the rate of empty results, then choose by what each error costs.
• Latency. Run EXPLAIN (pgvector) or check p95 on a replayed query log whenever the query shape or metric changes.
• Drift. Chart the daily median top-1 score. It should only move when you change something on purpose.

Before moving on, take one incident from the final round and explain it aloud from a blank page: which number the engine returned, which direction was better, what the model was trained for, and how you would prove the fix on labeled data.

Next: Vector Database Architecture for what an engine adds around these computations, ANN Algorithms for searching millions of vectors without comparing against all of them, and Qdrant for putting a metric into a real collection.