Vector Embeddings

The refund clause that wasn't there

A customer types "Can I still get a refund after 30 days?" into your shop's help search. The answer exists. Clause 9 of your terms of service says late returns get store credit for up to 90 days. You embedded every document with all-MiniLM-L6-v2, a popular small open model, and the search returns the closest vectors. Here is the setup and what came back:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

clauses = [
    "1. Accounts. You must be 18 or older to open an account. Keep your password private; you are responsible for all activity under your login. Tell us at once if you think someone else has used your account.",
    "2. Orders. An order is accepted when we send a confirmation email, not when you click the order button. We may cancel an order that contains an obvious pricing error, and we will tell you if we do.",
    "3. Prices and payment. Prices include VAT for deliveries inside the EU. Delivery charges are shown at checkout before you pay. We accept major cards and bank transfers; we charge your card when the order ships.",
    "4. Delivery. We ship to the address you give us. Risk passes to you when the parcel is delivered. If a parcel is lost in transit, contact us within 14 days and we will send a replacement or cancel the order.",
    "5. Product information. Colours on screen may differ slightly from the real product. Sizes and materials are listed in each product description, and we update them when a supplier changes a product.",
    "6. Reviews. Reviews you post must be honest and must not contain personal data about other people. We may remove reviews that break these rules, and we do not pay for reviews.",
    "7. Warranty. Electronics carry a two-year warranty against manufacturing defects. The warranty does not cover damage caused by accidents, misuse or repairs by anyone other than us or our partners.",
    "8. Returns. You may return unused items within 30 days of delivery for a full refund to your original payment method.",
    "9. Late returns. After 30 days we do not refund your card, but we issue store credit for the full value of an unused item for up to 90 days after delivery.",
    "10. Liability. Nothing in these terms limits liability for death or personal injury caused by negligence.",
]
terms = " ".join(clauses)
faq = [
    "Store credit is added to your account and can be used on any future order.",
    "Gift cards can be redeemed at checkout and never expire.",
    "Track your parcel with the link in your shipping confirmation email.",
]

n_tokens = len(model.tokenizer(terms)["input_ids"])
print(f"terms: {len(terms.split())} words, {n_tokens} tokens; the model reads {model.max_seq_length}")

query = "Can I still get a refund after 30 days?"
q = model.encode_query(query)
docs = [terms] + faq
for score, doc in sorted(zip(model.encode_document(docs) @ q, docs), reverse=True):
    print(f"{score:.3f}  {doc[:45]}")
print(f"{model.encode_document(clauses[8]) @ q:.3f}  <- clause 9 on its own")

terms: 316 words, 378 tokens; the model reads 256
0.337  Store credit is added to your account and can
0.285  Gift cards can be redeemed at checkout and ne
0.232  1. Accounts. You must be 18 or older to open
0.224  Track your parcel with the link in your shipp
0.582  <- clause 9 on its own

The terms document, which holds the answer, ranks third, below a gift-card FAQ. Clause 9 on its own would have scored 0.582, far above everything else. The cause is the model's input limit: it reads at most 256 word pieces (tokens), and the terms are 378. sentence-transformers cuts the rest off without any warning. The only warning you see when you run this comes from our own token count: transformers complains when you tokenize a long text by hand, but encode stays silent.

Predict first: delete clauses 8, 9 and 10 from the document and embed it again. How much does the vector change?

without_8_to_10 = " ".join(clauses[:7])
print(f"{model.encode_document(terms) @ model.encode_document(without_8_to_10):.4f}")

1.0000

That is the pattern of this whole lesson. An embedding model is a function with a contract: how much text it reads, what it outputs, which similarity it was trained for, and what it expects in front of a query. Most embedding bugs are contract violations that fail silently. Chunking would fix this particular one. The deeper skill is reading the contract before you index ten million documents.

Read the contract first

Answer five questions for any embedding model before you write the indexing job:

The first half of the lesson shows how we got from word IDs to 2026 embedding models, because each step explains one line of that table. The second half uses the table.

Some topics live in other lessons. Why distance can stand for meaning at all, and why queries and passages are treated differently, is covered in Semantic Search Principles. The metrics themselves are in Cosine Similarity & Distance Metrics. The code uses Python 3.12 with numpy and sentence-transformers 6.1 (which installs transformers and PyTorch), plus qdrant-client in Move 6. Model downloads range from about 90 MB to 1.2 GB, and the last digits of the scores may differ slightly on your hardware.

Move 1: Give words neighbors

Smell: every pair of different words looks equally unrelated.

The oldest way to turn words into numbers gives each vocabulary word its own dimension. The result is a one-hot vector with a single 1 in it:

import numpy as np

vocab = ["refund", "reimbursement", "shipping", "password"]
one_hot = np.eye(len(vocab))        # row i = the vector for vocab[i]
print(one_hot @ one_hot.T)          # all pairwise dot products (= cosines here)

[[1. 0. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 0. 1.]]

Every entry off the diagonal is 0. "refund" is exactly as far from "reimbursement" as from "password", because the representation stores identity and nothing else. A bag-of-words or TF-IDF document vector is a weighted sum of these one-hot vectors, which is why pure keyword matching cannot see a synonym. It does see an exact identifier perfectly, and that is why lexical search stays in the toolbox (Sparse vs Dense Retrieval).

Learn the coordinates from context

word2vec (Mikolov et al., 2013) replaced the one-hot identity with a short dense vector, typically 100 to 300 numbers, learned from a prediction task. Skip-gram predicts the words around a given word; CBOW predicts a word from the words around it. GloVe (Pennington, Socher & Manning, 2014) reaches similar vectors by a different route: it fits them to global counts of how often words occur together within a window. Both rely on the distributional idea that words used in similar contexts get similar vectors. Both produce the same kind of artifact: a lookup table with one fixed vector per vocabulary word.

To embed a sentence with a lookup table, the classic trick is to average its word vectors. sentence-transformers ships exactly that as a model, built on GloVe 6B (trained on Wikipedia and Gigaword: 400,000 lowercase words, 300 dimensions), so you can poke at it with the same API as everything else:

import numpy as np
from sentence_transformers import SentenceTransformer

glove = SentenceTransformer("sentence-transformers/average_word_embeddings_glove.6B.300d")  # about 480 MB

def cos(a, b):
    va, vb = glove.encode([a, b])
    return float(va @ vb / (np.linalg.norm(va) * np.linalg.norm(vb)))

for a, b in [("refund", "reimbursement"), ("car", "automobile"), ("hot", "cold"), ("dog", "democracy")]:
    print(f"{a:>6} / {b:<13} {cos(a, b):.3f}")

refund / reimbursement 0.539
   car / automobile    0.596
   hot / cold          0.509
   dog / democracy     0.067

Neighbors exist now: refund and reimbursement score 0.539, dog and democracy 0.067. Notice hot and cold at 0.509. Opposites occur in the same contexts ("the water is too ___"), so a distributional model puts them close together. "Similar" here means "used alike", not "means the same".

Predict first: what does the same model give for "dog bites man" against "man bites dog"? And what is the vector for the part number "XK-4471"?

print(f"{cos('dog bites man', 'man bites dog'):.3f}")
print(np.linalg.norm(glove.encode("XK-4471")))

1.000
0.0

The famous analogy, measured

words = glove[0].tokenizer.vocab                  # index -> word
ix = glove[0].tokenizer.word2idx                  # word -> index
W = glove[0].emb_layer.weight.detach().cpu().numpy()
W = W / np.maximum(np.linalg.norm(W, axis=1, keepdims=True), 1e-9)   # unit rows

def nearest(v, k=4, skip=()):
    scores = W @ (v / np.linalg.norm(v))
    best = [i for i in np.argsort(-scores) if words[i] not in skip][:k]
    return [(words[i], round(float(scores[i]), 2)) for i in best]

v = W[ix["king"]] - W[ix["man"]] + W[ix["woman"]]
print(nearest(v))
print(nearest(v, skip={"king", "man", "woman"}))
print(nearest(W[ix["bank"]], k=6, skip={"bank"}))

[('king', 0.76), ('queen', 0.67), ('princess', 0.54), ('throne', 0.54)]
[('queen', 0.67), ('princess', 0.54), ('throne', 0.54), ('monarch', 0.53)]
[('banks', 0.7), ('banking', 0.6), ('central', 0.54), ('credit', 0.53), ('bankers', 0.52), ('financial', 0.5)]

The idea behind king − man + woman is that woman − man is a direction for gender, so adding it to "king" should land on a female royal. The nearest vector is king itself. "queen" wins only after you exclude the three input words, which is what the standard analogy tools do (gensim's most_similar, for example, never returns an input word). The offset is real but approximate. For a critical look at what analogies do and do not show, see Nissim, van Noord & van der Goot (2020).

The last line matters more for search. "bank" has one vector, and all of its nearest neighbors are about money. A question about a river bank gets the financial vector anyway.

Static vectors leave three silent gaps: one sense per word, no word order, and nothing at all for words outside the vocabulary.

💡 Static vectors are back in a new form. In January 2025 Hugging Face released static embedding models for sentence-transformers: still a token lookup table with mean pooling, but trained on retrieval pairs with the contrastive loss you will meet in Move 3. The authors report CPU inference 100 to 400 times faster than common transformer embedders, while reaching at least 85% of their benchmark performance. Word order and context are still gone, so test them on your own queries.

Move 2: Let the sentence decide what a word means

Smell: a word gets the same vector whatever surrounds it.

A transformer encoder such as BERT (Devlin et al., 2019) does not look words up. It computes one output vector per token, and self-attention lets each token's vector depend on every other token in the input. Ask it for the vector of "bank" in four sentences:

import numpy as np, torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
bert = AutoModel.from_pretrained("bert-base-uncased").eval()

sentences = [
    "We sat on the river bank and fished.",
    "Fish swim near the muddy bank of the stream.",
    "The bank approved my loan.",
    "I deposited cash at the bank.",
]
vectors = []
for s in sentences:
    batch = tok(s, return_tensors="pt")
    with torch.no_grad():
        hidden = bert(**batch).last_hidden_state[0]      # one vector per token
    i = tok.convert_ids_to_tokens(batch["input_ids"][0]).index("bank")
    vectors.append(hidden[i].numpy() / np.linalg.norm(hidden[i].numpy()))
V = np.array(vectors)
print(np.round(V @ V.T, 2))

Predict first: which two "bank" vectors will be most similar, and which two least?

[[1.   0.8  0.44 0.52]
 [0.8  1.   0.41 0.45]
 [0.44 0.41 1.   0.82]
 [0.52 0.45 0.82 1.  ]]

Subwords: nothing is out of vocabulary, but…

Transformer tokenizers split rare strings into known pieces, so the part number no longer vanishes:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
print(tok.tokenize("XK-4471"))
print(tok.tokenize("Hesabımı silersem dosyalarım ne olur?"))   # Turkish: "If I delete my account, what happens to my files?"

['x', '##k', '-', '44', '##7', '##1']
['he', '##sa', '##b', '##ı', '##m', '##ı', 'si', '##lers', '##em', 'dos', '##yala', '##r', '##ı', '##m', 'ne', 'ol', '##ur', '?']

Two consequences you will meet again:

• An identifier becomes six generic pieces. The model can represent it, but "XK-4417" becomes x, ##k, -, 441, ##7, sharing four pieces with "XK-4471", and nothing in training forced the two vectors apart. Exact identifiers stay a job for lexical or learned sparse retrieval (Hybrid Retrieval Systems).
• This English tokenizer shreds a five-word Turkish question into 18 fragments, which is a sign the model barely knows the language. Token limits are counted in each model's own tokens, so "256 tokens" is a different amount of text for every model and language. For the 316-word terms document above, the count (including special tokens) was 378 for all-MiniLM-L6-v2, 395 for multilingual-e5-small, 388 for Qwen3-Embedding-0.6B and 381 for cl100k_base, the tokenizer of OpenAI's text-embedding-3 models.

A contextual encoder gives you token vectors, 378 of them for the terms document. It does not give you one vector per passage, and simply averaging raw BERT token vectors works badly for search: the Sentence-BERT paper found that averaged BERT embeddings scored below averaged GloVe vectors on semantic similarity benchmarks. The encoder has to be trained for the job.

Move 3: One vector per passage

Smell: you have a vector per token, but you need one score per (query, passage) pair, for millions of passages, in milliseconds.

Encode separately, compare cheaply

A bi-encoder (dual encoder) runs the model on the query and on each passage separately, then compares the two resulting vectors. Passage vectors are computed once, offline. At query time you embed one short query and do vector arithmetic.

OFFLINE, once per passage:
  passage -> encoder -> pooling -> vector -> index --+
                                                     +--> similarity -> top-k
ONLINE, once per query:                              |
  query ---> encoder -> pooling -> vector -----------+

Same model and version on both sides; each side gets the prefix its card prescribes.

A bi-encoder has to squeeze each passage into one vector before it has seen any query, so its similarity is a coarse guess at relevance. The alternative feeds the query and the passage together through one model. This cross-encoder judges each pair more precisely, but it needs a forward pass for every pair and nothing can be precomputed. That is why it reranks a hundred candidates rather than searching a million, and why it can only reorder what the bi-encoder retrieved: a passage the first stage missed stays missed (Reranking Models). Sentence-BERT (Reimers & Gurevych, 2019) made BERT-quality bi-encoders practical. Its abstract puts the difference at about 65 hours with BERT as a cross-encoder, against about 5 seconds with SBERT, to find the most similar pair among 10,000 sentences.

Pooling: from many token vectors to one

Pooling turns the token vectors into a single vector. Each model fixes its choice at training time:

• Mean pooling averages the real tokens (all-MiniLM-L6-v2, multilingual-e5-small).
• CLS pooling takes the vector of a special first token (bge-small-en-v1.5, multi-qa-mpnet-base-dot-v1).
• Last-token pooling takes the final token's vector. Embedders built from decoder LLMs, such as Qwen3-Embedding, read left to right, so only the last token has seen the whole input.

Many models then add a Normalize step that scales the vector to length 1. You will print these pipelines in Move 4. The pooling is part of the contract, and if you re-implement it wrongly, the vectors change.

Predict first: the function below embeds text with all-MiniLM-L6-v2 by hand. It embeds "reset my password" once on its own and once in a batch next to a longer question. With the attention mask, the two vectors agree. Without the mask, what cosine do you expect?

import torch
from transformers import AutoTokenizer, AutoModel

name = "sentence-transformers/all-MiniLM-L6-v2"
tok = AutoTokenizer.from_pretrained(name)
encoder = AutoModel.from_pretrained(name).eval()

def embed(texts, use_mask=True):
    batch = tok(texts, padding=True, truncation=True, return_tensors="pt")
    with torch.no_grad():
        h = encoder(**batch).last_hidden_state                 # (texts, tokens, 384)
    if use_mask:
        m = batch["attention_mask"].unsqueeze(-1).float()     # 1 = real token, 0 = padding
        v = (h * m).sum(1) / m.sum(1)
    else:
        v = h.mean(1)                                         # BUG: padding counts as tokens
    return torch.nn.functional.normalize(v, dim=-1)

short = "reset my password"
long_text = "How long does a refund take to reach my credit card after the returned item arrives at your warehouse?"
alone = embed([short])[0]
for use_mask in (True, False):
    in_batch = embed([short, long_text], use_mask)[0]
    print(f"use_mask={use_mask}: cos(alone, in a batch) = {float(alone @ in_batch):.4f}")

use_mask=True: cos(alone, in a batch) = 1.0000
use_mask=False: cos(alone, in a batch) = 0.6376

Training: pull the true pair together, push the rest of the batch away

The original SBERT was fine-tuned on natural-language-inference sentence pairs with a classification objective. Current embedding models are trained contrastively on very large sets of (query, relevant passage) pairs. all-MiniLM-L6-v2's model card, for example, lists more than one billion training pairs. The standard loss (InfoNCE, called MultipleNegativesRankingLoss in sentence-transformers) treats every other passage in the batch as a negative example. For each query, the true passage must win a softmax over the whole batch.

What does this print? And which query contributes the most loss?

import numpy as np

S = np.array([          # cosines in one batch: row = query, column = passage, true pairs on the diagonal
    [0.62, 0.30, 0.15],   # "reset my password"          vs password / refund / store-credit passages
    [0.25, 0.55, 0.48],   # "money back after return"     (the store-credit passage is close)
    [0.10, 0.35, 0.64],   # "store credit for a late return?"
])
scale = 20.0                                   # sentence-transformers' default (temperature 0.05)
logits = scale * S
p = np.exp(logits) / np.exp(logits).sum(axis=1, keepdims=True)   # softmax per query
loss = -np.log(np.diag(p))                     # cross-entropy with the true pair as the label
print("p(true pair):", np.round(np.diag(p), 3))
print("loss per query:", np.round(loss, 3), " mean:", round(float(loss.mean()), 3))

p(true pair): [0.998 0.801 0.997]
loss per query: [0.002 0.222 0.003]  mean: 0.076

Why it works

The loss rewards exactly one thing: under the chosen similarity function, the true pair must outscore everything else in the batch. The space you search in is whatever geometry achieved that. So the similarity function used in training, the pooling and the input format are all part of the model. The temperature matters too. multilingual-e5 was trained with a temperature of 0.01, and its card warns that its cosine scores cluster between about 0.7 and 1.0. The model learned relative order; absolute scores and any threshold on them are model-specific (Cosine Similarity & Distance Metrics).

⚠️ Vectors from one model only mean something relative to each other. A vector from a different model lives in a different learned geometry, even if it has the same number of dimensions. Move 8 measures the damage.

Move 4: Read the model card

Smell: you are about to choose a model because someone said it is the best one.

Every row of the contract table can be read from the model object or its card in a few lines:

import numpy as np
from sentence_transformers import SentenceTransformer

def contract(name):
    m = SentenceTransformer(name)
    pooling = next(mod for mod in m if type(mod).__name__ == "Pooling")
    v = m.encode(["short text", "a longer text about refunds, returns and store credit " * 4])
    print(name.split("/")[-1])
    print("  modules:", " -> ".join(type(mod).__name__ for mod in m), f"(pooling: {pooling.pooling_mode})")
    print("  dim:", m.get_embedding_dimension(), "| max tokens:", m.max_seq_length,
          "| similarity:", m.similarity_fn_name, "| norms:", np.round(np.linalg.norm(v, axis=1), 2))
    print("  prompts:", m.prompts)

for name in ["sentence-transformers/all-MiniLM-L6-v2",
             "sentence-transformers/multi-qa-mpnet-base-dot-v1",
             "intfloat/multilingual-e5-small",
             "Qwen/Qwen3-Embedding-0.6B"]:
    contract(name)

all-MiniLM-L6-v2
  modules: Transformer -> Pooling -> Normalize (pooling: mean)
  dim: 384 | max tokens: 256 | similarity: cosine | norms: [1. 1.]
  prompts: {'query': '', 'document': ''}
multi-qa-mpnet-base-dot-v1
  modules: Transformer -> Pooling (pooling: cls)
  dim: 768 | max tokens: 512 | similarity: dot | norms: [5.94 5.85]
  prompts: {'query': '', 'document': ''}
multilingual-e5-small
  modules: Transformer -> Pooling -> Normalize (pooling: mean)
  dim: 384 | max tokens: 512 | similarity: cosine | norms: [1. 1.]
  prompts: {'query': '', 'document': ''}
Qwen3-Embedding-0.6B
  modules: Transformer -> Pooling -> Normalize (pooling: lasttoken)
  dim: 1024 | max tokens: 32768 | similarity: cosine | norms: [1. 1.]
  prompts: {'query': 'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:', 'document': ''}

Read it like a checklist:

• all-MiniLM-L6-v2: mean pooling, normalized, cosine, 256 tokens, no prompts.
• multi-qa-mpnet-base-dot-v1: CLS pooling, no Normalize step, norms near 6, similarity dot. It was trained for unnormalized dot product, so use it that way. Normalizing its vectors, or indexing them under cosine, changes the ranking it was trained to produce.
• multilingual-e5-small: 512 tokens, but prompts is empty, even though its card says every input must start with "query: " or "passage: " ("otherwise you will see a performance degradation"). The library can only apply prompts that the model's configuration declares.
• Qwen3-Embedding-0.6B: last-token pooling, 32,768 tokens, and a declared query instruction, which encode_query adds for you.

The third bullet is a trap worth seeing once:

import numpy as np
from sentence_transformers import SentenceTransformer

e5 = SentenceTransformer("intfloat/multilingual-e5-small")
a = e5.encode_query("reset password")                  # adds only what model.prompts declares: nothing
b = e5.encode("reset password", prompt="query: ")      # what the model card asks for
print(f"{float(a @ b):.3f}")

0.962

encode_query added nothing, so the "query" vector is the unprefixed one: a slightly different vector (cosine 0.962 with the correct one), and no error. For e5 models, pass prompt="query: " and prompt="passage: " yourself. For any model, compare model.prompts with the card.

The 2026 landscape as a table of contracts

The models below are examples as of 2026-09-25, checked against their own documentation. They are not a ranking.

What each column means for your decision:

API or open weights. An API needs no hardware and bills per token. The vendor controls versions and retirement, and a retired model forces a full re-index (Move 8). Your text leaves your network, which is a policy and contract question (a data processing agreement, retention terms, a business associate agreement for US health data) rather than a technical impossibility. Ask whoever owns compliance. With open weights you pin an exact model revision, the text stays in-house, and you pay in GPU or CPU time and in operations work.

Languages. An English-only model has an English tokenizer (the 18-fragment Turkish question) and never learned to align other languages with English. Multilingual models learn that alignment on purpose. The first training stage of multilingual-e5, for example, mixed 160 million translation pairs sampled from NLLB into about a billion training pairs (Wang et al., 2024), so a Spanish question can land near an English answer. Move 5 shows the difference on real queries.

Prefixes and instructions. Many 2024–2026 models are deliberately asymmetric: a query gets one prefix or task instruction, and a document gets another or none. Whatever you used when you built the index becomes part of that index's contract (Move 8).

Max input length: truncation versus dilution. Providers disagree about what happens past the limit. sentence-transformers truncates silently. OpenAI rejects the request. Cohere cuts the end unless you set truncate to NONE, which returns an error instead. Voyage truncates by default. Long-context models (8k to 128k tokens) remove the truncation, but not the dilution. Continuing the refund example with Qwen3-Embedding-0.6B, which reads up to 32,768 tokens:

qwen = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")      # about 1.2 GB download
qv = qwen.encode_query(query)                                 # adds its retrieval instruction
for score, doc in sorted(zip(qwen.encode_document(docs) @ qv, docs), reverse=True):
    print(f"{score:.3f}  {doc[:45]}")
print(f"{qwen.encode_document(clauses[8]) @ qv:.3f}  <- clause 9 on its own")
print(f"cos(terms, terms without 8-10) = {qwen.encode_document(terms) @ qwen.encode_document(without_8_to_10):.3f}")

0.461  1. Accounts. You must be 18 or older to open
0.422  Gift cards can be redeemed at checkout and ne
0.300  Track your parcel with the link in your shipp
0.267  Store credit is added to your account and can
0.710  <- clause 9 on its own
cos(terms, terms without 8-10) = 0.951

The terms document now ranks first, because the model read all of it. But clause 9 on its own still scores 0.710 against the whole document's 0.461, and deleting clauses 8 to 10 now moves the vector only a little (cosine 0.951). One vector for ten clauses is a blend of ten topics. Long-context embedders suit documents that are about one thing. For precise retrieval inside long documents you still chunk, and then you return the chunk rather than the whole document.

Normalization. Check the norms, which takes one line. If the model normalizes, cosine and dot product rank identically. If it doesn't, as with multi-qa-mpnet-base-dot-v1, use the similarity named on its card. Truncating vectors yourself breaks unit length (Move 6). APIs that shorten vectors for you differ: OpenAI asks you to normalize only when you shorten vectors yourself rather than through dimensions, gemini-embedding-2 re-normalizes shortened outputs, while for gemini-embedding-001 Google tells you to normalize any size other than 3,072 yourself. Check the norms of what you actually store; normalizing twice is harmless.

Multimodal, in brief. CLIP (Radford et al., 2021) trained an image encoder and a text encoder contrastively on 400 million image–caption pairs, so a photo and its description land near each other. The same idea now ships in retrieval APIs: gemini-embedding-2 and Cohere embed-v4.0 put images and PDFs (and, for Gemini, audio and video) into the same space as text. The contract rules don't change: one space per model, a maximum input for each kind of input, and measurement on your own content.

Vectors are not anonymized text. Morris et al. (2023) recovered 92% of 32-token inputs exactly from their embeddings, using an iterative inversion method. Protect a vector store as carefully as the documents it came from (Infrastructure & Security).

Your turn: a colleague wants to index 5 million support tickets written in English and Portuguese with EmbeddingGemma at 256 dimensions. Using the table above, write the contract lines the indexing code must enforce, and the ones you must test rather than assume.

Move 5: Choose with your own queries, not a leaderboard

Smell: the model choice came from a leaderboard screenshot.

MTEB (Muennighoff et al., 2023) and its multilingual successor MMTEB (Enevoldsen et al., 2025) are the standard public benchmarks, with a public leaderboard. Use them to build a shortlist. Don't let them make the decision. The scores average many tasks that are not yours, results are submitted by the model authors, and a model trained on data close to a benchmark's tasks can score higher there than it will on your data. The deciding test is small, cheap and yours: real queries, each labeled with the passage that answers it.

Here is one for a help center: 14 passages and 12 queries. The queries include paraphrases, an error code, and questions in German, Spanish and Turkish, because this shop's logs say some customers write in those languages. The code compares three models, with and without the prompts their cards ask for:

import numpy as np
from sentence_transformers import SentenceTransformer

passages = [
    "To reset your password, click 'Forgot password' on the sign-in page. We email you a link that expires after 30 minutes.",
    "Lost your phone? Use one of the backup codes you saved when you turned on two-factor authentication, or contact support to verify your identity.",
    "You can change the email address on your account under Settings > Profile. We send a confirmation to both the old and the new address.",
    "Refunds go back to the original payment method. Card refunds usually appear within 5 to 10 business days after we receive the returned item.",
    "Returns after 30 days are not refunded to your card; we issue store credit for the item's value instead.",
    "If you see the same order charged twice, the second charge is usually a pending authorization that your bank releases within a few days.",
    "You can cancel your subscription at any time under Settings > Billing. Your plan stays active until the end of the current billing period.",
    "Instead of cancelling, you can pause your subscription for up to three months. You will not be billed while it is paused.",
    "Error E-4471 means the sync service could not reach your storage bucket. Check that the bucket still exists and that the access key has not expired.",
    "Error E-4417 appears when two devices edit the same file at once. The later save is kept as a conflicted copy.",
    "Standard shipping takes 3 to 5 business days within the EU. Express orders placed before 14:00 ship the same day.",
    "Orders shipped outside the EU may be charged import duties and taxes by the destination country's customs office.",
    "To export your data, go to Settings > Privacy and request an archive. We email a download link when it is ready.",
    "Deleting your account permanently removes your files after 14 days. Download an export first if you want to keep a copy.",
]
labeled = [                      # (query, index of the passage that answers it)
    ("I forgot my login and can't get in", 0),
    ("got a new phone and my authenticator codes are gone", 1),
    ("how long until I see my money back on my card", 3),
    ("I sent it back after six weeks, do I get cash back?", 4),
    ("why was I billed two times for one purchase", 5),
    ("can I take a break from paying without losing my account", 7),
    ("what does E-4471 mean", 8),
    ("Wie kann ich mein Passwort zurücksetzen?", 0),          # German
    ("¿Cuánto tarda el envío estándar?", 10),                 # Spanish
    ("Hesabımı silersem dosyalarım ne olur?", 13),            # Turkish
    ("do I pay extra tax if you ship to Norway", 11),
    ("download a copy of everything I stored", 12),
]

def ranks(model, query_prompt="", doc_prompt="", dim=None):
    """Rank (1 = top) of the correct passage for every labeled query."""
    P = model.encode([doc_prompt + p for p in passages], truncate_dim=dim, normalize_embeddings=True)
    Q = model.encode([query_prompt + q for q, _ in labeled], truncate_dim=dim, normalize_embeddings=True)
    S = Q @ P.T
    return np.array([1 + int(np.sum(S[i] > S[i, rel])) for i, (_, rel) in enumerate(labeled)])

qwen_instruction = "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
candidates = [
    ("sentence-transformers/all-MiniLM-L6-v2", "", ""),
    ("intfloat/multilingual-e5-small", "", ""),
    ("intfloat/multilingual-e5-small", "query: ", "passage: "),
    ("Qwen/Qwen3-Embedding-0.6B", "", ""),
    ("Qwen/Qwen3-Embedding-0.6B", qwen_instruction, ""),
]
models = {}
for name, qp, dp in candidates:
    if name not in models:
        models[name] = SentenceTransformer(name)
    r = ranks(models[name], qp, dp)
    print(f"{name.split('/')[-1]:<22} {'prompted' if qp else 'raw':<8} hit@1={np.mean(r == 1):.2f} "
          f"recall@3={np.mean(r <= 3):.2f} MRR={np.mean(1 / r):.3f}  ranks={r.tolist()}")

The metrics: hit@1 is the share of queries whose answer ranks first. recall@3 is the share with the answer somewhere in the top 3; each query here has exactly one relevant passage, so this is ordinary retrieval recall at 3. MRR (mean reciprocal rank) is the average of 1 ÷ rank of the answer. Retrieval Metrics covers them properly.

Predict first: which of the 12 queries will the English-only all-MiniLM-L6-v2 get wrong?

all-MiniLM-L6-v2       raw      hit@1=0.83 recall@3=0.83 MRR=0.850  ranks=[1, 1, 1, 1, 1, 1, 1, 1, 14, 8, 1, 1]
multilingual-e5-small  raw      hit@1=0.83 recall@3=1.00 MRR=0.917  ranks=[1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 2]
multilingual-e5-small  prompted hit@1=0.83 recall@3=1.00 MRR=0.917  ranks=[2, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1]
Qwen3-Embedding-0.6B   raw      hit@1=0.75 recall@3=1.00 MRR=0.875  ranks=[2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2]
Qwen3-Embedding-0.6B   prompted hit@1=0.83 recall@3=1.00 MRR=0.917  ranks=[2, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1]

To make this test decide real choices:

• Size. Use 50 to 200 queries, taken from real logs where possible and labeled by someone who knows the content. Label every relevant passage, not just one.
• Slices. Report each slice separately: each language, identifiers, paraphrases, negation, long documents. A 0.02 overall gain can hide a slice that broke.
• The production pipeline. Use the same chunking, prompts and similarity as production. A test that embeds differently from the index measures something else.
• Cost and latency next to quality. Parameter count is a rough guide (23M for all-MiniLM-L6-v2, 118M for multilingual-e5-small, 596M for Qwen3-Embedding-0.6B), but measure the p95 query-embedding latency on your hardware and the cost per 1,000 queries.
• Keep it. The same set gates every later change of model, chunking or prompt (Move 8).

Your turn: the requirements change. 70% of users now write in Turkish, query embedding must take under 20 ms at p95 on a 4-core CPU, and customer text may not leave your network. What do you do next?

Move 6: Shrink on purpose, never by accident

Smell: the vectors don't fit in memory, or storage is the biggest line on the bill.

Raw vectors cost n × dim × 4 bytes in float32. Ten million chunks at 1,024 dimensions take 10,000,000 × 1,024 × 4 B = 40.96 GB, before any index structure.

Matryoshka Representation Learning (Kusupati et al., 2022) trains the model so that the first 64, 128, 256 … numbers of the vector are themselves a usable embedding. The training loss is computed at several prefix lengths at once; sentence-transformers calls this MatryoshkaLoss. You can then keep a prefix and throw away the rest. It is a training-time property. OpenAI's dimensions, Gemini's output_dimensionality, Cohere's and Voyage's output_dimension, Qwen3-Embedding (32 to 1,024) and EmbeddingGemma (512, 256, 128) all expose it. Cutting the end off a model that was not trained this way just deletes information.

Continuing with the labeled test:

qwen = models["Qwen/Qwen3-Embedding-0.6B"]
mini = models["sentence-transformers/all-MiniLM-L6-v2"]
for dim in (1024, 256, 64, 32):
    print(f"Qwen3-Embedding-0.6B {dim:>4} dims: MRR={np.mean(1 / ranks(qwen, qwen_instruction, '', dim)):.3f}")
for dim in (384, 96, 64, 32):
    print(f"all-MiniLM-L6-v2     {dim:>4} dims: MRR={np.mean(1 / ranks(mini, '', '', dim)):.3f}")
v = qwen.encode_document(passages[:3], truncate_dim=256)
print("norms after truncate_dim=256:", np.round(np.linalg.norm(v, axis=1), 2))

Qwen3-Embedding-0.6B 1024 dims: MRR=0.917
Qwen3-Embedding-0.6B  256 dims: MRR=0.917
Qwen3-Embedding-0.6B   64 dims: MRR=0.958
Qwen3-Embedding-0.6B   32 dims: MRR=0.958
all-MiniLM-L6-v2      384 dims: MRR=0.850
all-MiniLM-L6-v2       96 dims: MRR=0.719
all-MiniLM-L6-v2       64 dims: MRR=0.678
all-MiniLM-L6-v2       32 dims: MRR=0.499
norms after truncate_dim=256: [0.5  0.5  0.47]

• The Matryoshka-trained Qwen3 keeps its MRR all the way down to 32 dimensions on this set. The small rise at 64 is one query changing places, which is noise. all-MiniLM-L6-v2 was not trained for truncation, and it drops from 0.850 to 0.499 at 32 dimensions.
• Don't conclude that 32 dimensions are enough. Fourteen passages make an easy corpus. With millions of chunks, expect a gradual decline, and find the knee (the smallest size that still meets your recall target) on your labeled set.
• Look at the norms: 0.5, 0.5 and 0.47. Truncation keeps the first quarter of a unit vector, and that quarter is no longer unit length.

The re-normalization trap

For unit vectors, dot product and cosine give the same ranking. Truncated vectors are not unit vectors any more, so the metric you index with starts to matter. Store the truncated vectors in Qdrant twice, once under cosine and once under dot product:

from qdrant_client import QdrantClient, models as qm

P = qwen.encode_document(passages, truncate_dim=256)              # truncated, NOT re-normalized
Q = qwen.encode_query([q for q, _ in labeled], truncate_dim=256)
client = QdrantClient(":memory:")
top3 = {}
for dist in (qm.Distance.COSINE, qm.Distance.DOT):
    name = f"trunc_{dist.value.lower()}"
    client.create_collection(name, vectors_config=qm.VectorParams(size=256, distance=dist))
    client.upsert(name, points=[qm.PointStruct(id=i, vector=v.tolist()) for i, v in enumerate(P)])
    top3[dist] = [[p.id for p in client.query_points(name, query=q.tolist(), limit=3).points] for q in Q]
    stored = client.retrieve(name, ids=[0], with_vectors=True)[0].vector
    print(f"{dist.value}: stored norm {np.linalg.norm(stored):.2f}")
    client.delete_collection(name)
changed = sum(a != b for a, b in zip(top3[qm.Distance.COSINE], top3[qm.Distance.DOT]))
print(f"queries whose top-3 differs: {changed} of {len(Q)}")

Cosine: stored norm 1.00
Dot: stored norm 0.50
queries whose top-3 differs: 4 of 12

Qdrant's cosine distance normalizes vectors when they are inserted (and normalizes the query too), so that collection silently fixed the problem. The dot-product collection ranks by cosine × document norm. The query's own norm doesn't matter for the order, because it multiplies every score for that query by the same factor, but it does move all of those scores against any fixed threshold. The document norms vary from 0.47 to 0.53 in this corpus, so 4 of the 12 queries got a different top 3 from the one you would measure offline with cosine. For three of them only the order changed, one got a different passage in its top 3, and two got a different top result. On this tiny set the dot-product order even happened to score better (MRR 1.000 against 0.917). That is the real problem: production no longer ranks the way your evaluation measured, and you cannot predict in which direction it drifts. The rule: re-normalize after truncating, with normalize_embeddings=True or v / np.linalg.norm(v). After that, dot product, cosine and L2 all give the cosine ranking. The same code also runs against a Qdrant server; the Qdrant lesson covers the client properly.

Fewer bits instead of fewer dimensions

Matryoshka truncation removes dimensions. Quantization stores each dimension in fewer bits: int8 is 4 times smaller than float32 and binary is 32 times smaller, and search usually oversamples and rescores with the full vectors. The two techniques combine. Quantization belongs to the index (ANN Algorithms, Vector Database Architecture), although some tools can return quantized vectors directly (Cohere's embedding_types, the precision argument of sentence-transformers' encode).

Your turn: you have 10 million chunks, a Matryoshka model with 1,024 dimensions, and a 16 GB RAM budget for the float32 vectors plus the HNSW graph. Estimate the graph as 2·M neighbor ids of 4 bytes per vector, with M = 16. Which sizes fit? What must be true before you ship the one you pick?

Move 7: Batch it, then price it

Smell: the indexing job embeds one text at a time, or nobody knows what a re-index costs.

The encoder is fastest when it processes many texts in one forward pass. sentence-transformers' encode takes a list and a batch_size (default 32). It also sorts the inputs by length internally, so each batch needs as little padding as possible, and then restores your original order. Continuing with the help-center passages:

import time, random

random.seed(0)
texts = [" ".join(random.sample(passages, 2)) for _ in range(2000)]   # 2,000 short chunks
mini.encode(texts[:64])                                               # warm-up
t = time.perf_counter()
for text in texts[:200]:
    mini.encode(text)
print(f"one at a time:  {200 / (time.perf_counter() - t):.0f} texts/s")
for bs in (32, 128):
    t = time.perf_counter()
    mini.encode(texts, batch_size=bs)
    print(f"batch_size={bs:<4} {len(texts) / (time.perf_counter() - t):.0f} texts/s")

one at a time:  144 texts/s
batch_size=32   1667 texts/s
batch_size=128  3417 texts/s

That run used an Apple M2 Max laptop GPU while other jobs competed for it. A second run a few minutes later, under heavier load, printed 35, 1,313 and 1,875 texts per second. The absolute numbers swing with the machine, but batching won by a wide margin both times (about 24 and 54 times faster at a batch size of 128). Measure on your own hardware, and watch memory. Activation memory grows with batch size × sequence length, so a batch of long chunks can fail with an out-of-memory error where short ones succeeded. Start moderate and increase the batch size until throughput stops improving.

APIs batch too, with their own limits. OpenAI accepts up to 2,048 inputs and 300,000 tokens per request, Cohere 96 texts per call, and Voyage 1,000 texts. Rate limits for your account tier come on top of that. Production concerns such as retries, idempotent writes and re-embedding only changed documents belong to Embedding Pipeline.

Price it before you run it

Assume 2,000,000 chunks of 400 tokens on average and 3,000,000 queries a month of 20 tokens each, at OpenAI's September 2026 prices:

For a self-hosted model, the time is chunks ÷ throughput. At an assumed 500 chunks per second on one GPU, 2,000,000 chunks take 4,000 seconds, about 67 minutes. The bill is the GPU hours plus the engineering to run them.

What surprises most teams is that the embedding fee is often the smallest number in the project. Storage, and the memory for the index, recurs every month. A model change means paying for indexing again plus running two indexes side by side during the migration (Move 8). Track cost per 1,000 queries and p95 query-embedding latency next to your quality numbers.

Move 8: A new model means a new index

Smell: results got worse right after an upgrade, and nothing logged an error.

Keep the index built with all-MiniLM-L6-v2, and send it query vectors from other 384-dimensional models, the way a half-finished upgrade would:

index = mini.encode_document(passages)            # the index you built last year
queries = [q for q, _ in labeled]
suspects = [
    ("all-MiniLM-L6-v2 (same model)", mini, ""),
    ("bge-small-en-v1.5 (also 384-d)", SentenceTransformer("BAAI/bge-small-en-v1.5"), ""),
    ("multilingual-e5-small (also 384-d)", models["intfloat/multilingual-e5-small"], "query: "),
]
for label, encoder, prompt in suspects:
    S = encoder.encode_query(queries, prompt=prompt) @ index.T
    r = np.array([1 + int(np.sum(S[i] > S[i, rel])) for i, (_, rel) in enumerate(labeled)])
    print(f"{label:<36} MRR={np.mean(1 / r):.3f}  top-1 scores {S.max(1).min():.2f} to {S.max(1).max():.2f}")
print(f"{'random order':<36} MRR={np.mean(1 / np.arange(1, len(passages) + 1)):.3f}")
try:
    SentenceTransformer("sentence-transformers/all-mpnet-base-v2").encode_query(queries) @ index.T
except ValueError as err:
    print("768-d queries:", str(err).split(",")[0])

all-MiniLM-L6-v2 (same model)        MRR=0.850  top-1 scores 0.03 to 0.71
bge-small-en-v1.5 (also 384-d)       MRR=0.694  top-1 scores 0.06 to 0.31
multilingual-e5-small (also 384-d)   MRR=0.261  top-1 scores 0.01 to 0.08
random order                         MRR=0.232
768-d queries: matmul: Input operand 1 has a mismatch in its core dimension 0

• A different dimension fails loudly. numpy raises an error, and a vector database rejects the query (Qdrant answers a dimension mismatch with a 400 error). You find this one in minutes.
• The same dimension degrades quietly. The e5 queries score barely above random order, yet every query still returns a full list of results. The bge queries fail more subtly: they are partly wrong rather than all wrong, and an MRR of 0.694 looks almost healthy, so a quick spot check could pass.
• The scores give it away. The highest top-1 score fell from 0.71 to 0.31 or 0.08. Tracking the daily distribution of top-1 scores is a cheap alarm for "something changed in the embedding path".

Everything that counts as a new model

Each of these changes the vectors, and each needs the same treatment as swapping the model:

• a different model, or a new version or revision of the same model name (pin the revision for open weights, and watch deprecation notices for APIs);
• a different prefix or instruction on either side (Move 4);
• a different Matryoshka size, or normalization switched on or off (Move 6);
• a different chunker or text cleanup, at least for the documents it touches, because different input text means different vectors.

Vendors can document exceptions. Voyage states that all of its 4-series models share one embedding space, so documents embedded with voyage-4-large can be searched with voyage-4-lite queries. Google states the opposite for gemini-embedding-001 and gemini-embedding-2: both have 3,072 dimensions, and their spaces are incompatible. Rely only on compatibility that the vendor documents, and still run your labeled set.

The migration recipe

BEFORE        query --> model A --> index A

MIGRATION     query --> model A --> index A      (users still served by A)
              every document --> model B --> index B   (then catch up on edits and deletes)
              labeled set: A vs B, per slice

AFTER         query --> model B --> index B      (alias now points at B; A kept for rollback)

1. Record the contract with the index: model id and revision, dimensions, prefixes, normalization and chunker version, in the collection's metadata or in each point's payload.
2. Build the new index beside the old one. Re-embed everything with model B, then catch up before the switch: re-sync index B against the current source so that edits and deletions made during the re-embedding reach it. Sending new and changed documents to both indexes also works, but only if they go through the same per-document queue as the re-embedding; otherwise the re-embedding can overwrite a newer version or bring back a deleted document (Embedding Pipeline shows both at scale).
3. Evaluate B against A on the labeled set, slice by slice.
4. Switch the query encoder and the index together, atomically, for example by repointing a collection alias (Qdrant, Elasticsearch and OpenSearch all support aliases).
5. Keep index A until the rollback window has passed.

The cost is the full re-embedding plus double storage during the migration. For the 2-million-chunk example in Move 7, moving to text-embedding-3-large at full size means USD 104 in fees and another 24.58 GB of raw vectors while both indexes exist. Keeping indexes current over months is covered in Data Freshness & Lifecycle.

Your turn: next year your corpus is 10 times bigger (20 million chunks of 400 tokens), and a better model is released. A teammate proposes embedding only new documents with it, "to save cost". What do you answer?

Final round: no label on the symptom

Real incidents don't say which move they need. For each one, name the likely cause and the first thing you would check.

1. A manual's troubleshooting sections, which sit at the end of each chunk, never show up in results. Questions about the start of the same chunks work fine. Chunks are 1,000 tokens long, and the model's max_seq_length is 512.
2. After a deploy, top-1 scores dropped from around 0.6 to around 0.05 for every query, and the answers are unrelated to the questions. Nobody touched the index.
3. You switched to an e5 model, indexed with encode_document and queried with encode_query. Retrieval is noticeably worse than the model's reported results.
4. A team truncated its Matryoshka vectors from 1,024 to 256 dimensions to save memory and stored them in a dot-product collection. Their offline evaluation ranked the same truncated vectors by cosine similarity in numpy, and for some queries production now returns a different order than the evaluation did.
5. A new embedding model raised MRR on your 400-query labeled set from 0.81 to 0.83, so you switched. A week later, complaints from Turkish-speaking customers (60 of the 400 test queries are in Turkish) have doubled.

Cheat sheet: symptom → move → cost

Before moving on, pick one model you might really use and write its contract from memory: max tokens and what happens past them, dimensions and allowed truncations, similarity and norm, prompts on each side, languages. Then check your answer against the card and the output of the contract() function from Move 4. Finally, say how you would test it on your own data: which 50 queries, which slices, which metric, and which number would make you choose it.

Next: Cosine Similarity & Distance Metrics explains what the scores mean and how metrics interact with normalization. Vector Database Architecture stores and indexes these vectors. Sparse vs Dense Retrieval covers the identifiers and rare terms that embeddings handle badly.