Vector DB Selection

The tenant who got one answer

A B2B help-desk assistant keeps every customer's documents apart: each chunk carries a tenant_id, and every query filters on it. The team already runs Postgres, so they store the chunks there with pgvector (0.8.6 is the current release, checked 2026-09-25) and add an HNSW index with the default settings. The demo tenant is huge, and every answer looks great.

Then a user at Acme, a mid-sized customer, asks how to rotate the API key for their SSO integration. The app runs:

-- pgvector 0.8.6 on Postgres 18; HNSW index with defaults (m = 16, ef_construction = 64)
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

SELECT id
FROM chunks
WHERE tenant_id = 'acme'
ORDER BY embedding <=> $1      -- cosine distance to the query embedding
LIMIT 10;

What came back: one chunk, about SAML certificates. What should have come back: ten Acme chunks, led by Acme's "Rotating API keys" page. No error, no warning. The LLM answered from one loosely related chunk.

The story is a composite, but the numbers are not. We rebuilt the situation with pgvector 0.8.6: 60,000 synthetic 64-dimensional embeddings, Acme owning 1,759 of them (2.9%), and 200 queries.

Predict first: pgvector's HNSW scan keeps a list of 40 candidates by default (hnsw.ef_search = 40). If the WHERE clause is checked only on those candidates, how many rows do you expect per query?

The first fix is one setting, added in pgvector 0.8.0 and off by default: iterative index scans, which keep scanning more of the index when too few rows survive the filter. Same 200 queries:

ANN recall@10 is the overlap between what the index returned and the exact top 10 among Acme's chunks. For comparison, unfiltered queries on the same index scored 0.92. The timings come from a laptop running Postgres in Docker, so compare the settings with each other, not with your production numbers.

Iterative scans are not unlimited, though. The README says a scan ends when enough rows are found or when it has visited about hnsw.max_scan_tuples tuples (20,000 by default); a memory limit, hnsw.scan_mem_multiplier × work_mem, can end it too. Acme holds 2.9% of the rows, so the cap is no problem. A tenant whose rows are rarer than about 10 in 20,000 (0.05%) still comes back short, and for such tenants a partial index, a partition or exact search over the tenant's rows is the better tool. Also, relaxed_order may return rows slightly out of distance order; strict_order keeps the exact order.

So the database was not broken. The team had simply never asked, before choosing it, "what does this engine do when a filter matches 3% of the rows?" And nobody had measured it. Most painful vector-database stories look like this. The product did what its documentation said, and the requirement that mattered was never written down.

Picking a vector database is not a contest for "the best". It is a short list of requirements that actually decide, checked against current documentation, and then proved on your own data. Each requirement has a symptom you will meet if you skip it:

This lesson takes you through them in order. First you decide whether you need a new database at all. Then you walk the requirements with real numbers, profile the 2026 options with dated facts, and run a fair proof of concept. What a vector database is made of is covered in Vector Database Architecture. How HNSW, IVF and quantization work is covered in ANN Algorithms. Here you use those ideas to choose.

⚠️ Product facts rot quickly. Every version, default and limit below carries the date it was checked (2026-09-25) and a link. Re-check the ones your decision depends on.

Step zero: you may not need a new database

A new database is a new thing to deploy, secure, back up, monitor and keep in sync with your source of truth. It has to earn its place. Three cheaper answers come first.

Answer 1: a numpy array

Exact nearest-neighbour search is one matrix-vector product. It has perfect recall and no index to tune:

import time
import numpy as np

def exact_top_k(X, q, k=10):
    scores = X @ q                              # cosine: rows of X and q are unit length
    top = np.argpartition(-scores, k)[:k]
    return top[np.argsort(-scores[top])]

rng = np.random.default_rng(1)
dim = 768
for n in (100_000, 1_000_000):
    X = rng.standard_normal((n, dim), dtype=np.float32)
    X /= np.linalg.norm(X, axis=1, keepdims=True)
    q = X[0]
    times = []
    for _ in range(30):
        t0 = time.perf_counter()
        exact_top_k(X, q)
        times.append((time.perf_counter() - t0) * 1000)
    print(f"n={n:>9,}  vectors {X.nbytes / 1e9:.2f} GB  median {np.median(times):5.1f} ms/query")

# On an Apple M2 Max laptop shared with other jobs, one run printed:
# n=  100,000  vectors 0.31 GB  median   6.7 ms/query
# n=1,000,000  vectors 3.07 GB  median  56.1 ms/query

Other runs, while the laptop was busy with other jobs, were up to 4× slower, so measure on your own hardware. The cost is mostly reading memory: when the machine was quiet, roughly 18–25 ms per GB of float32 vectors scanned. The exercises below use about 20 ms per GB. At 100,000 chunks, exact search takes a few milliseconds, which is nothing next to an LLM call. At 1M chunks it no longer fits a 50 ms retrieval budget, though batch jobs will not mind. These are single-query latencies: batching many queries into one matrix multiply raises throughput a lot, which is why offline jobs can afford exact search at sizes an interactive service cannot.

What a bare array does not give you: durable writes (you can np.save a file, but that is not a write-ahead log), concurrent writers, one shared copy across many server processes, and fast filtering over millions of rows.

Answer 2: the database you already run

• Postgres → pgvector. Vectors sit next to your rows. You get transactions, joins, the backups and replicas you already have, and exact or approximate search in SQL.
• Elasticsearch or OpenSearch → vector fields. You already have BM25, analyzers and the operations know-how. Both engines add HNSW vector search to the same index (details in the landscape section).
• Several other general-purpose databases have added vector indexes too. Check what yours supports before you add a new system.

What you give up: vector-specific features can lag behind the dedicated engines, as the Acme story showed with filtering. Vector queries also compete with your transactional load on the same machines.

Answer 3: an embedded library or database

FAISS and hnswlib are indexing libraries: they run inside your process and can save and load index files. Chroma (with PersistentClient) and LanceDB are embedded databases that also keep documents and metadata on disk. Both kinds work well for a single process or a notebook. In embedded mode, none of them replicates your data or serves many application servers.

💡 Rule of thumb: a dedicated vector database has to win on a requirement that the cheaper options measurably fail. "We might need to scale" is a forecast, not a measurement.

Your turn: an internal tool has 250,000 chunks of 1,024 float32 dimensions, peaks at 3 queries per second, and has a 50 ms retrieval budget. Is exact search in numpy acceptable? What would push you to an index?

Requirement 1: scale and memory — do the arithmetic before the demo

Symptom: "It was fast in the demo." The demo had 50,000 vectors.

Take a legal-research product with 40 million chunks from a 1,024-dimensional model. Before you look at any product, work out what the vectors alone need:

def footprint_gb(n, dim, M=16):
    GB = 1e9
    return {
        "float32 vectors": n * dim * 4 / GB,
        "int8 vectors": n * dim * 1 / GB,
        "binary vectors": n * dim / 8 / GB,
        "HNSW links": n * 2 * M * 4 / GB,     # base layer: 2*M neighbour ids of 4 bytes; an estimate
    }

for part, gb in footprint_gb(40_000_000, 1024).items():
    print(f"{part:16} {gb:7.2f} GB")

# float32 vectors   163.84 GB
# int8 vectors       40.96 GB
# binary vectors      5.12 GB
# HNSW links          5.12 GB

Those numbers turn into three candidate layouts:

Add payload (text, metadata), engine overhead, and a full copy for every replica. Those are estimates to check against the engine's own sizing guide and a real load test. The mechanics of quantization and rescoring are in ANN Algorithms.

Questions to ask every candidate:

• Which parts must stay in RAM, and which can live on disk (memory-mapped) or in object storage?
• Which quantization does it support, and does it rescore with the original vectors automatically?
• How does it grow at 10×: a bigger box, more shards, or automatic scaling you don't see?
• What is the largest dimension it can index? pgvector stores vector values of up to 16,000 dimensions, but its HNSW and IVFFlat indexes accept up to 2,000 (halfvec up to 4,000, bit up to 64,000). An Elasticsearch dense_vector field allows up to 4,096 dimensions (pgvector README, Elasticsearch docs, both checked 2026-09-25). A 3,072-dimensional model therefore needs a workaround before pgvector can index it: index it at half precision (halfvec), index a binary-quantized copy (bit) and re-rank the candidates with the full vectors, or index a truncated (Matryoshka-style) embedding if the model supports truncation.

Requirement 2: filter selectivity — the one that fails silently

Symptom: broad queries look fine, but queries for a small tenant, a narrow date range or a rare category return a handful of results with no error.

Engines combine a filter with approximate search in three ways:

• Search, then filter: take the top C candidates from the whole index, then drop the ones that don't match. Expect about C × (fraction of rows that match). Acme was this case with C = 40.
• Filter, then exact scan: fetch the matching ids from a filter index and compute exact distances for them. Recall is perfect, and the cost grows with the number of matches: 2,000 matches × 768 × 4 B = 6 MB, well under a millisecond in numpy.
• Filter during search: the graph walk skips non-matching nodes. The page fills up, but at very low match rates the walk visits many nodes of other tenants.

Good engines run a query planner that estimates how many rows match, scans exactly when there are few and filters inside the graph walk when there are many. How each strategy works and what it costs, along with filter design itself (payload schemas, access control as a filter, nulls, testing for leaks), is covered in Metadata & Filtering. To choose a database, you need a narrower answer: which of these does your engine do for your filter mix, and can you see which one it picked?

What current engines do with a filter (checked 2026-09-25):

Managed services often do not document the mechanism at all. That is fine, because you are going to measure it anyway.

⚠️ Same engine, different answer. In Elasticsearch and OpenSearch, a filter in the right place is applied during the search, while the same filter one level up in the query becomes a post-filter. In pgvector, the fix is a session setting that is off by default. Read your own query plan (EXPLAIN in Postgres). In our Acme test, Postgres chose the HNSW index and post-filtered for Acme (2.9% of the rows), with or without a B-tree index on tenant_id. For a tenant with only 60 rows (0.1%), the same planner skipped the vector index. It fetched the 60 rows through the B-tree (or, without one, with a sequential scan), sorted them by exact distance and returned a perfect page. The danger zone is the middle: filters narrow enough to starve a 40-candidate list, but not narrow enough for the planner to give up on the index.

Your turn: 5,000,000 chunks of 768 dimensions; a filter matches 0.2% of them. Engine A searches first with 100 candidates and filters afterwards. Engine B's planner scans exactly when fewer than 20,000 rows match. Predict what each returns for LIMIT 10, and roughly what B's exact scan costs.

Requirement 3: hybrid retrieval — one engine or two

Symptom: the query XK-4471 overheating returns three good articles about thermal management and misses the one page that mentions XK-4471. Dense embeddings blur exact identifiers. You need a lexical signal as well (why this happens is covered in Sparse vs Dense Retrieval).

You can get it two ways.

Inside one engine. One query runs lexical (or learned-sparse) and dense retrieval and fuses the results. As of 2026-09-25:

• Postgres: pgvector plus Postgres full-text search in the same database. The pgvector README suggests fusing the two with RRF, in SQL or in your application. Both indexes change in the same transaction, so there is nothing to keep in sync.
• Elasticsearch and OpenSearch: BM25 is native. Elastic recommends its semantic_text workflow with RRF, but its rrf and linear retrievers need an Enterprise licence (checked in the Elasticsearch source, 2026-09-25). On a lower tier, you run the BM25 and kNN searches and fuse the two lists in your application. OpenSearch fuses a hybrid query with a normalization processor or an RRF score-ranker processor.
• Weaviate: hybrid combines BM25 with vector search. alpha = 1 means vector only and alpha = 0 means keyword only, the same direction as this course's α. The server default is 0.75.
• Milvus: a built-in BM25 function turns text into sparse vectors, and hybrid search fuses the results with rankers such as RRF.
• Qdrant: named dense and sparse vectors in one collection, fused in the Query API with RRF or DBSF. An IDF modifier on the sparse vector gives BM25-style weighting.
• Vespa: rank profiles combine bm25(...) and vector closeness(...) over several ranking phases.
• Pinecone: full-text search went GA on 2026-09-09. It "adds BM25 keyword ranking and text-match filters to the same index as your dense vectors" (blog).
• LanceDB: vector search, full-text search and SQL over the same table.

Across two systems. A vector store plus a search engine you already run, fused in your application. With two systems you get two indexes to keep in sync, two failure modes and two latency tails. Every write, delete included, has to reach both, and until it does the two sides disagree. What you gain is freedom to pick each side on its merits.

In-engine hybrid ties you to that engine's tokenizers and analyzers, so check its language support (stemming, compound words, CJK segmentation) against your content. Measure it the same way as everything else: recall@10 on a labeled query set, reported separately for queries that contain identifiers and queries that don't. Fusion and weight tuning are covered in Hybrid Retrieval Systems.

Requirement 4: multitenancy — count tenants, then look at the biggest

Symptom: the design works for ten tenants. At 3,000 you have 3,000 collections, one tenant holds a quarter of the data, and offboarding a customer means a script that runs all night.

Three patterns, from cheapest to most separated:

Sources: Qdrant multitenancy, Milvus multi-tenancy, Pinecone multitenancy, Weaviate tenant states, pgvector README.

Tenant sizes are rarely even. They usually follow a long tail: a few giants and hundreds of tiny tenants. The giants set your memory and shard layout. The tail stresses your filters, because each tiny tenant is a very selective filter. Some engines handle both ends explicitly: Qdrant 1.16 added tiered multitenancy, which keeps small tenants together in a shared shard and lets you promote a large one to a dedicated shard.

Two more questions to ask:

• Can you delete a tenant quickly and completely? Pinecone describes deleting a namespace as "a lightweight and almost instant operation". With a shared index and a filter, it is a delete-by-filter followed by background cleanup. Find out how long that takes and when the space is actually reclaimed.
• Is one tenant's traffic isolated from another's? Separate partitions do not automatically mean separate compute. Ask what is shared.

⚠️ Storage isolation is not authorization. In every one of these patterns, a query sent with the wrong tenant id returns the wrong tenant's data. Placing tenants on their own shards does not change that: in Qdrant, a query sent without a shard key runs on all shards. The tenant must come from the authenticated identity on the server, never from a request parameter the client controls. See Metadata & Filtering and, for Qdrant specifically, Advanced Qdrant.

Requirement 5: writes, freshness and consistency

Symptom: a support agent uploads a new policy and asks about it a minute later. The old policy comes back. Or a customer invokes their right to erasure, and their chunks keep showing up for a day.

Questions to ask:

• Write rate. Peak upserts per second, deletes, and the occasional full re-embed. Does heavy writing slow queries down, and for how long does a bulk load leave the index un-optimized?
• Visibility. When can a query see a write? In Postgres, when the transaction commits. Qdrant's upsert(..., wait=True) "will be returned only when all changes are applied". Watch out: the Python client's bulk upload_collection defaults to wait=False. Pinecone documents itself as "eventually consistent, so there can be a slight delay before new or changed records are visible to queries", and it returns log sequence numbers you can use to check (docs).
• Durability. Is a write acknowledged only after it reaches a write-ahead log? pgvector rides on Postgres's WAL, which gives replication and point-in-time recovery (README FAQ). FAISS and hnswlib can save index files, but they have no write-ahead log and no replication, and sharing one index between several writing processes is your problem.
• Updates that are not vectors. A price or a permission changes, but the text does not. Can you update metadata without rewriting the vector? Qdrant's set_payload and Vespa's partial updates of attribute fields change it in place. In Elasticsearch and OpenSearch, an update rewrites the whole document, vector included ("The document must still be reindexed", says the Elasticsearch update API). A Milvus upsert, even a partial update, inserts the merged entity and deletes the old one.

Your vector index is derived data. You can always rebuild it from the source documents, so it is not the source of truth. But a rebuild costs time. Re-embedding 40 million chunks at an assumed 500 chunks per second takes 40,000,000 / 500 = 80,000 s, about 22 hours. At the roughly 220 chunks per second that all-MiniLM-L6-v2 managed for 180-word chunks on one laptop CPU (Apple M2 Max, September 2026), it takes about 182,000 s, or 2.1 days. So snapshots and replicas still matter. And switching embedding models is a planned migration whatever database you pick (see Vector Embeddings and Data Freshness & Lifecycle).

Requirement 6: ops model and cost

Symptom: the database was "free", but the on-call rotation, the upgrade weekends and the 3 a.m. disk-full page were not. Or the opposite: the managed bill grew in step with traffic.

Four ops models:

Object storage changed the cost picture. Several 2026 services keep vectors in object storage and cache hot data on local SSD or in memory: Pinecone serverless ("All your vector data is stored in highly efficient, distributed object storage", architecture), Chroma Cloud, turbopuffer, LanceDB on S3, and Amazon S3 Vectors (GA on 2025-12-02, up to two billion vectors per index, announcement). Storage is cheap, and the price is cold-query latency. turbopuffer's architecture page reports a p50 of 874 ms for a cold query against 1M documents and 14 ms once they are cached. AWS says infrequent S3 Vectors queries return "in under one second" and frequent ones in about 100 ms. If your traffic touches many rarely-used tenants, cold start is a requirement, not a footnote.

Usage-based vs provisioned, with made-up prices. Say Plan U charges USD 0.10 per 1,000 queries, and Plan P is a provisioned node at USD 600 per month that handles your peak. Plan P costs the same at any traffic. The break-even volume is 600 / 0.10 × 1,000 = 6,000,000 queries per month, which is an average of 6,000,000 / 2,592,000 s ≈ 2.3 queries per second over a 30-day month.

Sustained, high-QPS traffic favours provisioned capacity. That is why Pinecone added Dedicated Read Nodes (GA on 2026-04-15) with "fixed hourly per-node pricing" for such workloads (blog). Spiky or low traffic favours usage pricing. Real prices change often, so plug in the current price sheet and your own query mix. Then add people: a self-hosted cluster needs someone who can upgrade it, restore it and get paged for it.

Requirement 7: ecosystem and portability

Symptom: permission checks, joins with the customer table, and "only documents the user can see" all live in application code, copied into three services.

• Next to your data. If filters depend on relational data (permissions, subscriptions, org charts), a database that can join, or at least filter on fields you keep in sync, saves a lot of glue code. This is pgvector's strongest argument.
• Clients and integrations. SDKs in your languages, and connectors for your ingestion and orchestration tools. Frameworks change quickly, so treat a framework integration as a convenience, not a reason to choose.
• Lock-in is not where people think. Your vectors come from your model. Keep a copy (in object storage or next to the source documents) and they move with you. What is actually sticky: the query and filter syntax, fusion semantics, the tenant model, hosted embedding or reranking features, and operational tooling. Put a thin retrieval interface in your application (search(query, filters, k)), so a migration touches one module.

The 2026 landscape, profiled

These profiles were checked against official docs and release pages on 2026-09-25. The groups tell you what kind of system each one is, and the "pick it when" line ties each to the requirements above. None of them is "the best".

A few notes the table cannot hold:

• pgvector's scaling story is Postgres's: "Scale horizontally with replicas", or shard with Citus or a similar tool, per the README. Read replicas scale reads; writes still go to one primary.
• Milvus 2.6 (August 2025) introduced a native write-ahead log called Woodpecker, merged the index and data nodes, and unified its coordinators (release notes). Milvus 3.0 (2026-07-29) adds "external collections" that index lake tables in place (release notes). Milvus Lite, the embedded mode, was rebuilt in pure Python for 3.0. Version 3.2.1 (2026-08-25) offers FAISS-backed HNSW and IVF indexes but no product quantization, and its README scopes it to "local development and small-scale workloads" in a single process.
• Weaviate calls data containers collections (older material says "classes"), and its managed service is Weaviate Cloud.
• Vespa streaming search skips building indexes and scans each user's own documents at query time. The docs pitch it for "personal data applications" where each query touches a small subset.

Shortlists from requirements

Selection goes like this: write the requirements as numbers, cross out the options that fail a hard one, and take two or three survivors into a proof of concept.

Worked: the internal knowledge base

Requirements: 80,000 chunks of 768 dimensions; peak 5 QPS; retrieval p95 under 100 ms; a department filter (12 departments, the largest holds 20% of the chunks and the smallest 1.5%); Postgres already in production; two engineers, no platform team.

• Scale: 80,000 × 768 × 4 B = 0.25 GB, which numpy would scan exactly in about 5 ms. Postgres is much slower per row: each 768-dimension vector (about 3 KB) is stored out of line, fetched and compared row by row. On a heavily loaded laptop (pgvector 0.8.6, Postgres 18.6 in Docker), an exact ORDER BY embedding <=> $1 LIMIT 10 over all 80,000 rows took about half a second. Over the 16,000 rows of the largest department, found through a B-tree on department, it took 70–160 ms.
• Filters: at 1.5%, default post-filtering expects 40 × 0.015 = 0.6 rows. That is the Acme failure again, so iterative scans must be on.
• Ops: no new system is the cheapest system.

Shortlist: pgvector in the existing Postgres, with HNSW and hnsw.iterative_scan = relaxed_order set for the retrieval role (ALTER ROLE … SET works for it). The runner-up is exact search in SQL with no vector index: a B-tree on department narrows each query to one department. It is viable only if p95 for the largest department fits the 100 ms budget on your hardware, so measure that first. Add Postgres full-text search with RRF if users paste error codes. Measure ANN recall@10 per department against exact search, and p95 at 5 QPS. What would change it: 10× growth plus a strict 20 ms budget, or a primary database already near its CPU limit.

Your turn: the multitenant SaaS

Requirements: 3,000 tenants, 120 million chunks in total, 1,024 dimensions; sizes follow a long tail (the largest tenant has 25 million chunks, the median about 20,000); a tenant must be deletable within 24 hours; hybrid search is needed for product codes; one platform engineer. Which patterns survive, and what do you measure first?

Worked: the marketplace

Requirements: 400 million listings of 768 dimensions; 1,500 QPS at peak; retrieval p99 under 60 ms; every query filters on category, region and price range and ranks with text relevance plus business signals; 2,000 price updates per second; a six-person platform team.

• Memory: 400M × 768 × 4 B = 1.23 TB in float32, 307.2 GB in int8, 38.4 GB in binary. That means sharding across nodes, quantized vectors in RAM, and full precision on disk.
• Updates: 2,000 price changes per second must update attributes without rewriting vectors. Qdrant (set_payload) and Vespa (partial updates of attribute fields) change attributes in place. Elasticsearch, OpenSearch and Milvus rewrite the whole record, vector included (see Requirement 5), so for them this means 2,000 vector re-insertions per second plus the merges that follow. Keep them only if a proof of concept shows that write load is affordable.
• Ranking: text + vector + business signals in one pass is Vespa's home ground. Qdrant, run distributed, with sparse + dense retrieval and score boosting in the Query API, is the other candidate.

Shortlist Vespa and Qdrant, and take both into a proof of concept at production-like data volume, with the price updates running. A 1% sample can leave segments too small to get an HNSW graph at all, or put them under the planner's exact-scan threshold, and then you are benchmarking brute force.

A fair proof of concept on your own data

Symptom: the vendor chart said 2 ms. Your p99 in production is 180 ms. The chart used unfiltered queries on a public dataset with a warm cache.

A fair proof of concept is boring on purpose:

1. Write the requirements as numbers: corpus size today and in 18 months, peak and sustained QPS, a p95/p99 budget, filter mix, update rate, tenants.
2. Use real data at a size that exercises the production code path. At minimum, that is your largest tenant or shard. Toy samples leave segments too small for a graph, or push planners into exact scans.
3. Replay real queries with their real filters: a few hundred from your logs, and label 50–100 of them for relevance (see Retrieval Metrics).
4. Get two kinds of ground truth. Exact search under the same filter gives you ANN recall@k (did the index find the true neighbours?). Human labels give you retrieval recall@k (did the system find what the user needed?). A perfect index cannot fix a bad embedding model.
5. Bucket the results by filter selectivity and tenant size. Averages hide tails.
6. Compare at equal recall. Tune each candidate to the same ANN recall target, then compare latency and cost. One tuned candidate against one on defaults is not a comparison.
7. Load it: target QPS × 1.5, with concurrency, for at least 30 minutes, while writing at your update rate. Record p50/p95/p99.
8. Break it: restart a node, query cold, restore a backup, delete a tenant, run a version upgrade.
9. Price it: cost per 1,000 queries at projected load, plus storage per month, plus the people who run it.

Here is the core of steps 4–6 as a harness. Each candidate is just a search(q, tenant, k) function, so the same queries and the same ground truth score every one of them. Three toy candidates reproduce the strategies from Requirement 2 on a skewed tenant mix. The fourth calls a real engine.

import time
import numpy as np
import hnswlib

# A stand-in for a sample of your data: 60k chunks, 100 tenants with skewed sizes
rng = np.random.default_rng(3)
n, dim = 60_000, 64
centers = rng.normal(size=(40, dim))
X = centers[rng.integers(0, 40, n)] + 0.8 * rng.normal(size=(n, dim))
X = (X / np.linalg.norm(X, axis=1, keepdims=True)).astype(np.float32)
w = 1 / np.arange(1, 101) ** 1.2
tenant = rng.choice(100, size=n, p=w / w.sum())
sizes = np.bincount(tenant, minlength=100)

index = hnswlib.Index(space="cosine", dim=dim)
index.init_index(max_elements=n, M=16, ef_construction=64)
index.add_items(X, num_threads=1)     # single thread: same graph on every run
index.set_ef(40)

def search_then_filter(q, t, k):             # candidate A: ANN first, WHERE afterwards
    ids, _ = index.knn_query(q, k=40)
    return [i for i in ids[0] if tenant[i] == t][:k]

def filter_during_search(q, t, k):           # candidate B: filter-aware traversal
    ids, _ = index.knn_query(q, k=k, filter=lambda i: tenant[i] == t)
    return list(ids[0])

def planner(q, t, k, cutoff=1_000):          # candidate C: exact scan when few rows match
    if sizes[t] < cutoff:
        subset = np.flatnonzero(tenant == t)
        return list(subset[np.argsort(-(X[subset] @ q))[:k]])
    return filter_during_search(q, t, k)

def make_queries(tenants, k=10):             # same queries and ground truth for every candidate
    out = []
    for t in tenants:
        subset = np.flatnonzero(tenant == t)
        q = X[rng.choice(subset)] + 0.06 * rng.normal(size=dim)
        q = (q / np.linalg.norm(q)).astype(np.float32)
        out.append((q, t, set(subset[np.argsort(-(X[subset] @ q))[:k]])))
    return out

def evaluate(search, queries, k=10):
    rows, recall, ms = [], [], []
    for q, t, truth in queries:
        t0 = time.perf_counter()
        got = search(q, t, k)
        ms.append((time.perf_counter() - t0) * 1000)
        rows.append(len(got))
        recall.append(len(truth & set(got)) / k)
    return f"rows {np.mean(rows):4.1f}  recall@{k} {np.mean(recall):.2f}  p95 {np.percentile(ms, 95):4.1f} ms"

buckets = {"big tenants ": make_queries(rng.choice(np.flatnonzero(sizes > 3_000), 100)),
           "tail tenants": make_queries(rng.choice(np.flatnonzero(sizes < 300), 100))}
print("tenant sizes: largest", sizes.max(), "smallest", sizes.min())
for name, fn in [("A search, then filter ", search_then_filter),
                 ("B filter during search", filter_during_search),
                 ("C planner             ", planner)]:
    for bucket, queries in buckets.items():
        print(name, bucket, evaluate(fn, queries))

# tenant sizes: largest 16605 smallest 54
# A search, then filter  big tenants  rows  6.1  recall@10 0.58  p95  0.3 ms
# A search, then filter  tail tenants rows  1.1  recall@10 0.11  p95  0.1 ms
# B filter during search big tenants  rows 10.0  recall@10 1.00  p95  0.7 ms
# B filter during search tail tenants rows 10.0  recall@10 1.00  p95 32.3 ms
# C planner              big tenants  rows 10.0  recall@10 1.00  p95  0.4 ms
# C planner              tail tenants rows 10.0  recall@10 1.00  p95  0.1 ms

Rows and recall are the same on every run. The timings are not: over eleven runs on a shared laptop, B's tail p95 ranged from 23 to 119 ms. Read the output bucket by bucket:

• A starves the tail exactly as the Acme arithmetic predicted. The tail still gets about one row only because each test query was built near one of that tenant's own chunks. It also hurts the big tenants: the four of them hold between 5% and 28% of the chunks, so only the largest can expect 10 matches among 40 candidates.
• B always fills the page, but its tail p95 is tens of milliseconds. For a tenant with a few hundred chunks, the graph walk visits a great many nodes of other tenants before it collects 10 matches. Part of that time is this toy's Python callback per visited node. A real engine checks the filter in native code, but it still has to walk the graph.
• C is what planners in Qdrant, Weaviate and Vespa do: exact search for small match sets, filtered traversal for large ones. It gets full recall at low latency in both buckets.

Now plug in a real engine. This adapter runs against a local Qdrant server (docker run -p 6333:6333 qdrant/qdrant), in the same session as the harness:

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
name = "poc_chunks"
client.create_collection(name, vectors_config=models.VectorParams(
    size=dim, distance=models.Distance.COSINE))
client.create_payload_index(name, "tenant", field_schema=models.KeywordIndexParams(
    type=models.KeywordIndexType.KEYWORD, is_tenant=True))
client.upload_collection(name, vectors=X, ids=range(n), wait=True,   # default is wait=False
                         payload=({"tenant": f"t{t}"} for t in tenant))
while client.get_collection(name).status != models.CollectionStatus.GREEN:
    time.sleep(0.5)                          # at real scale: let indexing finish before timing
info = client.get_collection(name)
print("points", info.points_count, " indexed vectors", info.indexed_vectors_count)

def qdrant_search(q, t, k):                  # candidate D: a real engine, same harness
    flt = models.Filter(must=[models.FieldCondition(
        key="tenant", match=models.MatchValue(value=f"t{t}"))])
    hits = client.query_points(name, query=q, query_filter=flt, limit=k).points
    return [h.id for h in hits]

for bucket, queries in buckets.items():
    print("D Qdrant              ", bucket, evaluate(qdrant_search, queries))
client.delete_collection(name)

# points 60000  indexed vectors 0
# D Qdrant               big tenants  rows 10.0  recall@10 1.00  p95  5.6 ms
# D Qdrant               tail tenants rows 10.0  recall@10 1.00  p95  9.8 ms
# (p95 ranged from 2 to 58 ms over eight runs against a busy shared local server)

Predict first: is Qdrant's 1.00 recall here evidence that its filtered HNSW search is exact?

What goes on the scorecard for each candidate:

⚠️ Benchmark theatre is a test built to confirm a choice already made. The usual forms: vendor numbers on vendor data; unfiltered queries only; latency without recall; candidates at different recall levels; a 60-second run on a warm cache; defaults for one candidate and expert tuning for another; a sample so small that every query is an exact scan.

Final round: no label on the problem

Real incidents don't come labeled. For each one, name the requirement that was skipped and the first move.

1. A team moved from a self-hosted engine to a usage-priced serverless service to "stop paying for idle nodes". Traffic is a steady 40 QPS, day and night. The bill tripled.

2. After a migration to a new engine, a proof-of-concept run on 1% of the data showed recall 1.00 everywhere. In production, recall for large tenants dropped to 0.85.

3. Answers cite a policy that was replaced yesterday. The new document is in the database, and a direct lookup by id finds it.

4. A teammate proposes migrating off pgvector "because it can't filter". The table has 2 million rows, and filtered queries return 2–3 rows instead of 10.

Cheat sheet: symptom → move → cost

Before moving on, take a system you know (or the multitenant SaaS above). Write its seven requirements as numbers. Cross out every option that fails a hard one, and name the two survivors. Then say, in one sentence each, what you would measure in the proof of concept, on which data, and what result would change your mind. If you can do that, the product names in this lesson can change next year without changing your decision process.

Next: Qdrant takes one of these engines hands-on: collections, payload indexes, filtering and a hybrid query. Metadata & Filtering goes deeper on the requirement that bit Acme.