Vector Database Architecture

The refund policy that would not die

Your team ships a support assistant over 2 million help-center chunks from 40 customer companies (tenants). It started as the simplest thing that works: an HNSW index from hnswlib in memory, payloads in a Python dict, and a pickle written every night. On Tuesday an agent at the tenant acme asks:

What is the refund window for annual plans?

What came back: one chunk, from acme/refunds-2024.md: "Annual plans can be refunded within 14 days."

What should have come back: chunk 1 of acme/refunds-2026.md, which replaced the 2024 file on Monday: "Annual plans can be refunded within 30 days."

Why. Two separate bugs, both in plumbing around the index rather than in the embeddings:

1. The code asked the index for the top 10 and then dropped other tenants' chunks. Forty companies have refund policies that read almost the same, so nine of the ten slots went to other tenants. The 2026 chunk ranked 14th overall and was never fetched.
2. Monday's ingestion added the chunks of the new file but never removed the old ones. The index labels were auto-increment integers, and nothing recorded which labels belonged to which file, so there was no way to delete them.

Neither fix is hard on a numpy array: a boolean mask filters exactly, and deleting a row is one line. On an HNSW graph both are hard. That is what a vector database is: an approximate nearest neighbour (ANN) index plus the machinery that lets you filter it, change it, survive a crash with it, and spread it over machines. This lesson opens that machinery up. Each move starts with a symptom you can see in results or metrics.

You should already know how embeddings and cosine similarity rank text (Vector Embeddings, Cosine Similarity & Distance Metrics). The insides of HNSW, IVF and quantization belong to ANN Algorithms, and product comparisons to Vector DB Selection. The examples use Qdrant 1.19 (September 2026) as one concrete engine; its API is taught in Qdrant and Advanced Qdrant.

Before any move: what the database stores

A vector database stores points in collections. A point has three parts:

A collection is a contract. When you create it you fix, for each dense named vector, the dimension and the metric. One point can carry a dense vector and a sparse vector side by side, which is how engines run hybrid retrieval in a single query (Hybrid Retrieval Systems).

What a collection cannot check is which model made the vectors. Query with a model of a different dimension and the failure is loud: Qdrant rejects the request with a 400 error, Vector dimension error: expected dim: 8, got 16. Query with a different model of the same dimension, or the same model without its query prefix, and every search still returns results. Nothing errors: the ranking quietly gets worse, sometimes much worse, and the scores shift, so any threshold you tuned stops meaning anything. So record the model with the data. Qdrant 1.19's create_collection accepts a metadata dict (for example {"embedding_model": "all-MiniLM-L6-v2"}), and your query service can compare it with its own model name at startup.

Before trusting any engine, including one you build yourself, get answers to six questions. Each move below explains one answer:

Move 1: Know when brute force is enough

Symptom: p95 latency grows in step with the corpus.

Start with the version that has no index at all. For unit vectors, cosine similarity is a dot product, so exact search is one matrix-vector product and a partial sort:

import time
import numpy as np

rng = np.random.default_rng(42)
n, dim = 200_000, 384
X = rng.standard_normal((n, dim), dtype=np.float32)
X /= np.linalg.norm(X, axis=1, keepdims=True)       # unit length: dot product = cosine

def search(q, k=10):
    scores = X @ q                                   # n dot products, one pass over X
    top = np.argpartition(-scores, k)[:k]            # the k best, unordered, O(n)
    return top[np.argsort(-scores[top])], scores     # sort only those k

q = X[1234] + 0.02 * rng.standard_normal(dim, dtype=np.float32)
q /= np.linalg.norm(q)
ids, scores = search(q)
print(ids[:3], scores[ids[:3]].round(3))

times = []
for _ in range(20):
    t = time.perf_counter()
    search(q)
    times.append(time.perf_counter() - t)
print(f"scanned {X.nbytes / 1e6:.1f} MB per query, median {np.median(times) * 1000:.1f} ms")
# [  1234  98295 163234] [0.927 0.216 0.214]
# scanned 307.2 MB per query, median 6.7 ms     <- Apple M2 Max, all cores; yours will differ

The result is exact: this is the true top 10, by definition. The cost model is simple too. Every scan reads every byte of X: n × dim × 4 bytes. On a modern CPU the multiply-adds are cheap, so the limit is memory bandwidth. On the laptop used for this lesson (an Apple M2 Max), numpy sustained about 40–57 GB/s on this scan with all cores.

Predict first: you have 1,000,000 chunks of 768-dimensional float32 vectors. How many bytes does one query read, and how long does it take at 50 GB/s? How many queries per second can the whole machine serve?

The ceiling is per scan, not per query. Queries that arrive together can share one pass: stack them as columns of a matrix Q and compute X @ Q, a matrix-matrix product that reads X once for all of them (FAISS's flat index does this for batched searches). On the same laptop, one pass over 1,000,000 × 768 with 64 queries took 78–135 ms against about 60 ms for a single query: roughly 500 queries per second instead of 16. The price is latency, because a query waits for the scan in progress to finish, and very large batches stop being bandwidth-bound and become compute-bound.

So brute force is enough when the vectors fit comfortably in RAM, one scan fits the latency budget, and scans per second × bytes per scan stays well below the machine's bandwidth, with batching if you need it. Hundreds of thousands of chunks usually pass all three. Tens of millions usually do not.

Brute force also gives you things for free, and it is worth listing them because the rest of this lesson is the price of giving them up:

• Exact results. Recall against the true top k is 1.0 by definition.
• Filters are a boolean mask over the rows before the scan.
• Deletes and updates are "drop that row" and "overwrite that row".
• There is no build step and nothing to tune.

The ANN index: skip most of the work

An ANN index organizes the vectors so that a query computes distances to a small fraction of them. Here is a real run: FAISS HNSW (M = 16, efConstruction = 100) on 200,000 × 384 structured synthetic vectors, 200 queries, with ANN recall@10 measured against exact search on the same vectors. The synthetic data has clusters and a low intrinsic dimension, which is easy for HNSW, so treat the recall column as an illustration: real embeddings give different numbers at the same settings, and only a run on your own vectors tells you yours.

At efSearch = 128 the index computes 1.5% of the distances and returns 98.8% of the true top-10 neighbours. The price: building the graph took 30–50 seconds on 8 threads, the graph costs memory on top of the vectors, the results are no longer exact, and filters, updates and deletes are no longer one line each.

import time
import numpy as np
import faiss

rng = np.random.default_rng(0)
n, dim, nq, k = 200_000, 384, 200, 10
# structured synthetic data: 64 topics around a 24-dimensional subspace, plus noise
Z = rng.standard_normal((n, 24)).astype(np.float32)
W = rng.standard_normal((24, dim)).astype(np.float32)
C = rng.standard_normal((64, dim)).astype(np.float32) * 2
X = Z @ W + C[rng.integers(0, 64, n)] + 0.5 * rng.standard_normal((n, dim)).astype(np.float32)
X /= np.linalg.norm(X, axis=1, keepdims=True)
Q = X[rng.choice(n, nq, replace=False)] + 0.3 * rng.standard_normal((nq, dim)).astype(np.float32) / np.sqrt(dim)
Q /= np.linalg.norm(Q, axis=1, keepdims=True)

flat = faiss.IndexFlatIP(dim)
flat.add(X)
faiss.omp_set_num_threads(1)
t = time.perf_counter()
_, exact = flat.search(Q, k)
print(f"exact: {n} distances/query, {(time.perf_counter() - t) / nq * 1000:.2f} ms/query")

faiss.omp_set_num_threads(8)
hnsw = faiss.IndexHNSWFlat(dim, 16, faiss.METRIC_INNER_PRODUCT)
hnsw.hnsw.efConstruction = 100
hnsw.add(X)
faiss.omp_set_num_threads(1)
for ef in [16, 64, 128, 256]:
    hnsw.hnsw.efSearch = ef
    faiss.cvar.hnsw_stats.reset()
    t = time.perf_counter()
    _, I = hnsw.search(Q, k)
    ms = (time.perf_counter() - t) / nq * 1000
    recall = np.mean([len(set(I[i]) & set(exact[i])) / k for i in range(nq)])
    print(f"efSearch={ef}: {faiss.cvar.hnsw_stats.ndis / nq:.0f} distances/query, "
          f"{ms:.3f} ms/query, ANN recall@10={recall:.3f}")

⚠️ Two different recalls. ANN recall@k is |ANN top-k ∩ exact top-k| / k: how faithfully the index reproduces brute force on the same vectors. Retrieval recall@k is the share of truly relevant chunks that make the top k, and it needs human or LLM relevance labels. An ANN recall of 0.99 cannot rescue a weak embedding model: exact search could change at most 1% of the result slots, so it would miss almost the same chunks. Always say which one you measured.

Measure it. Most engines have an exact mode for exactly this: in Qdrant it is search_params=SearchParams(exact=True). Sample 100–500 real queries, run each one with ANN and exact (with the same filter), and average the overlap. Separately, keep a small labeled query set for retrieval recall (Retrieval Metrics goes deeper). The query-time search effort is a per-query knob, and defaults differ by engine: efSearch = 16 in FAISS, ef = 10 in hnswlib, and in Qdrant hnsw_ef defaults to the collection's ef_construct (100 unless you change it). How to tune it lives in ANN Algorithms.

Your turn. An internal search has 400,000 chunks of 1,024-dimensional float32 vectors. It needs p95 ≤ 100 ms at a peak of 30 queries per second, on a server with 32 GB of RAM that also runs the application. Is brute force enough?

Move 2: Filter inside the search, not after it

Symptom: filtered queries return fewer than k results, or the right chunk appears only when you drop the filter.

This is the first bug from the hook. Post-filtering means fetching the top N and then dropping what does not match. When the filter has nothing to do with similarity, about N × selectivity results survive: a tenant with 2% of the chunks keeps about 2 of the top 100. When the filter is tied to similarity, it can be worse. In the hook, other tenants' near-identical refund policies outranked acme's 2026 chunk and pushed it to rank 14, just outside the 10 that were fetched. Fetching more candidates only moves the cliff.

The fix is to hand the filter to the engine, so it searches only the points that match. In Qdrant that is query_filter= on query_points, backed by a payload index on the field. Qdrant never post-filters. Its query planner estimates from the payload index how many points match, then, per segment, either scans the matching points exactly (few matches) or walks a filterable HNSW graph that checks the filter as it goes (many matches) (Qdrant indexing docs). Create payload indexes before bulk loading: Qdrant adds its extra filter-aware graph links only for fields that are indexed when the graph is built, so an index added later can leave filtered searches with lower recall until the graph is rebuilt. Not every engine works this way. pgvector's README (0.8.x) says that with approximate indexes "filtering is applied after the index is scanned": a condition matching 10% of rows with the default hnsw.ef_search of 40 returns about 4 rows. Version 0.8.0 added iterative index scans (hnsw.iterative_scan) to keep scanning until enough rows match (pgvector).

Measure it as in Move 1, with the filter on both sides: ANN results against exact=True results with the same filter, on sampled real queries. Also count the queries that return fewer results than the limit. Metadata & Filtering takes all of this apart: the filtering strategies, the planner, payload index design, and starvation computed for your own selectivities.

⚠️ When the filter is a permission. A tenant or ACL filter is a security boundary, not a relevance tweak. Apply it in the engine, on every query, and make it fail closed: a chunk with no ACL field is denied, never public. Metadata & Filtering also covers ACL staleness and leak testing.

Move 3: Make the index changeable

Symptom: a deleted or replaced document still shows up, or re-running ingestion doubles the results.

ANN indexes are built for reading, not for editing. Inserting into HNSW means running a search to find the new node's neighbours and rewiring their links. Deleting is worse: the removed node's neighbours lose links, so libraries usually just mark it deleted (hnswlib has mark_deleted) and keep routing through it. As marks pile up, searches get slower and the memory is never freed. IVF centroids trained on last year's data slowly stop fitting this year's. Rebuilding from scratch fixes everything, but the 200,000-vector graph above already took 30–50 seconds to build.

Engines deal with this the way search engines and databases have for decades: never edit a big index in place.

write ──► WAL ──► small appendable segment        (searched by brute force)
                        │ big enough: seal it
                        ▼
            optimizer builds an HNSW graph ──► indexed segment (read-only + delete marks)
                        │ too many deletes, or too many small segments
                        ▼
            vacuum / merge: copy the live points into a new segment, swap it in

query ──► search every segment ──► skip deleted points ──► merge into one top k

New writes land in a small appendable segment, cheap to brute-force and searchable at once. A full segment is sealed and indexed in the background. A delete writes a tombstone: the vector stays on disk, but no search returns it. When deletes pile up, vacuum rewrites the segment without them, and merges combine small segments, because each segment is searched separately.

The same pattern appears across engines, under different names. Qdrant has appendable and non-appendable segments. By default it vacuums a segment once at least 20% of its vectors are deleted and it holds at least 1,000 vectors, and it keeps serving reads during a rebuild through a proxy that sends new changes to a copy-on-write segment (optimizer docs). Milvus has growing and sealed segments and builds indexes on the sealed ones. Elasticsearch stores "the vector values of each segment as a separate HNSW graph", and warns that kNN search "can be up to several times faster if there are fewer segments" (Elastic docs).

🧠 Analogy, and where it stops. This is how Lucene has handled text indexes for twenty years: immutable segments, a bitset of deleted documents, background merges that drop them. The difference is price. Merging two inverted indexes is mostly sequential copying. Merging two HNSW graphs means rebuilding links, which costs a real search per point.

A vector store in about 50 lines

Here is the whole idea as runnable Python: append-only segments, a map from id to its one live row, tombstones, a filter applied while scanning, and compaction. The _log method is Move 4's job; ignore it for now.

import json, os
import numpy as np

class TinyStore:
    """Append-only segments, tombstones and a write-ahead log. Exact search."""

    def __init__(self, wal_path, seal_at=3):
        self.wal_path, self.seal_at = wal_path, seal_at
        self.segments = [[]]   # each segment: list of rows (id, vector, payload)
        self.live = {}         # id -> (segment, row); any other row is a tombstone

    def _log(self, op):        # Move 4: append to the log and fsync BEFORE applying
        with open(self.wal_path, "a") as f:
            f.write(json.dumps(op) + "\n")
            f.flush()
            os.fsync(f.fileno())

    def _apply(self, op):
        if op["op"] == "upsert":
            if len(self.segments[-1]) == self.seal_at:
                self.segments.append([])          # seal the full segment, open a new one
            self.segments[-1].append((op["id"], np.array(op["vec"]), op["payload"]))
            self.live[op["id"]] = (len(self.segments) - 1, len(self.segments[-1]) - 1)
        else:
            self.live.pop(op["id"], None)         # the old row stays, now unreachable

    def upsert(self, pid, vec, payload):
        op = {"op": "upsert", "id": pid, "vec": vec, "payload": payload}
        self._log(op)
        self._apply(op)

    def delete(self, pid):
        op = {"op": "delete", "id": pid}
        self._log(op)
        self._apply(op)

    def search(self, q, k=3, tenant=None):
        hits = []
        for s, seg in enumerate(self.segments):
            for r, (pid, vec, payload) in enumerate(seg):
                if self.live.get(pid) != (s, r):
                    continue                      # tombstone: skip
                if tenant and payload["tenant"] != tenant:
                    continue                      # filter while scanning, not after
                hits.append((round(float(vec @ q), 3), pid))
        return sorted(hits, reverse=True)[:k]

    def compact(self):                            # rewrite segments without tombstones
        rows = [self.segments[s][r] for s, r in self.live.values()]
        self.segments = [rows[i:i + self.seal_at] for i in range(0, len(rows), self.seal_at)] or [[]]
        self.live = {row[0]: (i // self.seal_at, i % self.seal_at) for i, row in enumerate(rows)}

    def rows(self):
        return sum(len(seg) for seg in self.segments), len(self.live)

if os.path.exists("wal.jsonl"):
    os.remove("wal.jsonl")
store = TinyStore("wal.jsonl")
store.upsert("refunds-2024#0", [0.9, 0.1, 0.0], {"tenant": "acme"})
store.upsert("refunds-2024#1", [0.7, 0.3, 0.0], {"tenant": "acme"})
store.upsert("pricing#0",      [0.1, 0.9, 0.0], {"tenant": "acme"})
store.upsert("refunds#0",      [0.8, 0.2, 0.0], {"tenant": "globex"})
store.delete("refunds-2024#0")                                    # Monday: old policy removed
store.delete("refunds-2024#1")
store.upsert("refunds-2026#0", [0.95, 0.05, 0.0], {"tenant": "acme"})
store.upsert("pricing#0",      [0.2, 0.8, 0.0], {"tenant": "acme"})  # re-embedded chunk

q = np.array([1.0, 0.0, 0.0])
print("rows stored, rows live:", store.rows())
print(store.search(q, k=2, tenant="acme"))
store.compact()
print("after compaction:", store.rows())

Predict first: after those eight operations, how many rows are physically stored and how many are live? What does the acme search for the top 2 return? (The toy vectors are not unit length, so the scores are plain dot products.)

rows stored, rows live: (6, 3)
[(0.95, 'refunds-2026#0'), (0.2, 'pricing#0')]
after compaction: (3, 3)

Why it works. One sentence holds the design together: a row is visible if and only if live[id] points at it. An upsert appends a row and moves the pointer. A delete removes the pointer. Compaction copies only the rows the pointers reach. Nothing already written is ever modified, which is what lets a real engine build a slow index over a sealed segment while queries keep reading it. The cost is that dead rows use disk and scan time until compaction runs.

Watch a real engine do it

The same lifecycle on the Qdrant server: 60,000 points, then a delete of 30% of them.

import time
import numpy as np
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
name = "lifecycle_demo"
if client.collection_exists(name):
    client.delete_collection(name)
client.create_collection(name, vectors_config=models.VectorParams(
    size=384, distance=models.Distance.COSINE))

def watch(label, limit_s=60):
    t0 = time.time()
    while True:
        info = client.get_collection(name)
        print(f"{label} +{time.time() - t0:3.0f}s {info.status.value:6} "
              f"points={info.points_count} indexed_vectors={info.indexed_vectors_count}")
        if info.status == models.CollectionStatus.GREEN or time.time() - t0 > limit_s:
            return
        time.sleep(3)

X = np.random.default_rng(1).standard_normal((60_000, 384), dtype=np.float32)
client.upload_collection(name, vectors=X, ids=range(60_000), batch_size=1000, wait=True)
watch("after upload")
client.delete(name, points_selector=models.PointIdsList(
    points=[i for i in range(60_000) if i % 10 < 3]), wait=True)       # delete 30%
watch("after delete")
client.delete_collection(name)
# after upload +  0s yellow points=60000 indexed_vectors=29864
# after upload +  3s yellow points=60000 indexed_vectors=42432
# after upload +  6s green  points=60000 indexed_vectors=58000
# after delete +  0s yellow points=42000 indexed_vectors=58000
# after delete +  3s yellow points=42000 indexed_vectors=47198
# after delete +  6s green  points=42000 indexed_vectors=40600

Timings depend on the machine. The shape of the trace does not:

The server's per-segment telemetry told the same story: each indexed segment had about 30% of its vectors marked deleted (for example 4,179 of 13,920) until vacuum rebuilt it. Runs differ in the split: another run left 4,000 points plain, and a third indexed all 60,000 (then 42,000 after vacuum). The pattern was the same every time.

⚠️ Deleted is not erased. Until its segment is rewritten, a deleted vector still sits in a segment file on disk, and every snapshot taken until then copies it. In our Qdrant 1.19.1 test, the original write also stayed in the write-ahead log, which snapshots include, after vacuum; a rebuild into a fresh collection removed both. If you must honour "delete my data within 30 days", your process has to cover vacuum and backup retention too (Data Freshness & Lifecycle).

Ids you can find again

The hook's second bug was an addressing problem. Give every chunk an id derived from where it came from, and keep the document id in the payload:

import uuid
import numpy as np
from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")
rng = np.random.default_rng(0)

def ingest(name, doc_id, n_chunks, id_style):
    if id_style == "replace-doc":                      # delete the old version first
        client.delete(name, points_selector=models.FilterSelector(filter=models.Filter(
            must=[models.FieldCondition(key="doc_id", match=models.MatchValue(value=doc_id))])))
    points = []
    for i in range(n_chunks):
        pid = (str(uuid.uuid4()) if id_style == "random"
               else str(uuid.uuid5(uuid.NAMESPACE_URL, f"{doc_id}#{i}")))
        points.append(models.PointStruct(id=pid, vector=rng.random(8).tolist(),
                                         payload={"doc_id": doc_id, "chunk": i}))
    client.upsert(name, points)

for style in ["random", "deterministic", "replace-doc"]:
    name = f"ids_{style}"
    client.create_collection(name, vectors_config=models.VectorParams(
        size=8, distance=models.Distance.COSINE))
    ingest(name, "refunds.md", 5, style)   # version 1: five chunks
    ingest(name, "refunds.md", 3, style)   # version 2: the edited file has three chunks
    print(f"{style:>13}: {client.count(name).count} points")

Predict first: the file had five chunks, and its edited version has three. How many points does each style leave behind?

       random: 8 points
deterministic: 5 points
  replace-doc: 3 points

Batch your writes. Each write request pays a network round trip, parsing, a log append and an apply step. In repeated runs of the script below on the local server (a busy, shared laptop), sending 1,000 points as four requests of 250 was about 10 to 25 times faster than sending them one per request, from nothing but request count. In four runs of a variant, single upserts with wait=False, which return before the change is applied, took about as long as with wait=True (1.6–3.0 seconds against 2.0–2.7 seconds for 1,000), so almost all of the cost is per request:

import time
import numpy as np
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
name = "batching_demo"
if client.collection_exists(name):
    client.delete_collection(name)
client.create_collection(name, vectors_config=models.VectorParams(
    size=384, distance=models.Distance.COSINE))
X = np.random.default_rng(3).standard_normal((2_000, 384), dtype=np.float32)
points = [models.PointStruct(id=i, vector=X[i].tolist()) for i in range(2_000)]

t = time.perf_counter()
for p in points[:1_000]:                       # one request per point
    client.upsert(name, [p], wait=True)
one_by_one = time.perf_counter() - t

t = time.perf_counter()
for i in range(1_000, 2_000, 250):             # four requests of 250 points
    client.upsert(name, points[i:i + 250], wait=True)
batched = time.perf_counter() - t

print(f"1,000 single upserts: {one_by_one:.2f} s | 4 batches of 250: {batched:.2f} s")
client.delete_collection(name)
# 1,000 single upserts: 2.23 s | 4 batches of 250: 0.20 s    <- varies by machine

Move 4: Survive the crash

Symptom: after a restart, writes the API acknowledged are gone, or some of them appear twice.

The nightly pickle in the hook has an obvious hole: a crash at 17:00 loses every write since midnight. Writing all the segments to disk after every request would be far too slow. The standard fix is a write-ahead log (WAL): before a change touches any data structure, append it to a sequential log and fsync it. Appends to the end of one file are cheap. Rewriting segment files is not, and that can happen later, in bulk. The textbook design:

upsert ──► 1. append the record (sequence number, length, checksum) to the WAL, fsync
       ──► 2. acknowledge: the write now survives a crash
       ──► 3. apply it to the appendable segment: now it is searchable
       ──► 4. later: flush segments to disk, remembering the last sequence number they contain
       ──► 5. truncate WAL records the flushed segments already cover

restart ──► load flushed segments ──► replay WAL records after their sequence number

Qdrant's storage docs make the same promise: "Once a change has been added to the WAL, it will not be lost even if a power loss occurs" (storage docs). What "added" means depends on the call. In the Qdrant 1.19.1 source (update worker), a wait=True write flushes the log to disk before the change is applied. A wait=False write is acknowledged as soon as its record is in the memory-mapped log file. That survives a crash of the Qdrant process, because the operating system still holds the page, but the record reaches the disk only at the next flush (a background flush runs every flush_interval_sec, 5 seconds by default). So a power cut can take the last few seconds of wait=False writes. Details like this change between versions and engines: check them for yours instead of trusting a one-line promise. The docs also explain why replay cannot double-apply a change. Each segment stores the version of the last change applied to it and of every point, and the updater ignores a change whose sequence number is older than the point's current version. Milvus goes further and runs its WAL as a separate service: its architecture docs list Kafka, Pulsar and Woodpecker as implementations.

Now crash the toy store. This block continues the previous TinyStore block, whose log file is still on disk:

def recover(wal_path):
    fresh = TinyStore(wal_path)
    with open(wal_path) as f:
        for line in f:
            fresh._apply(json.loads(line))     # replay only: do not log again
    return fresh

del store                                      # the process dies; the log file survives
store = recover("wal.jsonl")
print("after replay:", store.rows())
print(store.search(q, k=2, tenant="acme"))
# after replay: (6, 3)
# [(0.95, 'refunds-2026#0'), (0.2, 'pricing#0')]

Every acknowledged write comes back, deletes included: the replay rebuilds the tombstones too. Notice (6, 3). The compaction ran in memory before the crash, but it was never persisted, so replaying the whole log recreates the dead rows. A real engine avoids that by starting from its last flushed segments and replaying only the tail of the log. That is also why it can truncate the log instead of letting it grow forever.

Two details separate a real WAL from this toy:

• Torn writes. A crash can leave half a record at the end of the log. Real logs store a length and a checksum per record, and replay stops at the first record that fails the check. Our json.loads would just raise on it.
• Syncing is the price. A sync waits for the disk. An engine that syncs on every request pays that wait per request, and batching many points into one request pays it once. An engine that syncs on a timer answers sooner but can lose the last interval of writes on a power cut.

Durable is not the same as visible

A write can be safe from a crash and still not show up in search. Qdrant makes the two states explicit. With wait=False, the call returns status acknowledged: "Request is saved to WAL and will be process[ed] in a queue", which means in the log, not yet applied and not yet synced. With wait=True, it returns completed once the log is synced and the change is applied, and a query sent after that sees it. If a user uploads a file and asks about it two seconds later, write with wait=True, or poll until the point is there. Even then, a new point sits in an appendable segment that is searched by brute force until the optimizer indexes it. You do not have to wait for the collection to turn green (with default settings; Advanced Qdrant shows two opt-in settings, indexed_only and prevent_unoptimized, that hide unindexed points).

The WAL is not a backup. It lives on the same disk, and it is truncated in whole log files (32 MB each by default in Qdrant) once flushed segments cover them, so a record can outlive its flush. For backups and for moving data between clusters, engines take snapshots: a consistent copy of a collection's segments, write-ahead log and configuration. The only way to know a snapshot works is to restore one on a schedule (Advanced Qdrant covers the commands).

Your turn. An ingestion service receives 1,000 single-chunk writes per second from a message queue, one upsert request per chunk. The engine syncs its log to disk on every request, and a sync takes about 1 ms. What limits throughput, and what would you change? What must hold before you acknowledge the message on the queue?

Move 5: Do the memory arithmetic before you buy the machine

Symptom: a node is OOM-killed, starts swapping, or has a p95 that jumps after every reboot.

Everything in a vector database competes for RAM: the vectors, the graph, payload indexes, and the page cache that holds whatever lives on disk. Do the arithmetic before choosing hardware. For 10 million chunks of 768 dimensions (GB = 10⁹ bytes):

Three things stand out:

• The graph is small next to the vectors. An hnswlib index (M = 16) over the 200,000 vectors from Move 1's table saved to a file 149 bytes per vector larger than the raw 384-dimensional vectors: 128 bytes of base-layer links, plus labels and upper layers.
• Payload text can outweigh the vectors. Keep it on disk and fetch it only for the final k results. Qdrant 1.19 puts payload storage in the cold tier (below) by default, while payload indexes are pinned in RAM by default and can be moved to cached or cold.
• One vector per token multiplies everything. ColBERT-style late interaction stores about 50 times more than one 768-dimensional vector per chunk here. It usually runs as a reranking stage over a few hundred candidates rather than as the main index (Reranking Models).

Quantization with rescoring

The standard 2026 move when vectors do not fit: keep a compressed copy in RAM for the search, and keep the full-precision originals on disk for a final check.

query ──► score int8 (or binary) codes in RAM: cheap, a little wrong
      ──► keep limit × oversampling candidates, e.g. 10 × 3 = 30
      ──► re-score those 30 with float32 originals from disk: 30 × 3,072 B ≈ 92 KB read
      ──► return the top 10 by exact score

Engines that search segment by segment, Qdrant among them, do this inside every segment, so a query can rescore up to limit × oversampling originals per segment (Advanced Qdrant).

int8 cuts vector memory by 4 times and binary by 32 times. Rescoring repairs the order of the candidates. It cannot bring back a neighbour that the compressed search never put among the candidates. When ANN recall drops, raise the oversampling and measure again against exact search. How the codes are computed and how much each scheme loses is covered in ANN Algorithms. Qdrant's knobs (oversampling, rescore) are in Qdrant and, tuned for production, Advanced Qdrant. If your model was trained for Matryoshka truncation, keeping only the first 256 of 768 dimensions is another way to cut memory by 3 times, provided you truncate the queries the same way and test the quality loss (Vector Embeddings).

RAM, page cache, disk

Most engines keep their files on disk and memory-map them: the operating system pages data into RAM when it is touched, and evicts it under pressure. Qdrant 1.19 offers up to three placements per component, and says outright that "data is always persisted on disk regardless of this setting". Not every component accepts all three, and each has its own default (Advanced Qdrant lists them):

HNSW jumps around the graph, so every hop may touch a different page. Elastic's docs put it bluntly: HNSW "only works efficiently when most vector data is held in memory", and "once the working set no longer fits in cache, queries start reading from disk and latency climbs sharply". That gives you two different symptoms:

• Slow after a reboot, then fine. After the machine reboots, or the node starts on a fresh host, the page cache is empty, so early queries page-fault on their way through the graph. (A process or container restart on the same host usually finds much of the data still in the page cache.) Fixes: pre-load the hot files (the cached tier does this at startup), pin the small hot parts, and keep a node out of the load balancer until it is warm. A handful of "warm-up" queries touches only a sliver of a large graph, so on its own it does not make a node warm.
• Slow all the time. The working set is bigger than RAM, so queries keep reading from disk. Shrink what the search touches (quantize and pin the codes, keep originals cold), add RAM, or shard.

Your turn. You have 25 million chunks of 1,024 dimensions, and nodes with 64 GB of RAM and NVMe disks. Can one node serve an HNSW search with int8 codes in RAM, originals on disk for rescoring, 10 results and oversampling 3? What is in RAM, and what is read from disk per query?

Move 6: When one machine is not enough

Symptom: the data no longer fits on one node, one node's failure takes search down, or one node cannot serve peak QPS.

There are two separate tools here, and mixing them up is the classic mistake. Shards split the points. Replicas copy them.

                      query: limit 10
                            │
                     coordinator node
           ┌────────────────┼────────────────┐
        shard 0          shard 1          shard 2        each shard: its own segments and graphs
     (replicas a, b)  (replicas a, b)  (replicas a, b)   any one replica answers
        top 10           top 10           top 10
           └────────────────┼────────────────┘
                 merge 30 candidates → top 10

Sharding divides the data. Each shard holds a subset of the points: in Qdrant, "assigned to shards using a consistent hashing algorithm", or by a shard key you choose, such as a tenant (horizontal scaling). A query goes to every shard, each returns its own top 10, and the coordinator merges them. The merge is exact, as long as each shard's list is: any point in the global top 10 is also in its own shard's top 10. So every shard must return the full limit (plus any offset), not limit ÷ shards. Routing by a shard key lets a tenant's query touch one shard instead of all of them. It is a placement tool, not an isolation boundary: a query sent without a shard key searches every shard, so keep the tenant filter on every query. Plan the shard count up front. Qdrant's docs say shard_number cannot be changed without recreating the collection; resharding exists only in Qdrant Cloud (since 1.13).

Fan-out has a price: the slowest shard sets the pace. If each shard independently exceeds 50 ms on 1% of requests, a query that waits for 8 shards is slow whenever any of them is slow:

1 − 0.99⁸ ≈ 7.7% of queries, instead of 1%.

At 16 shards it is 14.9%. So adding shards can turn a rare hiccup into a common one. Mitigations: fewer, larger shards; route by key when you can; and hedged reads, which send the same request to a second replica if the first is slow. Qdrant exposes these as read_fan_out_factor and read_fan_out_delay_ms.

Replication copies each shard to several nodes. It buys two things. One is availability: a node dies and another replica keeps serving. The other is read throughput, because reads spread across replicas. It costs replication factor × the memory, and every write is applied on every replica. How many replicas must confirm a write (write_consistency_factor), and how many must agree on a read (consistency), decides whether a read can miss a write that was just acknowledged. Qdrant uses Raft "to keep the cluster topology and collection structure consistent across nodes"; for the points themselves, those two consistency settings decide what a read can see.

Your turn. 60 million chunks, 768 dimensions, int8 codes plus HNSW links (M = 16) in RAM, originals on disk. Nodes have 32 GB of RAM, of which you allow 22 GB for these structures. The cluster must survive losing any one node. How much RAM do the structures need in total, and what is the smallest cluster?

Boss level: one query through the whole machine

Put the moves together. An acme agent asks "refund window for annual plans", limit 5. The collection has 3 shards × 2 replicas, int8 codes with rescoring, and a payload index on tenant.

embed the query (same model and prefix as the documents)
  │
  ▼
any node ──► check the dimension (400 if it is wrong) ──► fan out to 3 shards, one replica each
  │
  ▼ inside each shard, for each segment:
  planner: how many acme points here?  few → exact scan   many → filtered HNSW walk
  score int8 codes, skip tombstones, keep 5 × oversampling candidates
  rescore candidates with float32 originals (page cache or disk)
  │
  ▼
shard merges its segments → top 5 ──► coordinator merges 3 × 5 → top 5
  │
  ▼
fetch payload (chunk text, doc_id) for the 5 winners → prompt

Each stage has its own way to fail, and its own measurement:

Final round: no label on the problem

Real incidents do not say which move they need. Find the symptom, then the move.

1. After each nightly re-ingest, some answers show the same chunk twice in the top 5, with identical text and identical scores of 0.83. Unchanged documents are affected too.

2. Every Monday a job deletes expired tickets, about 40% of the collection. For roughly an hour afterwards p95 doubles, then it recovers on its own. Nothing else changed.

3. To cut costs, a 40 GB collection (float32 vectors plus the HNSW graph, no quantization, default tiers) moved from a node with 64 GB of RAM to one with 32 GB. p95 went from 15 ms to 180 ms and stays there, days after the move.

Cheat sheet: symptom → move → cost

Before moving on, take a system you know (or the hook's assistant) and answer the six questions from the start of this lesson on paper: exact or approximate, and how you measured it; where the filter runs; what a delete does physically; when a write is durable and when it is visible; what must be in RAM; what happens when a node dies. Then write down the number you would measure for each on your own data: ANN recall@10 against exact search on 200 sampled real queries, with their real filters; retrieval recall on a small labeled set; p95 at peak QPS, both steady and right after a reboot; points per doc_id after a re-ingest; and a delete canary. If you can fill in every row, the product lessons will be comparisons, not surprises.

Next: ANN Algorithms for what happens inside HNSW, IVF and quantization; Vector DB Selection for choosing an engine, or deciding you do not need a new one; then Qdrant and Advanced Qdrant for one engine in depth.