Metadata & Filtering

Ten results, one survivor, and it is the wrong one

A support assistant serves 50 customer workspaces from one shared index of 400,000 chunks: contracts, policies, help articles. A user in the Northwind workspace asks:

How many days do we have to request a refund?

The retrieval code looks harmless:

# sketch: `store`, `embed` and `user` stand for your own vector store, embedder and session
hits = store.top_k(embed(question), k=10)
hits = [h for h in hits if h.workspace == user.workspace]   # keep our own documents

Here is what came back. Nine of the ten nearest chunks belonged to other workspaces: every customer has a refund clause, and they all read alike. The one survivor was Northwind's, section 4 of its 2024 contract: "refunds may be requested within 30 days". The model answered "30 days" and cited it.

What should have come back is the 2026 addendum to Northwind's contract, "refunds may be requested within 60 days", in force since 1 January. It ranked 23rd in the whole index and second among Northwind's own chunks.

That is two bugs, not one:

1. The filter ran after the search. Northwind owns 2% of the chunks, so a top-10 list holds 0.2 Northwind chunks on average, and none at all 82% of the time (0.98 multiplied by itself ten times is 0.817). This user got the unlucky kind of lucky.
2. Nothing said the 2024 section was superseded. Even a correct workspace filter lets it compete with the addendum. A similarity score measures closeness to the question, not whether a document is still in force.

The tempting fix is worse. A developer who notices empty answers and deletes the workspace line gets full top-10 lists, nine of them from other customers' contracts. The bug is now a data leak.

This lesson is about the half of retrieval that has nothing to do with meaning: deciding which chunks are allowed to compete, doing it without losing results or speed, and proving that nobody sees what they should not. There are eight moves, and each one starts from a symptom:

The examples run against a local Qdrant 1.19 server, because Qdrant filters during graph search and exposes its query planner. pgvector appears where it behaves differently. The Qdrant lesson teaches the client API, Advanced Qdrant covers its multitenancy and tuning features, and ANN Algorithms explains how HNSW graphs work.

Before any move: write the filter contract

Filters look like one-liners, so teams skip the design and meet the real requirements in an incident review. Answer four questions for every condition before you write code:

That last number is the filter's selectivity: the fraction of chunks that pass it. A filter that passes 1% is restrictive; one that passes 60% is broad. Vendors use "highly selective" in both directions, so this lesson just says "passes 1%".

Here is the contract the hook was missing:

💡 A filter decides eligibility; similarity decides order among the eligible. Keep the two jobs apart and most of this lesson follows.

Move 1: Filter inside the search, not after it

Symptom: fewer than k results, empty answers for small workspaces, and answer quality that depends on how big a customer is.

How many survive?

Predict first. There are 200,000 chunks and 2% of them are Northwind's. You take the unfiltered top 10 and drop everything that is not Northwind's. Over 200 queries, how many results come back on average, and how often none at all? What changes if you fetch 100 first, or 1,000?

import numpy as np

rng = np.random.default_rng(0)
n, dim, k = 200_000, 64, 10
X = rng.normal(size=(n, dim)).astype(np.float32)
X /= np.linalg.norm(X, axis=1, keepdims=True)
is_northwind = rng.random(n) < 0.02            # 2% of chunks belong to one workspace
northwind_ids = np.flatnonzero(is_northwind)

def post_filter(q, pool):
    """Search everything, keep the best `pool`, then drop other workspaces."""
    top = np.argpartition(-(X @ q), pool)[:pool]
    top = top[np.argsort(-(X[top] @ q))]
    return [int(i) for i in top if is_northwind[i]][:k]

def filter_first(q):
    """Restrict to the workspace, then search that subset exactly."""
    s = X[northwind_ids] @ q
    return [int(i) for i in northwind_ids[np.argsort(-s)[:k]]]

queries = rng.normal(size=(200, dim)).astype(np.float32)
queries /= np.linalg.norm(queries, axis=1, keepdims=True)
for pool in (10, 100, 1000):
    got = [post_filter(q, pool) for q in queries]
    truth = [set(filter_first(q)) for q in queries]
    returned = np.mean([len(g) for g in got])
    empty = np.mean([len(g) == 0 for g in got])
    recall = np.mean([len(set(g) & t) / k for g, t in zip(got, truth)])
    print(f"pool={pool:4d}  returned={returned:5.2f}  empty={empty:4.0%}  recall@10={recall:.2f}")
print("filter first: returned", np.mean([len(filter_first(q)) for q in queries]))

pool=  10  returned= 0.21  empty= 81%  recall@10=0.02
pool= 100  returned= 2.08  empty= 14%  recall@10=0.21
pool=1000  returned= 9.99  empty=  0%  recall@10=1.00
filter first: returned 10.0

To be 95% sure of 10 survivors at 2% you need a pool of 782 (from the binomial): 78 times what you return. At 0.1% it is 15,702. The pool grows like 1 / selectivity, and in an ANN index a bigger pool means a bigger ef, which means more graph work on every query. The model also assumes the workspace's chunks are spread like everyone else's. If the question is about something Northwind rarely writes about, its chunks sit further down the list and even fewer survive.

Oversampling is a fine patch for broad filters: at 50% a pool of 28 is enough for 95%. For restrictive filters, move the filter into the search.

Three places a filter can run

These are alternatives, not stages of one pipeline:

POST-FILTER               FILTER FIRST                      FILTER DURING GRAPH SEARCH
search all n chunks       payload index -> matching ids     walk the HNSW graph, stepping
  -> best pool hits         -> score only those ids,        only onto chunks that pass
drop non-matching           exactly -> top k                the filter -> top k
  -> 0..k results

Filtering first and filtering during the graph search complement each other. A filter that passes 200 chunks is the easy case: score 200 vectors exactly and you are done. The hard case is a filter that passes too many chunks to scan cheaply and too few to keep the graph connected. Move 3 shows how a query planner picks between them.

⚠️ Vendor vocabulary varies. Weaviate calls its approach "pre-filtering": it builds an allow-list from its inverted index first and then consults that list during the graph search. Engines also differ in whether the walk may pass through non-matching chunks or only step onto matching ones; Qdrant's does the latter, which is why it needs the extra edges in Move 3. Ask what happens to the graph walk, not what the feature is called.

Where pgvector sits

pgvector's HNSW index applies a WHERE clause after the index scan returns its candidates. That is post-filtering, with a pool of hnsw.ef_search, which defaults to 40. The pgvector README says a condition matching 10% of rows returns about 4 rows. We checked in a throwaway container (pgvector 0.8.6 on PostgreSQL 18.6, 100,000 random 64-dimensional vectors, LIMIT 10, 30 queries):

Iterative index scans (pgvector 0.8.0 and later) keep walking the graph until enough rows pass, up to hnsw.max_scan_tuples (20,000 by default). A filter that passes 0.01% can therefore still come back short. strict_order keeps exact distance order; relaxed_order gives up a little order for better recall. For a few known filter values the README suggests a partial HNSW index per value, and for many values, partitioning. Adding a B-tree index on the filtered column lets PostgreSQL's planner choose between the two indexes, and it does not always choose well: in one of our runs it kept the HNSW index for the 1% filter and returned 0 rows, while a comparable run switched to the B-tree and returned all 10. Read EXPLAIN instead of assuming.

Every retriever gets the filter

With hybrid retrieval (Hybrid Retrieval Systems), every branch must be filtered before fusion. In Qdrant, a query_filter on query_points also applies to each prefetch. On the local server, a filter set only at the top level returned 10 of 10 fused results from the right tenant. If your lexical index lives in a different engine, that engine needs the same ACL fields and the same filter. In our test, filtering only the dense branch let 5 of the 10 fused results come from other tenants. The same goes for a reranker hosted by a third party: filter before you send any candidates out. Filtering the fused list again is fine as a second guard, but not as the only one: an unfiltered branch fills its share of the fused list with chunks the filter then throws away.

Your turn. The product grows from 50 workspaces to 5,000, so each now owns about 0.02% of the 400,000 chunks, around 80 chunks. You return k = 8. How big must a post-filter pool be for a 95% chance of 8 survivors, and what should you do instead?

Move 2: Design the payload before you ingest

Symptom: the answer cites a superseded policy, a draft or another product's documentation. Or every filter needs five OR branches, because the same value is spelled three ways.

A payload (Qdrant's term; other engines say metadata) is the structured data stored next to each vector. Filters run on chunks, so every field a filter needs must be on every chunk. Copy document-level fields (workspace, ACL, status, dates) onto each chunk at ingestion. Here is a schema for the hook's corpus; the rest of the lesson uses these names:

Five rules keep it filterable:

1. Validity is a rule; recency is a preference. "Is this document in force?" (status is current, and now falls between effective_from and effective_to) is a hard filter: a superseded refund policy must never answer a refund question. "Newer is usually better" is soft (Move 7). The hook broke this rule.
2. Controlled vocabularies with one canonical spelling. Keyword matches are exact and case-sensitive: in Qdrant, a filter for "legal" does not match "Legal" (checked on the local server). Normalize at ingestion, not at query time.
3. Lists for multi-valued fields. A chunk about billing and payments gets product: ["billing", "payments"], and MatchValue(value="billing") matches it, because a condition on a list matches when any element does. The flip side: a field meant to hold one value (tenant, classification) must never hold a list, or a single allowed element lets the whole chunk through (Move 4).
4. One type per field, forever. Store dates either as RFC 3339 UTC strings with a datetime index and DatetimeRange (Qdrant has had datetime payload indexes since v1.8), or as integers with an integer index and Range. Pick one. Mixing them is the worst drift in Move 8.
5. Index only what you filter on. Every payload index costs memory and build time (Move 3).

Reading a filter

Qdrant's filter has four clauses: must (all of these hold), should (at least one holds), must_not (none holds) and min_should (at least N of a list hold). ⚠️ should is an OR: a chunk that matches none of its conditions is excluded. It is not a "nice to have" boost; boosting is Move 7.

Predict: which chunks survive this filter?

from qdrant_client import QdrantClient, models as m

client = QdrantClient(url="http://localhost:6333")
COLL = "policies"
chunks = {
    1: {"status": "current",    "product": ["billing"],             "lang": "en"},
    2: {"status": "superseded", "product": ["billing"],             "lang": "en"},
    3: {"status": "current",    "product": ["billing", "payments"], "lang": "de"},
    4: {"status": "current",    "product": ["payments"],            "lang": "en"},
    5: {"status": "draft",      "product": ["billing"],             "lang": "en"},
    6: {"status": "current",                                        "lang": "en"},
}
if client.collection_exists(COLL):
    client.delete_collection(COLL)
client.create_collection(COLL, vectors_config=m.VectorParams(size=2, distance=m.Distance.COSINE))
client.upsert(COLL, [m.PointStruct(id=i, vector=[1.0, float(i)], payload=p)
                     for i, p in chunks.items()], wait=True)

flt = m.Filter(
    must=[m.FieldCondition(key="status", match=m.MatchValue(value="current"))],
    should=[m.FieldCondition(key="product", match=m.MatchValue(value="billing")),
            m.FieldCondition(key="product", match=m.MatchValue(value="payments"))],
    must_not=[m.FieldCondition(key="lang", match=m.MatchValue(value="de"))],
)
hits, _ = client.scroll(COLL, scroll_filter=flt)
print(sorted(p.id for p in hits))
client.delete_collection(COLL)

The same logic is SQL's WHERE ... AND (... OR ...) AND NOT ... (over PostgreSQL's JSONB operators if your pgvector table keeps metadata in a JSONB column), or any other engine's filter language. The syntax changes; the logic and the missing-field surprises in Move 8 do not.

⚠️ Filters written by a model. Extracting product = billing from the question with an LLM ("self-querying", taught in Query Understanding & Intent) is fine for relevance filters. Validate the output against the schema's fields and allowed values, and put it under the server's access filter with AND. It must never be able to replace or widen that filter.

Move 3: Index what you filter on, before you build the graph

Symptom: filtered queries are slow, or they return lists that are short or full of wrong matches. No error, just worse answers.

The planner's decision

Qdrant plans each segment (a slice of a collection with its own index) separately. It first estimates how many points pass the filter, using the payload index: a per-field lookup structure from values to point ids. Then it chooses:

• estimate below full_scan_threshold: fetch the matching ids from the payload index and score them exactly (filter first);
• estimate above it: walk the HNSW graph, stepping only onto points that pass the filter (filter during search).

full_scan_threshold defaults to 10,000 KB, and Qdrant counts 1 KB as one 256-dimensional float32 vector. The threshold in points is therefore 10,000 × 256 / dim:

Why graphs break under filters

An HNSW graph stays navigable because every node links to its neighbors. Filter out most nodes and the matching ones become islands: a walk that may only step onto matching nodes runs out of neighbors before it reaches the true nearest ones. Qdrant's engineers measured about 21 links per node on the bottom layer of a one-million-point graph built with the default m = 16. Filter out 96% of the points and fewer than one link per node survives on average (Couzon & Ghafouri, Qdrant, August 2026).

Qdrant has two repairs:

• Filterable HNSW, at index time. For each indexed payload field, Qdrant adds extra edges between points that share a value, so a filter on that value still has a connected graph to walk. The edges exist per field value, never per combination of fields. A value shared by a large share of a segment gets none, because the main graph is dense enough for it.
• ACORN, at query time (Qdrant v1.16 and later, off by default). When a neighbor fails the filter, the search also looks at that neighbor's neighbors. Qdrant's docs put the cost at roughly 2–10× slower searches, and by default apply it only to filters with an estimated selectivity below 0.4.

A real run

This table comes from the local Qdrant 1.19.1 server: 100,000 synthetic 384-dimensional vectors in 500 clusters, one indexed segment, default HNSW settings (m = 16, ef_construct = 100), hnsw_ef = 64 and 100 queries. Each cell is recall@10 of the filtered search against exact filtered search.

Read it row by row:

• Small filters are safe everywhere. The planner scans them exactly. The popular claim that a tiny filtered set "collapses recall" is backwards: 184 points are scored exactly in no time. Without a payload index Qdrant still sent these small filters to an exact scan in our run; it was just slower, because every point's payload had to be checked.
• The 10% filter is the trap. It is too big for the exact path at 384 dimensions, so it goes to the graph. With edges built from an index created before the points arrived, recall is 1.00. Without them it is 0.06–0.16, and no error anywhere. The lists can come back full of wrong matches or short: in our repeat run they held ten results, while another run of the same setup averaged about seven. Raising hnsw_ef to 512 stayed below 0.4 in every run we made. On the late-index collection, turning on ACORN reached 1.00, and so did forcing a graph rebuild. On the collection with no payload index, ACORN changed nothing (0.08 with it and without it in a repeat run): it needs the index too.
• Broad filters lose a little. At 30% the graph stays connected, and recall is 0.91–0.98.

Qdrant's indexing docs say it plainly: create payload indexes before ingesting data, because filter-aware edges are only built when the HNSW index is built after the payload indexes exist. An index added later helps the planner at once, but the edges arrive only with a rebuild. In our run, changing an HNSW setting on the collection triggered one.

Combinations are the other gap. Edges exist per field, so an AND of two broad values can land on an intersection that nothing covers. In Qdrant's own benchmark (one million 96-dimensional vectors, v1.18.2, default settings), an AND of two keyword filters that passed 4% of points had recall@10 against exact search of 0.397 with ACORN off and 1.00 with it on, at 7.3 ms instead of 1.1 ms on their laptop. Their advice: measure recall for each filter shape, and turn ACORN on only for the shapes that need it.

Cardinality is not selectivity

Cardinality is how many distinct values a field has; selectivity is how much one condition keeps. They pull in opposite directions:

• A high-cardinality field (doc_id, user_id, a legal matter_id with 400,000 values) keeps a handful of chunks per value. Lookups are short, and the planner scores the matches exactly. Index it as a keyword field. Do not create a collection per value.
• A low-cardinality field (lang, status) keeps a huge slice per value. Those filters go to the graph, so they are the ones to check for recall. Their values are also the ones that may be too common to get extra edges.

Qdrant's docs use the word differently. When they say the planner estimates "the cardinality of a filtered result", they mean the estimated number of matching points: that is the selectivity side, not a field's number of distinct values.

For tenant ids, Qdrant (v1.11 and later) offers is_tenant=True on a keyword index, which stores each tenant's vectors together. Advanced Qdrant goes deeper.

In code, the order is the whole point: indexes first, points second. The helper below measures one filter shape against exact search:

from qdrant_client import QdrantClient, models as m

def create_chunk_collection(client: QdrantClient, name: str, dim: int) -> None:
    client.create_collection(
        name, vectors_config=m.VectorParams(size=dim, distance=m.Distance.COSINE))
    # Payload indexes first, points second: the graph is built with filter-aware edges.
    client.create_payload_index(
        name, "tenant", m.KeywordIndexParams(type=m.KeywordIndexType.KEYWORD, is_tenant=True))
    for field in ("doc_id", "status", "classification", "allowed_groups", "product", "lang"):
        client.create_payload_index(name, field, m.PayloadSchemaType.KEYWORD)
    for field in ("effective_from", "effective_to", "updated_at"):
        client.create_payload_index(name, field, m.PayloadSchemaType.DATETIME)

def filtered_recall(client, name, queries, flt, k=10):
    """ANN recall@k of one filter shape: approximate filtered search vs exact filtered search."""
    found = expected = 0
    for q in queries:
        approx = client.query_points(name, query=q, query_filter=flt, limit=k).points
        exact = client.query_points(name, query=q, query_filter=flt, limit=k,
                                    search_params=m.SearchParams(exact=True)).points
        found += len({p.id for p in approx} & {p.id for p in exact})
        expected += len(exact)          # a 4-chunk tenant can only ever return 4
    return found / expected if expected else 1.0

A guard helps too. Qdrant's strict mode with unindexed_filtering_retrieve=False rejects any search that filters on a field without an index. On the local server, a filter on an unindexed region field came back as HTTP 400, "Index required but not found". A missing index then shows up as an error in testing, not as a quiet recall drop in production.

Measure it. For every filter shape your product sends (each field, plus the common ANDs), sample a few hundred real queries and track filtered_recall and p95 latency. Our 10% row would have shown 0.06 on the first day.

Your turn. You have 3 million chunks at 768 dimensions in 12 segments of about 250,000 points. Queries filter on matter_id (400,000 distinct values, about 7.5 chunks each) and sometimes on practice_area (12 values; tax alone covers 20% of the chunks). A colleague wants a collection per matter, "because high-cardinality filters are slow". What happens to each filter, and what do you do?

Move 4: Access control is a filter the server writes

Symptom: a user sees a chunk they should not. Worse: the model paraphrases it, and nobody notices.

The model is not a security boundary

"Retrieve broadly and tell the model to ignore what the user may not see" fails three ways. Once a chunk is in the prompt, the model can quote, paraphrase or summarize it. Retrieved text can contain instructions that override yours (prompt injection). And nothing records whether the rule was followed. Access control has to decide which chunks may become candidates at all. Note what this does not cover: it keeps unauthorized chunks out, but it does not make authorized text safe to obey. Handling untrusted retrieved text belongs to Context Augmentation.

Four rules:

1. The server writes the filter from a verified identity: the tenant from the token, groups from the directory, clearance from the system that owns it. Never from the request body, a query parameter or model output.
2. Deny by default. Every condition must be positively satisfied. A chunk without ACL fields is visible to nobody, and an identity with a missing piece gets an error, not an unfiltered search. Watch for builders that can emit an empty filter: in Qdrant an empty filter matches every point.
3. Document-level security. A chunk inherits its document's ACL. If sections of one document have different audiences, chunk them separately and never merge them. Anything derived from documents (summaries, contextual chunk headers, cached answers) is visible only to users who may read every one of its sources.
4. Store groups, not users, on chunks. Move 5 explains why.

A builder that fails closed

from dataclasses import dataclass
from qdrant_client import models as m

LEVELS = {"public": 0, "internal": 1, "confidential": 2, "restricted": 3}

@dataclass(frozen=True)
class Principal:            # built by the server from a verified token and a directory lookup
    tenant: str
    groups: frozenset[str]
    clearance: str

def valid_identity(p: Principal) -> bool:
    """A non-empty tenant, a known clearance and at least one group, all strings."""
    return (isinstance(p.tenant, str) and p.tenant != ""
            and isinstance(p.clearance, str) and p.clearance in LEVELS
            and isinstance(p.groups, frozenset) and len(p.groups) > 0
            and all(isinstance(g, str) and g != "" for g in p.groups))

def access_filter(p: Principal) -> m.Filter:
    """Every condition must be positively satisfied; anything unknown raises."""
    if not valid_identity(p):
        raise PermissionError("incomplete identity")
    levels = [lvl for lvl, rank in LEVELS.items() if rank <= LEVELS[p.clearance]]
    single = m.ValuesCount(lte=1)         # a list such as ["public", "restricted"] fails
    return m.Filter(must=[
        m.FieldCondition(key="tenant", match=m.MatchValue(value=p.tenant)),
        m.FieldCondition(key="tenant", values_count=single),
        m.FieldCondition(key="classification", match=m.MatchAny(any=levels)),
        m.FieldCondition(key="classification", values_count=single),
        m.FieldCondition(key="allowed_groups", match=m.MatchAny(any=sorted(p.groups))),
    ])

def scoped_filter(p: Principal, user_filter: m.Filter | None = None) -> m.Filter:
    """A user's own filter may narrow the ACL, never widen it."""
    acl = access_filter(p)
    return m.Filter(must=[acl, user_filter]) if user_filter else acl

def one_value(x):
    """Qdrant treats "a" and ["a"] as the same single value; anything else counts as malformed."""
    if isinstance(x, list) and len(x) == 1:
        x = x[0]
    return x if isinstance(x, str) else None

def reference_allows(p: Principal, doc: dict) -> bool:
    """The same policy in plain Python, for tests only. Anything malformed means no."""
    if not valid_identity(p):
        return False
    level, groups = one_value(doc.get("classification")), doc.get("allowed_groups")
    groups = [groups] if isinstance(groups, str) else groups if isinstance(groups, list) else []
    return (one_value(doc.get("tenant")) == p.tenant
            and level in LEVELS and LEVELS[level] <= LEVELS[p.clearance]
            and any(isinstance(g, str) and g in p.groups for g in groups))

Four details carry the weight. Clearance is an allowed list of levels at or below the user's, so an unknown level matches nothing. tenant and classification must hold exactly one value: a condition on a list matches when any element does (Move 2), so without the count classification: ["public", "restricted"] would reach an intern and tenant: ["acme", "globex"] both workspaces. Reject such lists at ingestion too; the count is the second guard. The user's own filter goes inside a must next to the ACL, so it can only narrow the results. And reference_allows states the same policy in ordinary Python and answers "no" to anything malformed, so tests can check the database filter against something that is not the database filter.

⚠️ must_not with MatchExcept(allowed) looks like another way to say "no value outside the allowed list". Do not rely on it for security: in our run it caught ["public", 3] on a field without a payload index and let it through once a keyword index existed. The value count behaved the same both ways.

Test for leaks

A leak test checks every (principal, document) pair against the reference policy. Besides normal documents, it includes the ones your pipeline is "not supposed to" produce: missing fields, nulls, empty strings and lists, wrong casing, lists in single-valued fields, numbers and objects where strings belong. Its principals include lookups that failed or returned junk. It uses scroll with the filter to list everything each principal can see, on a collection indexed like production; in production the same filter object goes into query_points. This block continues the previous one:

import itertools
from qdrant_client import QdrantClient

MISSING = object()
TENANTS = ["acme", "globex", ["acme"], ["acme", "globex"], "ACME", "", None, 7, MISSING]
CLASSES = ["public", "internal", "confidential", "restricted", "Confidential", "", None, 2, MISSING,
           ["public"], ["public", "restricted"], ["public", None], [], {"level": "public"}]
GROUPS = [["everyone"], ["hr"], ["eng", "hr"], "everyone", ["everyone", None], [["everyone"]],
          [], None, 5, MISSING]
docs = []
for tenant, level, groups, dept in itertools.product(TENANTS, CLASSES, GROUPS, ["legal", "hr"]):
    doc = {"dept": dept}
    for key, value in (("tenant", tenant), ("classification", level), ("allowed_groups", groups)):
        if value is not MISSING:
            doc[key] = value
    docs.append(doc)

principals = [
    Principal("acme", frozenset({"everyone", "eng"}), "internal"),
    Principal("acme", frozenset({"everyone", "hr"}), "confidential"),
    Principal("acme", frozenset({"everyone"}), "restricted"),
    Principal("globex", frozenset({"everyone"}), "public"),
    Principal("acme", frozenset(), "restricted"),                  # group lookup failed
    Principal("acme", frozenset({"everyone"}), "Confidential"),    # clearance lookup drifted
    Principal("acme", frozenset({"everyone"}), None),              # clearance lookup failed
    Principal("", frozenset({"everyone"}), "internal"),            # no tenant in the token
    Principal(["acme", "globex"], frozenset({"everyone"}), "internal"),  # tenant claim is a list
]

client = QdrantClient(url="http://localhost:6333")
COLL = "acl_leak_test"
if client.collection_exists(COLL):
    client.delete_collection(COLL)
client.create_collection(COLL, vectors_config=m.VectorParams(size=2, distance=m.Distance.COSINE))
for field in ("tenant", "classification", "allowed_groups", "dept"):   # indexed as in production
    client.create_payload_index(COLL, field, m.PayloadSchemaType.KEYWORD)
client.upsert(COLL, [m.PointStruct(id=i, vector=[1.0, float(i)], payload=d)
                     for i, d in enumerate(docs)], wait=True)

def audit(make_filter, user_filter=None, dept=None):
    """(leaks, over-blocks): pairs the filter shows but the policy forbids, and vice versa."""
    leaks = over_blocked = 0
    for p in principals:
        try:
            flt = make_filter(p, user_filter)
            seen = {pt.id for pt in client.scroll(COLL, scroll_filter=flt, limit=10_000)[0]}
        except Exception:                      # a refused request shows nothing
            seen = set()
        allowed = {i for i, d in enumerate(docs)
                   if reference_allows(p, d) and (dept is None or d["dept"] == dept)}
        leaks += len(seen - allowed)
        over_blocked += len(allowed - seen)
    return leaks, over_blocked

legal_only = m.Filter(must=[m.FieldCondition(key="dept", match=m.MatchValue(value="legal"))])
print(len(docs), "docs x", len(principals), "principals =", len(docs) * len(principals), "pairs")
print("scoped_filter:        ", audit(scoped_filter), audit(scoped_filter, legal_only, "legal"),
      audit(scoped_filter, m.Filter()))

2520 docs x 9 principals = 22680 pairs
scoped_filter:         (0, 0) (0, 0) (0, 0)

No leaks and nothing over-blocked, with no user filter, with a "legal only" filter and with an empty one (m.Filter() matches everything on its own, so it must not widen anything). Now here are four builders that each look reasonable in code review:

def any_element(p, user_filter=None):       # positive lists, but a list passes on any element
    levels = [lvl for lvl, rank in LEVELS.items() if rank <= LEVELS[p.clearance]]
    return m.Filter(must=[
        m.FieldCondition(key="tenant", match=m.MatchValue(value=p.tenant)),
        m.FieldCondition(key="classification", match=m.MatchAny(any=levels)),
        m.FieldCondition(key="allowed_groups", match=m.MatchAny(any=sorted(p.groups)))])

def exclusion_style(p, user_filter=None):   # "hide what is above your clearance"
    above = [lvl for lvl, rank in LEVELS.items() if rank > LEVELS[p.clearance]]
    return m.Filter(
        must=[m.FieldCondition(key="tenant", match=m.MatchValue(value=p.tenant)),
              m.FieldCondition(key="allowed_groups", match=m.MatchAny(any=sorted(p.groups)))],
        must_not=[m.FieldCondition(key="classification", match=m.MatchAny(any=above))])

def optional_groups(p, user_filter=None):   # skips the group check when the list is empty
    levels = [lvl for lvl, rank in LEVELS.items() if rank <= LEVELS[p.clearance]]
    conds = [m.FieldCondition(key="tenant", match=m.MatchValue(value=p.tenant)),
             m.FieldCondition(key="classification", match=m.MatchAny(any=levels))]
    if p.groups:
        conds.append(m.FieldCondition(key="allowed_groups",
                                      match=m.MatchAny(any=sorted(p.groups))))
    return m.Filter(must=conds)

def or_merged(p, user_filter=None):         # "combines" the user's filter with should
    acl = access_filter(p)
    return m.Filter(should=[acl, user_filter]) if user_filter else acl

print("any_element:          ", audit(any_element))
print("exclusion_style:      ", audit(exclusion_style))
print("optional_groups:      ", audit(optional_groups))
print("or_merged + user flt: ", audit(or_merged, legal_only, "legal"))
client.delete_collection(COLL)

Predict which of them leak, and why, before you open the answer.

any_element:           (304, 0)
exclusion_style:       (862, 0)
optional_groups:       (724, 0)
or_merged + user flt:  (5040, 0)

Run this harness in CI whenever the builder, the schema or the ingestion code changes. In production, add canaries: one document per tenant containing a unique nonsense phrase, and a scheduled job that searches for each phrase as a principal from every other tenant and alerts on any hit.

Move 5: Permissions change; your index finds out later

Symptom: someone removed from a project still gets answers from its documents, or a document moved to a restricted folder stays findable all afternoon.

Your index holds a copy of the permissions, and two kinds of change reach it by different routes:

That is why chunks carry group ids rather than user ids. With user ids, adding one person to a project with 20,000 documents means rewriting every chunk of 20,000 documents. With groups, the chunks do not change at all.

A document-level change is a payload update, not a re-embedding. In Qdrant you select the chunks by doc_id:

from qdrant_client import QdrantClient, models as m

client = QdrantClient(url="http://localhost:6333")
COLL = "revoke_demo"
if client.collection_exists(COLL):
    client.delete_collection(COLL)
client.create_collection(COLL, vectors_config=m.VectorParams(size=2, distance=m.Distance.COSINE))
client.create_payload_index(COLL, "doc_id", m.PayloadSchemaType.KEYWORD)
client.upsert(COLL, [
    m.PointStruct(id=i, vector=[1.0, float(i)],
                  payload={"doc_id": "reorg-plan" if i < 3 else "handbook",
                           "allowed_groups": ["everyone"]})
    for i in range(5)], wait=True)

def visible_to(groups):
    flt = m.Filter(must=[m.FieldCondition(key="allowed_groups", match=m.MatchAny(any=groups))])
    return sorted(p.id for p in client.scroll(COLL, scroll_filter=flt)[0])

print("before:", visible_to(["everyone"]))
# The source system restricts one document: rewrite the ACL on all of its chunks.
this_doc = m.Filter(must=[m.FieldCondition(key="doc_id", match=m.MatchValue(value="reorg-plan"))])
client.set_payload(COLL, payload={"allowed_groups": ["leadership"]},
                   points=m.FilterSelector(filter=this_doc), wait=True)
print("after: ", visible_to(["everyone"]), "| leadership:", visible_to(["everyone", "leadership"]))
client.delete_collection(COLL)

before: [0, 1, 2, 3, 4]
after:  [3, 4] | leadership: [0, 1, 2, 3, 4]

Deleting a document works the same way: client.delete(COLL, points_selector=m.FilterSelector(filter=this_doc)) removes all of its chunks.

Compute your exposure window. Say the connector polls the source system's ACLs every 15 minutes and the update queue adds up to 2 minutes at p95. A revoked document then stays visible for up to about 17 minutes. If the requirement says "revocation within 60 seconds", you have two options, and they combine:

• Event-driven sync. Subscribe to the source system's permission-change events instead of polling. The window shrinks to queue lag, and you now own a pipeline that must never drop an event.
• Check at read time. After retrieval, ask the source system's authorization service about the few chunks you are about to use: one batched call for the 5–10 chunks of the context set. It adds a round trip to every query, and when that service is down you must fail closed and answer without those chunks rather than with unchecked ones.

Caches need the same care. An answer cached for one user must not be served to a user with fewer permissions, so key caches by the permission set or do not share them across users (Caching Strategies). The wider lifecycle of deletes, re-crawls and retention is in Data Freshness & Lifecycle.

Your turn. Group memberships travel in a token that is valid for 60 minutes and is not re-checked. The ACL sync runs every 10 minutes. At 09:00 Ana is removed from the finance group; at 09:05 the finance folder's audience is also narrowed from finance to finance-leads, a group Ana is not in. Her token was issued at 08:30. When does she stop seeing the finance documents?

Move 6: Tenancy: share the index or split it

Symptom: a customer asks "can another customer ever see our data?", and the honest answer is "only if we forget a filter".

Four patterns, from most shared to most isolated:

Look at the fourth column. A shard key is a placement and speed tool, not an access boundary: in Qdrant, a request that leaves the key out runs on all shards (distributed deployment docs), and Qdrant's own tiered-multitenancy example sends the tenant filter along with the shard key. So keep the server-built tenant filter on a sharded collection. A collection per tenant removes "forgot the filter" but not "passed the wrong name", because the application still chooses the name. Isolation becomes an infrastructure boundary only when the credential is scoped: Qdrant's granular JWT API keys (v1.9 and later) can be restricted to named collections, so a token issued for one tenant's collection is refused on another's.

Real systems mix patterns: most tenants share a collection, large ones get dedicated shards, and regulated ones get their own deployment. Qdrant supports the shared-plus-dedicated-shards part of that mix as tiered multitenancy (v1.16 and later), and its multitenancy guide warns that a collection per tenant is "rarely the most efficient approach", because each collection carries its own overhead. Weaviate stores each tenant on its own shard. The pgvector README notes that tenants sharing one approximate index affect each other's recall and speed, and suggests list partitioning. Product details are in Advanced Qdrant and Vector DB Selection.

Whatever you choose, keep running Move 4's leak test with principals from at least two tenants, and assert zero cross-tenant pairs.

Your turn. A group of 14 companies shares one HR-and-finance assistant: 8 million chunks, 70% of them from the largest company. One member is a bank whose regulator requires its data to sit in a separate deployment under its own keys. About once a year a company is sold, and its data must be gone within a week. Which pattern goes to whom?

Move 7: Boost preferences, filter only rules

Symptom: after you add "updated in the last 90 days", good older documents vanish. Or the opposite: a fresh but irrelevant note floats to the top.

Some conditions are preferences: newer is usually better, the user's own department is usually more relevant, reviewed articles are usually more reliable. Filtering on them turns a preference into a rule, and it deletes the best answer whenever that answer happens to be old. Rank them instead: retrieve with the hard rules, then blend a preference score into the ordering.

The trap is scale. Cosine scores bunch together (see Cosine Similarity & Distance Metrics), so a raw blend lets the preference swamp relevance. Here is a query about the export API, with recency on a 90-day half-life:

import numpy as np

names = ["export-api-guide", "changelog-2026-09", "export-faq", "release-notes-0921", "status-page-note"]
cos = np.array([0.82, 0.80, 0.79, 0.78, 0.61])       # cosine similarity to the query
age = np.array([700, 12, 200, 3, 1])                  # days since updated_at

recency = 0.5 ** (age / 90)                           # half-life of 90 days, in [0, 1]
raw = 0.7 * cos + 0.3 * recency                       # raw cosine blended with recency
sim_norm = (cos - cos.min()) / (cos.max() - cos.min())  # min-max over the candidates
normalized = 0.7 * sim_norm + 0.3 * recency

for label, s in [("cosine only", cos), ("raw blend", raw), ("normalized blend", normalized)]:
    print(f"{label:17}", [names[i] for i in np.argsort(-s)])
print("raw blend       ", np.round(raw, 3))
print("normalized blend", np.round(normalized, 3))

cosine only       ['export-api-guide', 'changelog-2026-09', 'export-faq', 'release-notes-0921', 'status-page-note']
raw blend         ['release-notes-0921', 'changelog-2026-09', 'status-page-note', 'export-faq', 'export-api-guide']
normalized blend  ['changelog-2026-09', 'release-notes-0921', 'export-api-guide', 'export-faq', 'status-page-note']
raw blend        [0.575 0.834 0.617 0.839 0.725]
normalized blend [0.701 0.907 0.664 0.86  0.298]

The raw blend puts status-page-note (cosine 0.61, clearly off topic) above both export documents, because recency spans 0 to 1 while cosine spans only 0.61 to 0.82 here. Min-max normalizing the similarity over the candidates puts both signals on the same 0–1 scale: the fresh relevant documents rise, and the off-topic note stays last. The weight (0.7) and the half-life are knobs to tune on labeled queries, not constants to copy. Min-max also depends on which candidates are present, so apply it to a pool of a stable size.

Two cautions:

• Validity stays a filter. Recency boosting must never be how you hide superseded documents. The old refund policy should not be eligible at all (Move 2).
• Relaxing a filter is also a preference mechanism. If a strict filter returns too few chunks, you may retry without the lowest-priority preference filter and say so ("no billing articles in German; showing English"). Never relax a security or validity rule.

Qdrant can compute such blends on the server with score-boosting formulas (v1.14 and later, including decay functions on datetimes); see Advanced Qdrant. Score normalization in hybrid fusion is covered in Hybrid Retrieval Systems.

Measure it. Label 50–100 queries where freshness matters and 50–100 where it does not. Compare retrieval recall@10 or nDCG@10 (Retrieval Metrics) for no boost, a date filter, and a boost with two or three half-lives. If a boost helps the first set and hurts the second, apply it only when the query asks for recent material.

Move 8: Missing fields, nulls and drift

Symptom: a filter that worked last month matches nothing for new documents, or matches documents it should not.

What does "not" mean for a missing field?

Predict each line's output before you open the answer:

from qdrant_client import QdrantClient, models as m

client = QdrantClient(url="http://localhost:6333")
COLL = "missing_fields"
docs = {
    1: {"classification": "public"},
    2: {"classification": "restricted"},
    3: {},                                   # field missing
    4: {"classification": None},             # field present, null
    5: {"classification": "Restricted"},     # drifted casing
}
if client.collection_exists(COLL):
    client.delete_collection(COLL)
client.create_collection(COLL, vectors_config=m.VectorParams(size=2, distance=m.Distance.COSINE))
client.upsert(COLL, [m.PointStruct(id=i, vector=[1.0, float(i)], payload=p)
                     for i, p in docs.items()], wait=True)

def ids(flt):
    return sorted(p.id for p in client.scroll(COLL, scroll_filter=flt)[0])

restricted = m.FieldCondition(key="classification", match=m.MatchValue(value="restricted"))
print("must_not restricted:     ", ids(m.Filter(must_not=[restricted])))
print("must be public:          ", ids(m.Filter(must=[m.FieldCondition(
    key="classification", match=m.MatchAny(any=["public"]))])))
print("except restricted:       ", ids(m.Filter(must=[m.FieldCondition(
    key="classification", match=m.MatchExcept(**{"except": ["restricted"]}))])))
print("is_empty classification: ", ids(m.Filter(must=[m.IsEmptyCondition(
    is_empty=m.PayloadField(key="classification"))])))
print("is_null classification:  ", ids(m.Filter(must=[m.IsNullCondition(
    is_null=m.PayloadField(key="classification"))])))
client.delete_collection(COLL)

must_not restricted:      [1, 3, 4, 5]
must be public:           [1]
except restricted:        [1, 5]
is_empty classification:  [3, 4]
is_null classification:   [4]

Other engines answer differently. In PostgreSQL (checked on 18.6 with a JSONB column), meta->>'dept' <> 'legal' excludes a row that has no dept key, because comparing NULL gives "unknown". But meta->>'dept' IS DISTINCT FROM 'legal' and NOT (meta @> '{"dept": "legal"}') include that row.

Instead of memorizing each engine's rules, write filters so that the answer does not matter: access and validity as positive lists of allowed values, and negation only for preferences.

Drift

Schema drift is the slow version of the same bug. A new connector writes "Confluence" instead of "confluence", or a pipeline rewrite stores updated_at as a Unix integer while older chunks hold ISO strings. Nothing raises an error. In Qdrant (checked on 1.19.1, with three string dates and four integer ones in one field), what each date filter sees depends on the payload index:

With an index, a range filter is answered by that index in its own units, and values of the other type are simply not in it. Whichever filter you write, part of the corpus silently drops out, and with a datetime index a numeric Range also lets old documents in.

Three defenses:

1. Validate at ingestion against one schema definition: types, allowed values, required ACL fields. Quarantine what fails. If a document must be indexed without a classification, give it the most restrictive level, never public.
2. Monitor the stored data, not only the pipeline code:

from qdrant_client import QdrantClient, models as m

def drift_report(client: QdrantClient, coll: str, required: list[str], enum_field: str) -> None:
    total = client.count(coll, exact=True).count
    for field in required:
        empty = m.Filter(must=[m.IsEmptyCondition(is_empty=m.PayloadField(key=field))])
        print(f"{field}: missing, null or [] in {client.count(coll, count_filter=empty).count}/{total}")
    for hit in client.facet(coll, key=enum_field, limit=20).hits:   # needs a keyword index
        print(f"{enum_field} = {hit.value!r}: {hit.count}")

client = QdrantClient(url="http://localhost:6333")
COLL = "drift_demo"
if client.collection_exists(COLL):
    client.delete_collection(COLL)
client.create_collection(COLL, vectors_config=m.VectorParams(size=2, distance=m.Distance.COSINE))
client.create_payload_index(COLL, "classification", m.PayloadSchemaType.KEYWORD)
payloads = ([{"classification": "internal", "allowed_groups": ["everyone"]}] * 6
            + [{"classification": "Internal", "allowed_groups": ["everyone"]}] * 3
            + [{"allowed_groups": []}])
client.upsert(COLL, [m.PointStruct(id=i, vector=[1.0, float(i)], payload=p)
                     for i, p in enumerate(payloads)], wait=True)
drift_report(client, COLL, ["classification", "allowed_groups"], "classification")
client.delete_collection(COLL)

classification: missing, null or [] in 1/10
allowed_groups: missing, null or [] in 1/10
classification = 'internal': 6
classification = 'Internal': 3

3. Alert when a required field's missing count rises above zero, or a keyword field grows a new spelling. For mixed date types, compare two counts: chunks that have the field at all (must_not + is_empty) and chunks a very wide range of the canonical type matches (a DatetimeRange from 1970 for RFC 3339 strings). The gap is the number of values stored as another type; on the table's data it was 7 − 3 = 4, with or without the datetime index. Do not compare a wide DatetimeRange with a wide Range: with a datetime index both match the same strings, so clean data raises a false alarm and integers stay invisible.

Final round: no label on the problem

Real incidents do not say which move they need. For each one, name the move, the first fix and what you would measure.

1. After you add the status = current rule, 30% of questions about a newly acquired product line get "I could not find this". The 12,000 chunks from that product's documentation were ingested last month by a new connector.

2. A filter on source = "confluence" returns 40% fewer chunks than it did last quarter, although nobody deleted anything. A second Confluence connector went live in June.

3. A penetration tester logged in as an intern finds an HR salary-band document in the answers. The document has no classification field. The filter builder was reviewed and "hides confidential and restricted".

4. Norwegian users say answers got worse after the team added lang = user's language as a filter. The Norwegian help center has 300 articles and the English one 40,000, and many Norwegian questions are now answered from the same few outdated articles.

Cheat sheet: symptom → move → cost

How would you measure it on your own data?

• Result counts: the share of queries per filter shape that return fewer than k results.
• Filtered ANN recall: approximate vs exact=True results for a few hundred real queries per filter shape, and p95 latency for the same shapes.
• Retrieval quality: retrieval recall@10 or nDCG@10 on a labeled query set, run with the filters your users actually send (Retrieval Metrics).
• Security: zero leaks in the harness on every build, zero canary hits in production, and the measured lag from a permission change in the source system to the change in your index.
• Schema health: missing required fields, unexpected spellings and mixed types, tracked over time.

Before moving on, take one system you know and write its filter contract: every condition, rule or preference, who sets it, what a missing field means, and what fraction it passes. Then name the one filter shape you would benchmark first.

Next: Reranking Models reorders the candidates your filters let through, and Infrastructure & Security covers the rest of the security picture: authentication, encryption and network boundaries.