Data Freshness & Lifecycle

Three numbers for one question, and none of them is right

It is 11:00 UTC on Friday 25 September 2026 (all times in this lesson are UTC unless a time zone is named). A customer asks your API's docs assistant: "What is the rate limit on the Pro plan?" The assistant is a classic RAG pipeline over the docs site. Its index is kept up to date by a job that runs every night at 02:00. The job lists the pages in the CMS, re-embeds the ones whose updated_at changed since the last run, and upserts their chunks. Here is a slice of that index and the retrieval step, using the small open model all-MiniLM-L6-v2:

import numpy as np
from sentence_transformers import SentenceTransformer

# What the vector index holds: chunk id -> text, as of the last nightly sync
INDEX = {
    "pricing#pro":         "Pro plan: USD 49 per month. Includes 600 requests per minute "
                           "per API key and 5 team seats.",
    "pricing#free":        "Free plan: 60 requests per minute per API key and one team seat.",
    "pricing#enterprise":  "Enterprise plan: custom rate limits, SSO and a dedicated support channel.",
    "pricing-2026-10#pro": "Pro plan: USD 59 per month. Includes 1,500 requests per minute "
                           "per API key and 10 team seats.",
    "legacy-plans#pro":    "Pro Legacy plan: 300 requests per minute per API key. "
                           "Upgrade to a current plan for higher limits.",
    "guide#429":           "When you exceed your plan's rate limit, the API returns HTTP 429 "
                           "with a Retry-After header.",
    "sdk#retries":         "The Python SDK retries 429 responses with exponential backoff, "
                           "up to 5 attempts.",
}
IDS = list(INDEX)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
VECS = model.encode([INDEX[i] for i in IDS])          # unit-length rows

q = model.encode("What is the rate limit on the Pro plan?")
scores = VECS @ q                                     # cosine, since all vectors are unit length
for rank, i in enumerate(np.argsort(-scores)[:4], start=1):
    print(rank, f"{IDS[i]:20}", round(float(scores[i]), 3))

Predict first: the top four chunks go into the prompt. Which one should the answer use?

1 pricing#pro          0.64
2 pricing-2026-10#pro  0.639
3 pricing#enterprise   0.622
4 legacy-plans#pro     0.587

Each wrong chunk is wrong for a different reason:

No retrieval setting fixes this. A bigger k, a reranker or a better embedding model can only reorder what the index holds, and the index holds a copy of the docs site taken at several different moments. It is derived data (Vector DB Selection), and this lesson is about keeping that copy honest over time.

• Freshness is how closely the index matches the source right now.
• Lifecycle is everything that happens to a piece of content between its first ingestion and its erasure: edits, new versions, effective dates, expiry, deletion and re-embedding.

Each move below starts from a symptom you can see:

It assumes payload filters (Metadata & Filtering) and deleting a document's chunks by doc_id (Qdrant). The pipeline itself (connectors, workers, backfills) is mapped in Data Pipeline & Indexing, and embedding at scale in Embedding Pipeline.

Before any move: write the freshness contract

Different content needs different promises. Before you pick a tool, answer five questions per source:

The third column holds the most useful answer: a value that changes every few seconds does not belong in an index at all. By the time it is embedded and searchable, it has changed again. Fetch it at answer time with a tool (Agentic RAG Systems, Move 2), and index what describes the product, not its current count.

Then sort every change by what it touches; the costs differ by orders of magnitude:

Move 1: Measure how stale you are

Symptom: someone asks "is the index up to date?" and gets a shrug. Users find out first.

Three quantities get mixed up here. Keep them apart:

• Index lag of a change: the time from a change at the source (an edit, a new page, a delete) to the moment search results reflect it. You measure it per change.
• Staleness of a document at time t: t minus the time of the oldest source change the index does not reflect yet, and zero when the index matches the source. At 11:00 the pricing page is 1 h 48 min stale; if tomorrow's sync works, that edit's lag ends at 16 h 48 min plus processing. The legacy page is 15 days stale and counting, because nothing in this pipeline will ever remove it.
• Document age: the time since a document last changed. A three-year-old page that never changed is old, not stale; monitor age as staleness and any archive breaches the target forever. Age is at most a ranking preference (Move 5).

Add coverage: documents at the source against documents in the index, in both directions. "Missing" means at the source but not searchable. "Zombie" means searchable but gone at the source.

Where the hours come from

The database is rarely where the hours go. With a poll interval P and processing time S, a change waits between S and P + S, and about P/2 + S on average if changes arrive evenly. Polling faster helps only until S dominates.

Averages and percentiles hide the stuck ones

Suppose a week brings 100,000 changes. Most take a few minutes, but 30 of them failed once and were never retried, so they sit unapplied for three days:

import numpy as np

rng = np.random.default_rng(7)
# One week of changes: index lag in minutes for each change (source change -> visible in search)
lag = rng.gamma(shape=2.0, scale=3.0, size=100_000)       # typical: a few minutes
lag[rng.choice(lag.size, 30, replace=False)] = 3 * 24 * 60  # 30 changes stuck for 3 days

SLA = 24 * 60                                              # "no change waits more than 24 h"
print(f"mean {lag.mean():.1f} min | p50 {np.percentile(lag, 50):.1f} | "
      f"p95 {np.percentile(lag, 95):.1f} | p99 {np.percentile(lag, 99):.1f} | "
      f"max {lag.max() / 60:.0f} h | over SLA: {(lag > SLA).sum()}")

Predict: does the p99 notice the 30 stuck changes?

mean 7.3 min | p50 5.0 | p95 14.2 | p99 20.0 | max 72 h | over SLA: 30

So write the freshness target in two parts:

1. A percentile for normal speed: "p95 index lag at most 15 minutes, per day". This watches the pipeline.
2. A ceiling that catches stuck changes: "no change older than 24 hours is still unapplied". Measure it as the age of the oldest unapplied change, or as the count of changes over the limit, and alert when it is above zero.

Give deletes a line of their own: they often travel a different path, and a pipeline can be fast at edits while never deleting anything, as the hook's was.

Every cache adds its own lag

Suppose the index applies a change within 15 minutes, a retrieval-result cache keeps entries for 1 hour, and an answer cache keeps answers for 6 hours. A retrieval result cached at minute 14 lives until about 1 h 14 min, and an answer built from it just before then lives 6 hours more. Users can see the old text for about 7 h 15 min, although the index was right after 15 minutes.

The end-to-end freshness target is the sum along the path, unless the caches are invalidated after the index changes (delete the entries built from the changed doc_id, which needs a record of each entry's sources, or, for an index rebuilt and published as a new version, put that version in the cache key: Caching Strategies).

Measure it end to end: canaries

Pipeline timestamps measure the stages you know about; a canary measures what users get. Every 10 minutes, create a small page in the source containing a unique token, such as canary 7f3a 2026-09-25T11:10. Search the production endpoint for it once a minute and record how long it takes to appear. Later, delete it at the source and record how long it takes to disappear. The canary crosses every stage, cache and replica. Put canaries in a group no real user can read.

Move 2: Detect change with the signal your source can give

Symptom: edits show up a day late, or deleted pages never leave.

The hook's nightly job asks the CMS "which pages changed since yesterday?" That question cannot return a page that no longer exists. Every change signal has blind spots like this, and you need to know them before you rely on one:

The design that works almost everywhere combines a fast signal for speed with a slow, complete one for correctness. Webhooks or CDC keep the normal lag in seconds. A daily listing diff catches whatever the fast path lost: dropped events, missed deletes, a connector that quietly stopped. (Hybrid Retrieval Systems uses the same idea to reconcile two stores.)

A listing diff in twenty lines

Continuing from the hook's code, here is a diff between a full listing of the source and what the index holds, at document level:

import hashlib

def text_hash(text):
    """Hash of the extracted text, whitespace-normalized, so re-rendering alone is not a change."""
    return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()[:12]

def doc_texts(chunks):                        # "doc#section" ids -> {doc_id: full text}
    docs = {}
    for chunk_id, text in chunks.items():
        docs.setdefault(chunk_id.split("#")[0], []).append(text)
    return {doc_id: " ".join(parts) for doc_id, parts in docs.items()}

def diff(source_texts, indexed_hashes):
    now = {doc_id: text_hash(t) for doc_id, t in source_texts.items()}
    return {"added":   sorted(now.keys() - indexed_hashes.keys()),
            "deleted": sorted(indexed_hashes.keys() - now.keys()),
            "changed": sorted(d for d in now.keys() & indexed_hashes.keys()
                              if now[d] != indexed_hashes[d])}

SOURCE = dict(INDEX)                                          # today's full CMS listing
SOURCE["pricing#pro"] = SOURCE["pricing#pro"].replace("600", "1,000")   # edited at 09:12
del SOURCE["legacy-plans#pro"]                                # page deleted on 10 September

indexed = {doc_id: text_hash(t) for doc_id, t in doc_texts(INDEX).items()}
print(diff(doc_texts(SOURCE), indexed))

Predict: what does it report? And which of the hook's three failures can no diff ever see?

{'added': [], 'deleted': ['legacy-plans'], 'changed': ['pricing']}

Three details decide whether hashing helps or hurts:

• Hash the extracted text, not the raw bytes. If the HTML footer says "rendered at 02:00:13", every page looks changed every night and you re-embed the corpus for nothing. Strip boilerplate first (Document Processing).
• Keep a second hash for the metadata you filter on (permissions, status, dates); together, the two hashes are the document's change fingerprint. A text hash alone misses a permission change completely, and a metadata-only change needs a payload update of those fields on all of the document's points, not re-embedding.
• Hashing is cheap; fetching is not. On this lesson's laptop, SHA-256 of a 12 KB document took 10–15 µs, and the whole text_hash above, whitespace normalization included, a few tenths of a millisecond. The real cost of re-crawl-and-hash is downloading every document within the source's rate limits.

Watermarks and clocks

Timestamp polling has a quieter trap. Suppose the indexer stores its own clock time as the watermark, and one CMS server's clock runs 4 minutes slow:

No later poll will pick up that edit. Nothing errors, and the page stays stale until someone edits it again or a listing diff notices. With this watermark, a server clock that runs fast only causes extra work: its rows look newer, so they are read again.

The fixes:

• Prefer a position that one authority assigns in commit order: a CDC log position or a change feed's cursor (the first and third rows of the table above). There is one sequence, and nothing can commit behind it.
• Otherwise, take the watermark from the source's own values (the largest updated_at among the rows you received) and re-read an overlap window (updated_at > watermark − overlap), skipping rows whose version you already hold. The overlap also catches transactions that commit late with an earlier timestamp.
• Make the overlap wider than the worst skew between the servers that stamp updated_at, or let the database stamp it, so there is one clock and only late commits to cover. A watermark taken from the rows turns the danger around: now the fast server is the problem. If one server runs 10 minutes fast, its next save pushes the watermark 10 minutes ahead of true time, and an edit from a correct server stamped inside that gap is skipped for good with a 5-minute overlap. A 10-minute overlap catches it.
• Keep the daily listing diff as a backstop for bulk scripts that update rows without touching updated_at.

Your turn. A partner's knowledge base offers only two endpoints: GET /articles?modified_since=… (paged, sorted by modification time) and GET /articles/ids (every live id). There are no webhooks. Edits must be searchable within 15 minutes and deletes gone within 24 hours. What do you build?

Move 3: Apply a change as a whole document, in order

Symptom: after an edit, answers mix old and new paragraphs. Or, after a burst of retries, the index quietly goes back to an older version.

Earlier lessons fixed the single-document case: a shrinking document leaves orphan chunks unless you delete by doc_id (Qdrant, Move 2), and upserting before deleting the rest means the page is never missing (Classic RAG Pipeline). At corpus scale, two more things happen all the time: events arrive out of order, and many workers write at once. Here is a replace function that survives both, using the local Qdrant server and the hook's model:

import hashlib, uuid
from datetime import datetime, timezone
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
COLL = "docs"
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"   # recorded on every point

if client.collection_exists(COLL):
    client.delete_collection(COLL)
client.create_collection(COLL, vectors_config=models.VectorParams(
    size=384, distance=models.Distance.COSINE))
client.create_payload_index(COLL, "doc_id", models.PayloadSchemaType.KEYWORD)

def match(key, value):
    return models.FieldCondition(key=key, match=models.MatchValue(value=value))

def stored_version(doc_id):
    points, _ = client.scroll(COLL, scroll_filter=models.Filter(must=[match("doc_id", doc_id)]),
                              with_payload=["version"], limit=10_000)
    return max((p.payload["version"] for p in points), default=None)

def apply_change(doc_id, version, chunks):
    """Make the index hold exactly this version of doc_id, unless it already holds a newer one."""
    held = stored_version(doc_id)
    if held is not None and version <= held:
        return f"skip  {doc_id} v{version} (index holds v{held})"
    vecs = model.encode(chunks)
    now = datetime.now(timezone.utc).isoformat()
    client.upsert(COLL, [models.PointStruct(
        id=str(uuid.uuid5(uuid.NAMESPACE_URL, f"{doc_id}#v{version}#{i}")),   # version-aware id
        vector=vecs[i].tolist(),
        payload={"doc_id": doc_id, "version": version, "chunk_index": i, "text": text,
                 "content_hash": hashlib.sha256(text.encode()).hexdigest(),
                 "embed_model": EMBED_MODEL, "indexed_at": now})
        for i, text in enumerate(chunks)], wait=True)                      # 1. new version in
    client.delete(COLL, points_selector=models.FilterSelector(filter=models.Filter(
        must=[match("doc_id", doc_id)], must_not=[match("version", version)])),
        wait=True)                                                         # 2. all others out
    return f"apply {doc_id} v{version} ({len(chunks)} chunks)"

def count(doc_id):
    return client.count(COLL, count_filter=models.Filter(must=[match("doc_id", doc_id)])).count

Now feed it three events for the pricing page: version 6 (three chunks), version 7 (two chunks), and then a retry of version 6 that a flaky worker delivers late:

v6 = ["Pro plan: USD 49 per month. Includes 600 requests per minute per API key and 5 team seats.",
      "Pro plan limits apply per API key; burst traffic above the limit returns HTTP 429.",
      "Pro plan add-ons: extra team seats at USD 8 per seat."]
v7 = ["Pro plan: USD 49 per month. Includes 1,000 requests per minute per API key and 5 team seats.",
      "Pro plan limits apply per API key; burst traffic above the limit returns HTTP 429."]

for version, chunks in [(6, v6), (7, v7), (6, v6)]:     # the last event is a late retry of v6
    print(apply_change("pricing", version, chunks), "| chunks now:", count("pricing"))

Predict: how many chunks does the page have after each event? And what would the late retry have done without the version check?

apply pricing v6 (3 chunks) | chunks now: 3
apply pricing v7 (2 chunks) | chunks now: 2
skip  pricing v6 (index holds v7) | chunks now: 2

Why it works

• Version-aware ids give v7's chunks new ids, so for the few milliseconds between the two requests both versions exist, and there is never a moment with neither. Deleting first leaves a gap. Overwriting in place, with ids derived from the chunk position alone (a uuid5 of pricing#0, pricing#1, …), also avoids the gap as long as step 2 still runs: the slots past the new end still carry the old version, so step 2 deletes them; without it, a document that shrinks keeps orphans (Embedding Pipeline works that way). Version-aware ids add one property: no id ever names two texts, so anything that stored point ids, such as evaluation labels or citation logs, never points at the wrong chunk.
• The version comes from the source: a revision number, a CMS version counter or a CDC log position. It never comes from the indexer's clock (Move 2).
• The check runs before model.encode, so a skipped event costs a scroll, not an embedding call.
• It keeps exactly one version per document. Step 2 deletes every other version, which is right when a new version takes effect the moment it is published. For versions published ahead of their effective date, or when users ask about the past, Move 5 keeps several versions with validity windows, and step 2 becomes "close the previous version's window".

⚠️ Check-then-write is not atomic. Two workers handling v6 and v7 of the same page at once can interleave. Route every event for one doc_id to the same worker (for example, make doc_id the queue's partition key), so one document's changes apply one at a time. Qdrant (since v1.16) also has a conditional upsert: with update_filter, an existing point is overwritten only if it matches the filter. In the default upsert mode, points whose ids do not exist yet are inserted regardless. update_mode="update_only" (since v1.17) blocks those inserts, but then a legitimately longer new version loses its extra chunks. Either way it guards points, not documents (Embedding Pipeline runs both cases).

Provenance on every point. The payload above records what you need later:

Move 5 adds the validity dates. A change to metadata alone goes through set_payload with a doc_id filter, as shown in Metadata & Filtering, Move 5.

Your turn. An editor saves the rate-limit guide three times in 40 seconds (v21, v22, v23). Your queue delivers v23, then v21, then v22. What does apply_change do with each, and how many embedding calls are spent? Could you do better when an event carries the page's full text?

Move 4: Make deletes stick

Symptom: a page deleted at the source is still cited. Or, worse, a customer whose data you erased finds their support ticket in an answer after a restore.

A delete has three enemies. It may never arrive, which Move 2 covered. It may arrive before an older update. And there are copies you forgot. Start with the second one, using the hook's legacy page, and say that this time the pipeline does hear about the delete:

LEGACY = ["Pro Legacy plan: 300 requests per minute per API key.",
          "Upgrade to a current plan for higher limits."]
apply_change("legacy-plans", 3, LEGACY)                   # revision 3, indexed long ago

def delete_document(doc_id):
    client.delete(COLL, points_selector=models.FilterSelector(
        filter=models.Filter(must=[match("doc_id", doc_id)])), wait=True)

delete_document("legacy-plans")                           # the page is deleted at the source
print(apply_change("legacy-plans", 3, LEGACY),            # a late retry of its last update
      "| chunks now:", count("legacy-plans"))

Predict: the page was deleted at the source, and then a delayed retry of its last update (revision 3) arrives. What prints?

apply legacy-plans v3 (2 chunks) | chunks now: 2

The fix is a tombstone: a durable record, in your pipeline's own database, that a document was deleted at a given version. (Qdrant and Lucene use the same word for their internal deletion markers; this one is yours and outlives the index.)

TOMBSTONES = {}     # doc_id -> version at deletion; a durable table in production, never a dict

def delete_document(doc_id, version):
    TOMBSTONES[doc_id] = version
    client.delete(COLL, points_selector=models.FilterSelector(
        filter=models.Filter(must=[match("doc_id", doc_id)])), wait=True)

def apply_change_checked(doc_id, version, chunks):
    if version <= TOMBSTONES.get(doc_id, -1):
        return f"skip  {doc_id} v{version} (deleted at v{TOMBSTONES[doc_id]})"
    return apply_change(doc_id, version, chunks)

delete_document("legacy-plans", 4)                       # the delete event carries revision 4
print(apply_change_checked("legacy-plans", 3, LEGACY), "| chunks now:", count("legacy-plans"))

skip  legacy-plans v3 (deleted at v4) | chunks now: 0

A tombstone has to outlive every older copy that could come back: queued and retried events, backups, and any export a rebuild might read. A rebuild from the live source cannot bring back a deleted page; a rebuild from last month's data-lake export can, unless it checks the tombstones.

Deleted is not erased

When you delete a point in Qdrant, it disappears from results at once. But on disk it stays in its segment, marked as deleted, until the vacuum optimizer rewrites that segment (Vector Database Architecture measured this). By default, a segment is vacuumed only when at least 20% of its vectors are deleted and it holds at least 1,000 vectors (optimizer docs). One customer's 40 chunks in a large segment may never cross 20%. Lucene-based engines behave similarly: deleted documents are dropped when segments merge.

Snapshots copy those files as they are. On Qdrant 1.19.1, a snapshot we took after deleting a point still held the point's text and vector in its segment files, and its text in the collection's write-ahead log, which snapshots include. After we forced a vacuum, the segment was clean, but the log still held the original write: Qdrant drops a log segment only after later writes have filled it (32 MB by default). A rebuild into a fresh collection held neither. So every snapshot taken until the data physically leaves the collection is another copy.

Then there are restores. Watch what one does:

apply_change_checked("ticket-4471", 1, ["J. Roe reports invoice 88213 was charged twice; "
                                        "refund requested to the card on file."])
snap = client.create_snapshot(COLL)                      # last night's backup
delete_document("ticket-4471", 2)                        # today: the customer asks for erasure
print("after delete :", count("ticket-4471"))

client.recover_snapshot(COLL, location=f"file:///qdrant/snapshots/{COLL}/{snap.name}",
                        priority=models.SnapshotPriority.SNAPSHOT)   # a disk fails; restore
print("after restore:", count("ticket-4471"))

for doc_id, version in TOMBSTONES.items():               # restore runbook: re-apply every delete
    delete_document(doc_id, version)
print("after replay :", count("ticket-4471"))
client.delete_snapshot(COLL, snap.name)

The file:// path is read by the Qdrant server itself (here, the Docker image's snapshot directory), as in Advanced Qdrant.

Predict: how many chunks of the ticket are searchable after each step?

after delete : 0
after restore: 1
after replay : 0

Every copy, with its own clock

A RAG system copies a document's text into more places than the vector index. Each copy needs a way for the delete to reach it and has its own moment when the data is really gone:

Vectors count as copies. Morris et al. (EMNLP 2023) showed that an inversion method recovered 92% of 32-token inputs exactly from their GTR-base embeddings. Do not treat an embedding as an anonymized form of the text.

Soft and hard delete are two steps, not two philosophies. A soft delete hides the document at once (in this lesson's schema, by closing its validity window, Move 5), so a mistake can be undone during a grace period, like a CMS trash that keeps pages 14 days. The hard delete follows when the grace period ends (Move 6's sweep). An erasure request skips the grace period.

Deadlines: what the rules say, and what you build

⚠️ This section is engineering, not legal advice. These are examples of rules you may meet:

• The EU GDPR gives people a right to erasure "without undue delay" (Article 17). Article 12(3) requires telling them what action was taken "within one month of receipt of the request", extendable "by two further months where necessary".
• The UK regulator's guidance says that backup data which cannot be overwritten at once should be put "beyond use" (ICO, right to erasure).
• California's CCPA regulations let a business delay deletion on archived or backup systems "until the archived or backup system relating to that data is restored to an active system or is next accessed or used for a sale, disclosure, or commercial purpose" (11 CCR § 7022).

Which rules apply to you, and whether a vector marked as deleted or an old backup counts as erased, are questions for your privacy counsel. Engineering's job is to know every copy and to prove when each one is gone.

Now the arithmetic. Say your privacy team's rule is "no copy anywhere 30 days after the request". A request arrives on day 0 at 10:00, and the index delete runs within the hour. Snapshots are taken nightly at 01:00 and kept 35 days. The delete does not take the ticket off the disk, so the snapshots of day 1, day 2 and every night after still hold it, and each lives 35 days. Two steps close this:

1. Take the data off the disk on a schedule you control. An erasure request triggers a rebuild into a fresh collection behind an alias (Advanced Qdrant), copying the remaining vectors rather than re-embedding them, and the old collection is dropped with its segments and log. (Relying on vacuum instead means checking, per erasure, that the segment was rewritten and the log has moved on.) Say the rebuild finishes on day 2, so the day-2 snapshot at 01:00 is the last one that holds the ticket.
2. Make that snapshot expire in time. Kept 35 days, it lives until day 37 at 01:00, almost a week too long; even 30 days misses by 39 hours. Kept 28 days, it goes on day 30 at 01:00, nine hours inside the deadline. Or delete the snapshots taken before the rebuild and give up those restore points.

The logs and caches need their own check, and until those snapshots expire, a restore from one of them still needs the tombstone replay.

Move 5: Validity is a time window, checked when the query runs

Symptom: the answer quotes a superseded policy, or one that has not taken effect yet (the hook's 1,500).

In the CMS, the October price list is really version 8 of the pricing page, scheduled for 1 October. The old sync indexed it as a separate page and dropped its date. The fix follows the rule from Metadata & Filtering: validity is a rule, so it is a filter. Store effective_from and effective_to on every chunk, and filter on them with the query's own time:

from collections import Counter

POLICIES = "policies"
if client.collection_exists(POLICIES):
    client.delete_collection(POLICIES)
client.create_collection(POLICIES, vectors_config=models.VectorParams(
    size=384, distance=models.Distance.COSINE))
for field in ("effective_from", "effective_to"):                 # payload indexes first
    client.create_payload_index(POLICIES, field, models.PayloadSchemaType.DATETIME)

OPEN = "9999-12-31T00:00:00Z"            # explicit "no end yet": a missing field would fail closed
VERSIONS = [  # (version, effective_from, effective_to, text)
    (6, "2025-01-01T00:00:00Z", "2026-09-25T09:12:00Z", "Pro plan: 600 requests per minute per API key."),
    (7, "2026-09-25T09:12:00Z", "2026-10-01T00:00:00Z", "Pro plan: 1,000 requests per minute per API key."),
    (8, "2026-10-01T00:00:00Z", OPEN,                   "Pro plan: 1,500 requests per minute per API key."),
]
client.upsert(POLICIES, [models.PointStruct(
    id=v, vector=model.encode(text).tolist(),
    payload={"doc_id": "pricing", "version": v, "text": text,
             "effective_from": start, "effective_to": end})
    for v, start, end, text in VERSIONS], wait=True)

def in_force(as_of):
    """Validity is evaluated when the query runs, so nothing has to flip at midnight."""
    return models.Filter(must=[
        models.FieldCondition(key="effective_from", range=models.DatetimeRange(lte=as_of)),
        models.FieldCondition(key="effective_to", range=models.DatetimeRange(gt=as_of))])

query = model.encode("What is the rate limit on the Pro plan?")
for as_of in ["2026-08-15T12:00:00Z", "2026-09-25T11:00:00Z", "2026-10-02T09:00:00Z"]:
    hits = client.query_points(POLICIES, query=query, query_filter=in_force(as_of), limit=3).points
    print(as_of[:10], [(h.payload["version"], h.payload["text"][10:31]) for h in hits])

Predict: the query asks for up to three hits. How many come back for each date, and which versions?

2026-08-15 [(6, '600 requests per minu')]
2026-09-25 [(7, '1,000 requests per mi')]
2026-10-02 [(8, '1,500 requests per mi')]

Why dates beat a status flag

A status field (current, superseded) set by the ingestion job is right only when written. At midnight on 1 October the flags go wrong until a job flips them: a flip at 02:00 leaves two hours of old prices, and a failed flip leaves them wrong for days. A date window compared with the query's time is right at every instant. Keep status for states that have no date, such as drafts and withdrawn pages. Metadata & Filtering's in-force filter checks status and the dates together. That works if a scheduled version is written as current when it is published and the dates alone decide when it applies.

Three details make the windows trustworthy:

1. Always write both dates. A chunk with no effective_to never matches the filter, so it fails closed: it drops out of the answers instead of giving a wrong one, and a coverage check that counts in-force points reports it as missing (Move 6). Hence the explicit far-future date for "no end yet", in line with the missing-field rules of Metadata & Filtering, Move 8.
2. Convert to UTC using the policy's own time zone. "Valid from 1 October" in Oslo means 2026-09-30T22:00:00Z (summer time, UTC+2). Stored naively as 2026-10-01T00:00:00Z, it goes live two hours late in Oslo. In New York, midnight is 04:00Z, so the same mistake turns a new policy on four hours early.
3. Publishing a new version must close the old window. If the publisher forgets, both versions are in force from 1 October. Check for that:

def overlaps(as_of):
    """Documents with more than one version in force at as_of. Should always be empty."""
    points, _ = client.scroll(POLICIES, scroll_filter=in_force(as_of),
                              with_payload=["doc_id", "version"], limit=10_000)
    versions = {(p.payload["doc_id"], p.payload["version"]) for p in points}
    return {d: n for d, n in Counter(d for d, _ in versions).items() if n > 1}

client.set_payload(POLICIES, payload={"effective_to": OPEN}, points=[7], wait=True)  # the bug
print(overlaps("2026-10-02T09:00:00Z"))

{'pricing': 2}

Run this check after every publish and every night, for "now" and for the next scheduled change date. Its count should be zero. It is the direct way to answer "how many policies have more than one version in force?"

Keep superseded versions or delete them?

• Keep them, with their windows, if users ask about the past ("what was covered on the date of my claim?") or auditors ask what the assistant could have said on a given day. You pay in storage, and a retrieval path that forgets the filter surfaces old versions.
• Delete them when their window closes if nobody asks about the past: a smaller index and no filter to forget. Scheduled versions still need their window until they take effect. Move 3's apply_change is the special case where a version takes effect the moment it is published, so it can delete the old one at write time.

The "as of" time is a query parameter. It defaults to now, and it comes from the question when the question names a date. Extracting that date is a job for Query Understanding & Intent.

Rules and preferences

A boost cannot do a rule's job. It only shifts scores, so a superseded chunk that matches the question better can still win. And a date filter used for a preference throws away the best old answer. When two current sources genuinely disagree, the prompt has to show the conflict (Context Augmentation, Move 1).

Your turn. A customer asks on 25 September: "Was the Pro limit higher in August than it is now?" What does retrieval have to do?

Move 6: Expire what must go, sweep what slipped through

Symptom: chat transcripts that should be gone after 30 days are still searchable. Zombie pages surface months later. Or, one night, a cleanup job deletes most of the index.

An expiry time is two jobs

An expiry answers two different questions, with different deadlines:

1. Stop showing it. This can be exact to the second. Set effective_to to the expiry time, and Move 5's filter hides the item the moment it expires.
2. Stop storing it. A sweep deletes the item. That happens minutes to a day later, and then the on-disk copies and backups follow (Move 4).

A built-in TTL changes who runs job 2, not whether it exists. As of September 2026:

• Qdrant has no built-in TTL. A maintainer's suggestion on GitHub is the two jobs above: an indexed payload field, a range filter, and a periodic job that deletes expired points.
• Milvus has a collection-level TTL (collection.ttl.seconds) and, in Milvus 3.0, a per-entity TTL field. Expired entities stop appearing in results at once and are "physically removed from storage on the next compaction cycle — typically within 24 hours" (docs).
• Weaviate has had object TTL since v1.36. Deletion runs on a schedule you configure, and whether queries hide expired objects that are not deleted yet is a separate setting (docs).
• MongoDB TTL indexes are cleaned by a background task every 60 seconds, with no guarantee of immediate deletion (docs).
• Elasticsearch removed per-document TTL in 5.0. ILM and data stream retention delete whole indices, so you expire data by writing it into time-based indices.

When TTL is the wrong tool

An expiry makes nothing fresh. On a page you already get change events for, a TTL either does nothing or expires correct content that simply had not changed. Use expiry for:

• retention rules: transcripts after 30 days, tickets after the retention period;
• sources without any change signal, such as a crawled website. Here the TTL means "re-fetch after", not "delete after", and the content hash decides whether to re-embed;
• caches.

⚠️ Sliding expiration (also called time-to-idle: the clock resets on every read) keeps popular items forever and never refreshes them. If a cache uses it, add an absolute maximum age.

The reconciliation sweep

The sweep is Move 2's slow, complete signal turned into a job: it compares a complete listing of the source with what the index holds and sorts every difference into a category. Data Pipeline & Indexing's sync removes vanished documents on every run; the sweep is the same diff for a pipeline driven by events. Continuing from Move 4's code:

KB = {f"kb-{n:03d}": 1 for n in range(100)}
for doc_id, version in KB.items():
    apply_change_checked(doc_id, version, [f"Help article {doc_id}, version {version}."])
LISTING = {"pricing": 7, **KB}            # the source's complete listing: doc_id -> current version
# Drift that no event reported: one page deleted, one edited at the source
del LISTING["kb-017"]
LISTING["kb-042"] = 2

def index_state(only=None):
    """doc_id -> versions and embedding models of the points matching `only` (default: all)."""
    state, offset = {}, None
    while True:
        points, offset = client.scroll(COLL, scroll_filter=only, limit=1_000, offset=offset,
                                       with_payload=["doc_id", "version", "embed_model"])
        for p in points:
            s = state.setdefault(p.payload["doc_id"], {"versions": set(), "models": set()})
            s["versions"].add(p.payload["version"])
            s["models"].add(p.payload["embed_model"])
        if offset is None:
            return state

def reconcile(listing, searchable=None, max_delete_share=0.05):
    """searchable: the filter every query applies (None: this index has no validity windows)."""
    stored = index_state()                                          # everything on disk
    held = stored if searchable is None else index_state(searchable)  # what a query can return
    report = {
        "missing":   sorted(listing.keys() - held.keys()),   # at the source, not searchable
        "zombie":    sorted(held.keys() - listing.keys()),   # gone at the source, still searchable
        "stale":     sorted(d for d in listing.keys() & held.keys()
                            if max(held[d]["versions"]) < listing[d]),
        "mixed":     sorted(d for d, s in held.items() if len(s["versions"]) > 1),
        "old_model": sorted(d for d, s in stored.items() if s["models"] != {EMBED_MODEL}),
    }
    if len(report["zombie"]) > max_delete_share * len(held):
        raise RuntimeError(f"refusing to delete {len(report['zombie'])} of {len(held)} "
                           "documents: is the listing complete?")
    for doc_id in report["zombie"]:
        delete_document(doc_id, max(held[doc_id]["versions"]))
    return {k: v for k, v in report.items() if v}

print(reconcile(LISTING))
truncated = dict(list(LISTING.items())[:40])      # the list API failed on page 3 of 5
try:
    reconcile(truncated)
except RuntimeError as e:
    print(e)

Predict: what does the first call report? And what happens on the second call, when the listing came back incomplete?

{'zombie': ['kb-017'], 'stale': ['kb-042']}
refusing to delete 60 of 100 documents: is the listing complete?

This demo index keeps one version per document and has no validity windows (Move 3), so everything stored is searchable and searchable stays None. In an index that keeps versions with windows (Move 5), pass searchable=in_force(now), so that the categories describe what a query can return now:

• a chunk written without effective_to shows up as missing;
• a superseded version with a closed window is history, not mixed, and mixed becomes Move 5's overlap check;
• a document whose window you closed on purpose is soft-deleted and waiting for its grace period, not a zombie;
• old_model still counts every stored point, because as-of queries search old versions too.

There, retire a zombie by closing its window and setting a delete_after date at the end of the grace period, instead of deleting it; the retention sweep below does the hard delete.

Graph each count per night; each should trend to zero. Together they are Move 1's coverage metric.

Guards for anything that deletes in bulk:

• Prove the listing is complete: pagination reached the end, and the count is plausible against yesterday's.
• Cap the share deleted per run (a delete brake), and ask a human above it.
• Soft-delete first where you have validity windows (close the window) and hard-delete after a grace period, so a mistake is a hidden page for a day rather than a lost one. The demo index has no windows, so the cap and the completeness check are its only brakes.

The sweep's interval is also the worst-case lifetime of a zombie the fast path missed: a nightly sweep would have removed the hook's legacy page within a day instead of never.

The retention half of expiry is a filter-based delete. Points carry a delete_after date, and a job removes everything past it:

def sweep_expired(now):
    """Retention: physically delete what must no longer be stored. The query filter hides it earlier."""
    expired = models.Filter(must=[models.FieldCondition(
        key="delete_after", range=models.DatetimeRange(lte=now))])
    n = client.count(COLL, count_filter=expired).count
    client.delete(COLL, points_selector=models.FilterSelector(filter=expired), wait=True)
    return n

client.create_payload_index(COLL, "delete_after", models.PayloadSchemaType.DATETIME)
client.set_payload(COLL, payload={"delete_after": "2026-09-20T00:00:00Z"},
                   points=models.Filter(must=[match("doc_id", "kb-003")]), wait=True)
print(sweep_expired("2026-09-25T00:00:00Z"), count("kb-003"))

1 0

Set effective_to no later than delete_after, so the item leaves the answers on time even if the sweep runs late. And apply the same retention at the source, or tombstone what the sweep removes: otherwise the next reconcile finds the document missing and ingests it again.

Move 7: Re-embed on a schedule, not by surprise

Symptom: your embedding vendor announces that your model will be retired next quarter. Or a chunker fix means everything must be re-chunked. Or the sweep's old_model count is not zero.

Not every change needs the same work:

Because every point carries embed_model, "how much of the index is still on the old model?" is a count, not an investigation, and outside a migration it should be zero in every collection. Two models with the same dimension in one collection degrade results quietly (Vector Embeddings).

Freshness during a migration

A full re-embed is long, and your ordinary edits keep arriving the whole time. Here is the arithmetic, with explicit assumptions:

# Assumptions for the exercise (not measurements):
CORPUS = 12_000_000        # chunks to re-embed with the new model
CAPACITY = 300             # chunks/s the embedding workers sustain in total
INCREMENTAL = 5            # chunks/s of ordinary edits, on average (432,000 per day)
SLA_MIN = 15               # freshness target for ordinary edits, minutes

alone = CORPUS / CAPACITY
print(f"backfill alone: {alone:,.0f} s = {alone / 3600:.1f} h")

# One FIFO queue: an edit queued right after the backfill waits for all of it
print(f"FIFO: an edit waits up to {alone / 3600:.1f} h (target {SLA_MIN} min)")

# Two queues: edits first, the backfill takes what is left
left = CAPACITY - INCREMENTAL
shared = CORPUS / left
print(f"priority: the backfill takes {shared / 3600:.1f} h; edits keep their usual lag")

# Changes made during the backfill must reach the new index before the switch.
# While they replay, edits keep coming: 5/s for the old index and 5/s more for the log.
replay = INCREMENTAL * shared
catch_up = replay / (left - INCREMENTAL)
print(f"replay from the recorded position: {replay:,.0f} chunks = {catch_up / 60:.1f} min")

Predict: if the backfill and the edits share one first-in, first-out queue, how late can an ordinary edit be?

backfill alone: 40,000 s = 11.1 h
FIFO: an edit waits up to 11.1 h (target 15 min)
priority: the backfill takes 11.3 h; edits keep their usual lag
replay from the recorded position: 203,390 chunks = 11.7 min

Four rules keep a migration fresh:

1. Give edits priority over the backfill. Use separate queues, or rate-limit the backfill below capacity.
2. Record where the change log was when the backfill started (a CDC offset, or a watermark with an overlap) and replay from there into the new collection before the switch: here about 203,000 chunks in 11.7 minutes. The alternative is to write every change to both collections during the backfill. That is safe only when the backfill and the live changes for one doc_id go through the same per-document queue, one writer per document (Embedding Pipeline); otherwise a backfill write can land after a newer edit or bring back a deleted page. Rule 3 is the second line of defence.
3. Guard the new collection against the backfill itself. The backfill may read a page at 01:00 and write it at 09:00, after an 08:00 edit already reached the new collection. Move 3's version check makes the older write a skip.
4. Send deletes to both collections, and keep sending them to the old one for as long as you keep it for rollback (Move 4).

Then plan backwards from a retirement date: start at least one backfill, one evaluation on your labeled set (Retrieval Metrics) and one rollback window before it. Rolling out behind an evaluation gate is covered in Production Deployment & Optimization.

Final round: no label on the incident

Real incidents do not say which move they need. Read the symptom, then pick.

Challenge 1: the morning the index shrank

At 02:10 the nightly reconcile job deleted 71% of the documents. The CMS listing API had returned pages 1 and 2 of 7 before its token expired, and the job treated the partial list as complete. Which move, what is the recovery, and what trap waits in it?

Challenge 2: the fix that did not show

Editors correct a wrong phone number on a support page. The pipeline dashboard shows the change applied in 3 minutes. Five hours later, users asking the usual question still get the old number, while a slightly reworded question gets the new one.

Challenge 3: the ticket that came back

After the quarterly full rebuild from last month's data-lake export, a customer whose account was erased three weeks ago finds their support ticket quoted in an answer.

Challenge 4: the first hour of January

A policy change takes effect at midnight on 1 January 2027 in Oslo. From 00:00 to 01:00 local time, answers quote the old deductible. From 01:00 they quote the new one. Nothing ran at 01:00.

Cheat sheet: symptom → move → cost

How would you measure it on your own data?

• Canaries: create and delete a canary page at the source every 10 minutes. Chart the time to appear and the time to disappear in production search.
• Lag distribution: record source-change time and indexed_at per change. Chart p50, p95, the maximum and the oldest unapplied change per day.
• Nightly reconciliation counts over what a query can return: missing, zombie, stale, mixed (with validity windows, the overlap check for now), old_model, plus the overlap check for the next scheduled change date. Each should sit at or near zero.
• Drills: once a quarter, erase a test subject and check every copy in the table of Move 4, including a snapshot taken after the erasure: search its files for the subject's text. Restore last night's backup into a scratch collection and count what the tombstone replay removes.
• Answer quality on changed facts: keep 30–50 labeled questions whose answers changed recently, such as prices, limits and policies. Rerun them after each change and weekly. That is the number users feel (Evaluation & Quality Metrics).

Before moving on, take one source you index today and write down its freshness contract: its edit and delete targets, its change signal and that signal's blind spot, where its copies live, and which date fields it needs. Then run the hook's question on your own system: one fact that changed yesterday, one page deleted last month, one change scheduled for next month.

Next: Embedding Pipeline for throughput, caching by content hash and re-embedding at scale; Caching Strategies for invalidation; Infrastructure & Security for access control, PII and deletion obligations across the whole system.