Vectorless RAG

The interest rate that lived in Schedule B

A finance analyst asks the contracts assistant: "How much interest do we owe if we pay an invoice 20 days late?" The assistant is a classic RAG pipeline. Each numbered section of every vendor agreement is one chunk, prefixed with its heading. The chunks are embedded with the small open model all-MiniLM-L6-v2, and the top 3 go into the prompt. Here is a 14-section slice of one agreement and the retrieval step:

import numpy as np
from sentence_transformers import SentenceTransformer

# One entry per numbered section of a services agreement: id -> (title, text)
CONTRACT = {
    "1":   ("Definitions", "\"Fees\" means the amounts payable under Section 7. \"Service\" means the hosted analytics platform described in Section 2."),
    "2":   ("Services", "Provider will make the Service available to Customer and its authorised users for the Term."),
    "3":   ("Service levels", "Provider will meet the availability targets and pay the service credits set out in Schedule A."),
    "4":   ("Customer data", "Customer owns Customer Data. Provider processes it only to provide the Service and deletes it within 30 days after termination."),
    "5":   ("Confidentiality", "Each party keeps the other party's confidential information secret for five years after disclosure."),
    "6":   ("Term and renewal", "The initial Term is 24 months. It renews for 12-month periods unless either party gives notice 60 days before the end of the current Term."),
    "7.1": ("Invoicing", "Provider invoices monthly in arrears. Invoices are due 30 days after the invoice date."),
    "7.2": ("Disputed invoices", "Customer may dispute an invoice in good faith within 15 days of receipt. Disputed amounts are not due until the dispute is resolved."),
    "7.3": ("Late payment", "Undisputed amounts not paid when due accrue interest at the rate set out in Schedule B, from the due date until the date of payment."),
    "8":   ("Suspension", "If undisputed Fees are more than 45 days overdue, Provider may suspend the Service after 10 days' written notice."),
    "9.1": ("Termination for convenience", "Either party may end this Agreement at any time by giving 90 days' written notice."),
    "9.2": ("Termination for cause", "Either party may terminate if the other commits a material breach that is not cured within 30 days of written notice."),
    "A":   ("Schedule A: Service levels", "Monthly availability target: 99.9%. Service credit: 5% of the monthly Fees for each 0.1% below target, capped at 30%."),
    "B":   ("Schedule B: Rates and charges", "Overdue amounts: 12% per year, accruing daily on the basis of a 365-day year. Reinstatement after suspension: EUR 500. Additional storage: EUR 0.02 per GB per month."),
}
IDS = list(CONTRACT)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
VECS = model.encode([f"{title}: {text}" for title, text in CONTRACT.values()])  # unit-length rows

def dense_top(query, k=3):
    scores = VECS @ model.encode(query)          # cosine, since the model normalizes its output
    order = np.argsort(-scores)[:k]
    return [(IDS[i], round(float(scores[i]), 3)) for i in order]

question = "How much interest do we owe if we pay an invoice 20 days late?"
for rank, (sid, score) in enumerate(dense_top(question, k=6), start=1):
    print(rank, sid, CONTRACT[sid][0], score)

Predict first: which three sections reach the prompt? Where does Schedule B, the section with the actual rate, land?

1 7.1 Invoicing 0.585
2 7.3 Late payment 0.466
3 7.2 Disputed invoices 0.425
4 B Schedule B: Rates and charges 0.36
5 8 Suspension 0.359
6 A Schedule A: Service levels 0.268

With those three sections, the model has two options. It can say the rate is in Schedule B, which it does not have. Or it can supply a plausible rate from its training data, a number that appears in many contracts on the web and in none of yours. The first answer is useless to the analyst. The second is wrong, and nothing in it looks wrong. The right answer is 12% a year, accruing daily: on a EUR 10,000 invoice, 20 days late costs 10,000 × 0.12 × 20/365 = EUR 65.75.

Why tuning this retriever does not fix it. Here k = 4 would scoop up Schedule B. In the real agreement, with 400 sections and a dozen schedules, a lead of 0.0003 over the suspension clause is not a plan. BM25 does worse: SQLite's BM25 ranks 7.3 first and never returns Schedule B at all, because Schedule B shares no word with the question. The question resembles the pointer (7.3), not the target (Schedule B). What connects them is the document's own structure: a reference from one section to another. In the failure taxonomy of RAG Architecture & Implementation, this pipeline's failure is not retrieved: its candidate pool is just the top 3, so Schedule B never enters it. With a pool of 50 and a cut-off of 3, the same miss would be ranked out: in the pool, below the cut-off. With BM25 it is not retrieved at any depth.

A lawyer does not answer this by searching for similar sentences. She opens the contents page, turns to 7.3, reads "Schedule B", and turns to Schedule B. That is one of several ways to retrieve without an embedding index, and they are the subject of this lesson:

Vectorless RAG: retrieval-augmented generation whose retrieval step uses no embedding index: no vectors computed by a model, dense or sparse. It matches words (BM25), queries structured data (SQL, APIs), lets a model navigate a document's structure (a table of contents, a file tree), or skips retrieval and puts the whole source in the prompt.

"Vectorless" describes the index, not the absence of models: an LLM may write the SQL or choose the section to read at question time. The test is what the index stores. Learned sparse retrieval such as SPLADE fails it: its inverted index holds term weights computed by a neural encoder, so changing the model means re-encoding the corpus, exactly as with dense embeddings (Sparse vs Dense Retrieval). BM25 passes: its index holds term counts taken straight from the text. You can draw a BM25 document as a sparse vector too, but no model computed it. A table of contents passes as well, even when an LLM wrote its section summaries: those are text, not vectors.

Here is the map. Each move starts from a symptom:

This lesson assumes the classic pipeline from RAG Architecture & Implementation, BM25 at the level of Sparse vs Dense Retrieval, and the tool loop from Agentic RAG Systems. The code is Python 3.12 with numpy, sentence-transformers and the standard library's sqlite3, whose SQLite build includes the FTS5 full-text engine. One benchmark also uses rank_bm25, and the one real API call needs the anthropic package and a key.

Move 0: Before any index, does it fit in the prompt?

Symptom: the source is small, and someone is already designing a chunker.

The whole agreement above is about 460 tokens (counted with tiktoken's cl100k_base; your provider's tokenizer will differ a little). Put all of it in the prompt and the model sees 7.3 and Schedule B side by side. Nothing can be ranked out. This is the degenerate case of vectorless RAG: retrieval returns everything.

RAG Architecture & Implementation works through long context against RAG in detail: the bill, prompt caching, the studies on accuracy at length, and access control. Here is the short version, with round numbers. Assume a 60-page contract at about 500 tokens a page, so 30,000 tokens. Add 300 tokens of instructions and question, and price input at USD 3 per million tokens:

• Uncached: 30,300 × 3 / 10⁶ = USD 0.091 per question, or USD 90.90 per 1,000.
• Cached, with cache reads billed at 10% of the input price (Anthropic's standard rate as of September 2026; some models bill less, down to 2.5%, per its pricing page): 300 + 0.1 × 30,000 = 3,300 token-equivalents, or USD 9.90 per 1,000. That is the same input as a six-chunk RAG prompt (6 × 500 + 300).

Output tokens cost the same in every design and are left out. Cache writes cost extra (1.25 × the input price for Anthropic's 5-minute cache, for example) every time the cache has gone cold, so the cached line holds only when questions about the same document keep arriving.

Put the source in the prompt when all of these hold:

• it fits the window with room to spare;
• the volume is low, or the cache stays warm;
• every user may see all of it (an instruction such as "do not reveal Schedule D" is not access control: by then the text is already in the prompt);
• you have measured accuracy at that prompt length on your own questions.

Your turn: change one requirement. Legal loads all 2,000 vendor agreements and asks "which agreements let us terminate for convenience with less than 60 days' notice?" Is "put the contract in the prompt" still the right design?

Move 1: When BM25 alone is enough

Symptom: users type error codes, clause numbers, part numbers and product names. The vocabulary is controlled, or you already run a search engine or a database, and an embedding pipeline is more machinery than the problem needs.

BM25-only RAG is the classic pipeline with a lexical index as its only retriever. The scoring function, its k1 and b parameters and analyzers are taught in Sparse vs Dense Retrieval. This move is about what changes when BM25 is all you have.

You probably already own a BM25 engine:

• Elasticsearch, OpenSearch and Solr use BM25 by default.
• SQLite's FTS5 extension has a bm25() ranking function with k1 = 1.2 and b = 0.75 fixed. It returns lower values for better matches (FTS5 docs).
• PostgreSQL's built-in ts_rank "ranks vectors based on the frequency of their matching lexemes" (docs); it is not BM25. BM25 in Postgres comes from extensions such as ParadeDB's pg_search (AGPL-3.0).

Engines tokenize and score differently, so switching engines changes your rankings. Re-run your labeled queries after any switch.

The first attempt: pass the question straight in

Here is a support knowledge base for a card terminal, indexed with FTS5:

import re, sqlite3

ARTICLES = {
    "kb-101": ("Error E-104: card read failed",
               "Clean the chip reader with a cleaning card. If E-104 repeats on three different cards, replace the reader module."),
    "kb-102": ("Error E-4471: firmware signature mismatch",
               "The terminal rejects firmware 2.3.0 images signed with the old key. Download 2.3.1 from the partner portal and install it from USB."),
    "kb-103": ("Error E-4417: printer cover open",
               "Close the paper cover until it clicks. The error clears within five seconds."),
    "kb-104": ("Blank display after power-on",
               "If the display shows no image but the power LED is green, hold the power key for 15 seconds to force a restart."),
    "kb-105": ("Installing firmware from USB",
               "Copy the image to a FAT32 USB drive, insert it and choose Settings > Update. Do not unplug the terminal during the update."),
    "kb-106": ("Contactless payments declined",
               "Check that NFC is enabled under Settings > Payments. Decline code 65 means the card requires a PIN."),
    "kb-107": ("Battery drains overnight",
               "Disable Wi-Fi scanning in standby. A healthy battery keeps 80% of its charge after 12 hours in standby."),
    "kb-108": ("Receipt paper jams",
               "Use 57 mm thermal rolls with an outer diameter of at most 40 mm. Larger rolls jam at the cutter."),
}
db = sqlite3.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE kb USING fts5(doc_id UNINDEXED, title, body)")
db.executemany("INSERT INTO kb VALUES (?, ?, ?)", [(i, t, b) for i, (t, b) in ARTICLES.items()])

for raw in ["What does E-4471 mean?", "How do I update the firmware?", "how do I update the firmware"]:
    try:
        print(repr(raw), "->", db.execute("SELECT doc_id FROM kb WHERE kb MATCH ?", (raw,)).fetchall())
    except sqlite3.OperationalError as e:
        print(repr(raw), "-> OperationalError:", e)

Predict: kb-105 answers the second and third questions. What do the three raw strings return?

'What does E-4471 mean?' -> OperationalError: no such column: 4471
'How do I update the firmware?' -> OperationalError: fts5: syntax error near "?"
'how do I update the firmware' -> []

Build the query yourself, and decide what "found" means

STOP = {"a", "an", "and", "the", "is", "it", "i", "my", "do", "does", "how", "what", "why", "when",
        "to", "of", "on", "in", "for", "with", "after", "can", "we", "you", "me", "this"}
IDENT = re.compile(r"\b[A-Za-z]{1,4}-\d{2,}\b")           # E-4471, INV-2291

def words(text):
    return re.findall(r"\w+", text.lower())

def search(question, k=3, min_coverage=0.5):
    """User text -> a safe FTS5 query -> top k rows, or [] when there is no real evidence."""
    idents = IDENT.findall(question)
    terms = list(dict.fromkeys(t for t in words(question) if t not in STOP))
    if idents:                                    # identifiers must match, each as a quoted phrase
        match = " AND ".join(f'"{i}"' for i in idents)
    elif terms:                                   # otherwise any content word may match
        match = " OR ".join(f'"{t}"' for t in terms)
    else:
        return []
    rows = db.execute("SELECT doc_id, title, body, bm25(kb) FROM kb WHERE kb MATCH ? "
                      "ORDER BY rank LIMIT ?", (match, k)).fetchall()
    if rows and not idents:                       # "found" = most content words in the top hit
        covered = set(terms) & set(words(rows[0][1] + " " + rows[0][2]))
        if len(covered) / len(terms) < min_coverage:
            return []
    return rows

for q in ["What does E-4471 mean after an update?",
          "What does E-9999 mean?",
          "The screen stays black when I turn it on",
          "How do I pair the terminal with a Bluetooth printer?",
          "Which thermal paper rolls fit?"]:
    hits = search(q)
    print(q, "->", [(d, round(s, 3)) for d, _, _, s in hits] or "no evidence")

It prints:

What does E-4471 mean after an update? -> [('kb-102', -1.43)]
What does E-9999 mean? -> no evidence
The screen stays black when I turn it on -> no evidence
How do I pair the terminal with a Bluetooth printer? -> no evidence
Which thermal paper rolls fit? -> [('kb-108', -4.883), ('kb-103', -1.052)]

Three decisions sit in those twenty lines.

Quote every term. Wrapping each token in double quotes turns any user text into a valid FTS5 query. That holds because tokens from \w+ never contain a double quote; if you ever quote raw text, double every " inside it, as FTS5's syntax requires. It is the search-engine version of binding SQL parameters: user text never becomes query syntax.

Identifiers must match. With every word ORed together, the first question ranks kb-105 (-2.176, "Installing firmware from USB") above kb-102 (-1.832). In eight articles, "update" is as rare as "4471", and kb-105 says "update" twice. In a 12,000-article base "update" would be common and the ranking would probably come out right, but a design that depends on corpus statistics breaks the day the corpus changes. When the user names an error code, only documents containing that code count. And when no document contains it (E-9999), the honest answer is "no evidence", not the nearest article about some other error.

Decide what "found" means. An OR query matches any document that shares any word. Without the coverage check, the Bluetooth question returns kb-103, "Error E-4417: printer cover open" (-1.772), on the single word "printer", and the model may explain how to close the printer cover. The check asks whether the top hit contains at least half of the question's content words. Because a BM25 score is a sum of per-term contributions, you can always see which terms matched. The raw score is no substitute: it has no fixed range and moves with the query and the corpus (Sparse vs Dense Retrieval). Treat 0.5 as a starting point to tune on labeled questions.

Measure it by slice

Here is a small labeled set, four questions per slice, scored by hit rate@1 (the share of questions whose top result is the relevant article). The same questions were run through search and through MiniLM dense retrieval over the same eight articles:

LABELED = [  # (slice, question, the article that answers it)
    ("identifiers", "What does E-4471 mean?", "kb-102"),
    ("identifiers", "E-104 keeps coming back", "kb-101"),
    ("identifiers", "terminal shows E-4417", "kb-103"),
    ("identifiers", "customer card declined with code 65", "kb-106"),
    ("paraphrases", "The screen stays black when I turn it on", "kb-104"),
    ("paraphrases", "tap to pay stopped working", "kb-106"),
    ("paraphrases", "the device is dead every morning", "kb-107"),
    ("paraphrases", "paper gets stuck when printing", "kb-108"),
]
KB_IDS = list(ARTICLES)
KB_VECS = model.encode([f"{title}: {body}" for title, body in ARTICLES.values()])  # MiniLM, from the opening

for _, question, gold in LABELED:
    hits = search(question)
    lexical = hits[0][0] if hits else "no evidence"
    dense = KB_IDS[int(np.argmax(KB_VECS @ model.encode(question)))]
    print(f"{question!r:43} BM25 {lexical:12} dense {dense}  gold {gold}")

'What does E-4471 mean?'                    BM25 kb-102       dense kb-102  gold kb-102
'E-104 keeps coming back'                   BM25 kb-101       dense kb-101  gold kb-101
'terminal shows E-4417'                     BM25 kb-103       dense kb-103  gold kb-103
'customer card declined with code 65'       BM25 kb-106       dense kb-106  gold kb-106
'The screen stays black when I turn it on'  BM25 no evidence  dense kb-104  gold kb-104
'tap to pay stopped working'                BM25 no evidence  dense kb-106  gold kb-106
'the device is dead every morning'          BM25 no evidence  dense kb-107  gold kb-107
'paper gets stuck when printing'            BM25 no evidence  dense kb-103  gold kb-108

Eight articles is a toy. At scale, dense retrieval starts confusing look-alike codes; Sparse vs Dense Retrieval measures that on a 2,000-chunk catalog. Look at how each one fails. BM25-only abstains on every paraphrase, which is safe but useless. Dense answers "paper gets stuck when printing" with the printer-cover article, which is wrong and confident. BM25-only is the right design when your real questions are mostly the first slice. Label a week of real questions by slice before you decide.

Speed is not the reason

Vectorless is often sold as "faster". Here is a measurement of retrieval alone, over 50,000 synthetic documents of 80 words each:

import random, sqlite3, time
from itertools import accumulate
import numpy as np
from rank_bm25 import BM25Okapi

rng = random.Random(0)
vocab = [f"w{i}" for i in range(30_000)]
cum = list(accumulate(1 / (r + 1) for r in range(len(vocab))))   # Zipf: a few common words, many rare ones
docs = [rng.choices(vocab, cum_weights=cum, k=80) for _ in range(50_000)]
queries = [rng.choices(vocab[100:5000], k=5) for _ in range(20)]

fts = sqlite3.connect(":memory:")
fts.execute("CREATE VIRTUAL TABLE t USING fts5(body)")
fts.executemany("INSERT INTO t VALUES (?)", ((" ".join(d),) for d in docs))
bm = BM25Okapi(docs)

def median_ms(fn):
    times = []
    for q in queries:
        t0 = time.perf_counter()
        fn(q)
        times.append(time.perf_counter() - t0)
    return round(1000 * float(np.median(times)), 1)

print("FTS5, inverted index:", median_ms(lambda q: fts.execute(
    "SELECT rowid FROM t WHERE t MATCH ? ORDER BY rank LIMIT 10", (" OR ".join(q),)).fetchall()), "ms")
print("rank_bm25, scores every document:", median_ms(lambda q: np.argsort(-bm.get_scores(q))[:10]), "ms")

rank_bm25 computes a score for every document on every query, so its time grows with the corpus; here it was roughly 12 to 20 times slower than FTS5 in every run. It is fine for prototypes. An inverted index reads only the postings of the query's terms. Local dense retrieval is in the same few-millisecond range, and an LLM call takes seconds. Retrieval speed is rarely why you go vectorless. The reasons are exact-term behaviour, no embedding pipeline to build and re-run when the model changes, and scores you can explain term by term.

What BM25-only costs you

• Paraphrases miss. The fixes each cost something. Search-time synonym lists need curating; Elastic recommends applying synonyms at search time because the sets can then change without reindexing. An LLM rewrite of the question into keywords costs one model call per question (Query Understanding & Intent). A dense retriever fused with BM25 is Hybrid Retrieval Systems, at which point you are no longer vectorless.
• Identifier formats. FTS5's default tokenizer turns E-4471 and e 4471 into the same two tokens, e and 4471, while E4471 is one different token, e4471. So the phrase query "E-4471" finds "e 4471" but not "E4471". Index a joined copy of every identifier and normalize queries the same way (Sparse vs Dense Retrieval, Move 3).
• Languages. BM25 matches only strings that the question and the document share. A German question finds nothing in an English knowledge base.
• Statistics move. Adding documents changes every term's IDF. The coverage check does not depend on raw scores, so it survives this better than a score threshold would.

Your turn: change one requirement. The knowledge base grows to 12,000 articles in nine languages. Support logs show 60% of questions quote an error code and 40% describe a symptom in the customer's language. What do you build?

Move 2: Ask the database

Symptom: "How many P1 tickets are open for the Berlin team?" The answer is a count over every row. Top-k retrieval returns k rows, so it hands the model a sample and invites it to count the sample.

When the knowledge is already a table (tickets, orders, balances, stock levels), the retriever is a query. There are two designs, and they carry different risks:

Start with fixed tools for the questions that make up most of your traffic. Add text-to-SQL, fenced in, for the long tail.

Fixed tools: the SQL never changes

import os, random, sqlite3, tempfile

# The ingestion job builds the database. The assistant will only ever read it.
path = os.path.join(tempfile.mkdtemp(), "support.db")
rng = random.Random(7)
TICKETS = [(f"T-{1000 + n}", rng.choice(["Berlin", "Munich", "Lisbon"]), rng.choice(["P1", "P2", "P3"]),
            rng.choice(["open", "open", "closed"]), f"2026-09-{rng.randint(1, 24):02d}") for n in range(60)]
rw = sqlite3.connect(path)
rw.execute("CREATE TABLE tickets(id TEXT PRIMARY KEY, team TEXT, priority TEXT, status TEXT, opened TEXT)")
rw.execute("CREATE TABLE salaries(employee TEXT, amount INTEGER)")
rw.executemany("INSERT INTO tickets VALUES (?, ?, ?, ?, ?)", TICKETS)
rw.execute("INSERT INTO salaries VALUES ('Ana', 91000)")
rw.commit()
rw.close()

def query(sql, params=()):
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)     # read-only connection
    try:
        return conn.execute(sql, params).fetchall()
    finally:
        conn.close()

# Allowed values come from the data (refresh them when it changes). Column names come from this code.
ALLOWED = {col: {v for (v,) in query(f"SELECT DISTINCT {col} FROM tickets")}
           for col in ("team", "priority", "status")}

def count_tickets(team, priority, status="open"):
    """Tool: the model fills in three arguments; the SQL text never changes."""
    for col, value in (("team", team), ("priority", priority), ("status", status)):
        if value not in ALLOWED[col]:
            raise ValueError(f"{col} must be one of {sorted(ALLOWED[col])}, got {value!r}")
    sql = "SELECT COUNT(*) FROM tickets WHERE team = ? AND priority = ? AND status = ?"
    return query(sql, (team, priority, status))[0][0]

print(count_tickets("Berlin", "P1"))
try:
    count_tickets("Berlin", "P1", status="Open")
except ValueError as e:
    print("ValueError:", e)

# The same kind of value pasted into the SQL text instead of bound as a parameter:
team = "Berlin' OR '1'='1"
print(query(f"SELECT COUNT(*) FROM tickets WHERE team = '{team}' AND priority = 'P1' AND status = 'open'")[0][0])

5
ValueError: status must be one of ['closed', 'open'], got 'Open'
30

The Berlin team has 5 open P1 tickets. The validation error is written for the model: it names the allowed values, so the next call can succeed. Send it back as a tool error, as in Agentic RAG Systems.

The last line shows why values are bound, never pasted. A model may copy whatever the user typed into its arguments, and a prompt injection can put anything there. Pasted into the SQL text, Berlin' OR '1'='1 changes the query's logic. AND binds tighter than OR, so the query counts every Berlin ticket plus every open P1 of every team: 30. Bound with ?, the same string is just a team name that matches nothing, and the allowed-values check rejects it before that.

API tools follow the same rules. Take order_status(order_id) in a customer chatbot. The model supplies only the order id, and the tool checks its format. The customer comes from the logged-in session, never from the model's arguments; otherwise an injected prompt can ask about someone else's order. The tool calls one fixed endpoint with a timeout, and it returns different messages for "no such order for this customer" and "the order service did not answer: say so, do not guess". The model chooses the arguments. Your code owns the endpoint, the identity and the error handling.

Text-to-SQL, fenced in

For the questions no tool covers, let the model write SQL. Then assume the SQL is hostile, whoever wrote the prompt. Below, the same ten model outputs run through two executors. The naive one is the check many tutorials show: the statement must start with SELECT. The guarded one adds four layers, three of them enforced inside the database engine:

import re, time

ALLOWED_TABLES = {"tickets"}                                   # lower case
ALLOWED_FUNCTIONS = {"count", "sum", "avg", "min", "max", "round", "lower", "upper",
                     "coalesce", "like", "date", "strftime"}   # the LIKE operator counts as one

def authorizer(action, arg1, arg2, db_name, trigger):
    """SQLite calls this while it compiles the statement, once per table/column read, function, write..."""
    if action == sqlite3.SQLITE_SELECT:
        return sqlite3.SQLITE_OK
    if action == sqlite3.SQLITE_READ:                   # arg1 = table (case as the query wrote it)
        return sqlite3.SQLITE_OK if arg1.lower() in ALLOWED_TABLES else sqlite3.SQLITE_DENY
    if action == sqlite3.SQLITE_FUNCTION:               # arg2 = function name
        return sqlite3.SQLITE_OK if arg2.lower() in ALLOWED_FUNCTIONS else sqlite3.SQLITE_DENY
    return sqlite3.SQLITE_DENY                          # writes, PRAGMA, ATTACH: everything else

def run_guarded(model_output, max_rows=50, seconds=1.0):
    sql = re.sub(r"(?i)^```(?:sql)?\s*|\s*```$", "", model_output.strip())  # models add fences
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)               # 1. read-only file
    conn.set_authorizer(authorizer)                                        # 2. allow-list
    deadline = time.monotonic() + seconds
    conn.set_progress_handler(lambda: time.monotonic() > deadline, 10_000) # 3. time limit
    try:
        rows = conn.execute(sql).fetchmany(max_rows + 1)                   # 4. row cap
        return {"rows": rows[:max_rows], "truncated": len(rows) > max_rows}
    finally:
        conn.close()

def run_naive(model_output):
    if not model_output.strip().upper().startswith("SELECT"):
        raise ValueError("only SELECT is allowed")
    conn = sqlite3.connect(path)
    try:
        return {"rows": conn.execute(model_output).fetchall()}
    finally:
        conn.close()

BERLIN = "SELECT COUNT(*) FROM tickets WHERE team = 'Berlin' AND priority = 'P1' AND status = 'open'"
MODEL_OUTPUTS = {
    "plain":     BERLIN,
    "fenced":    f"```sql\n{BERLIN}\n```",
    "cte":       "WITH p1 AS (SELECT * FROM tickets WHERE priority = 'P1') "
                 "SELECT COUNT(*) FROM p1 WHERE team = 'Berlin' AND status = 'open'",
    "like":      "SELECT COUNT(*) FROM Tickets WHERE team LIKE 'ber%' AND priority = 'P1' AND status = 'open'",
    "delete":    "DELETE FROM tickets WHERE status = 'closed'",
    "salaries":  "SELECT employee, amount FROM salaries",
    "two stmts": "SELECT 1; DROP TABLE tickets",
    "runaway":   "SELECT COUNT(*) FROM tickets a, tickets b, tickets c, tickets d, tickets e, tickets f",
    "age":       "SELECT ROUND(AVG(julianday('2026-09-25') - julianday(opened)), 1) FROM tickets",
    "wrong":     BERLIN.replace("'open'", "'Open'"),
}
for name, sql in MODEL_OUTPUTS.items():
    results = []
    for run in (run_naive, run_guarded):
        if run is run_naive and name == "runaway":
            results.append("(not run)")
            continue
        try:
            results.append(str(run(sql)["rows"]))
        except Exception as e:
            results.append(f"{type(e).__name__}: {e}")
    print(f"{name:9}  naive   -> {results[0]}\n{'':9}  guarded -> {results[1]}")

Predict: which outputs does each executor get wrong? Look for two kinds of mistake: a safe query rejected, and an unsafe one allowed.

plain      naive   -> [(5,)]
           guarded -> [(5,)]
fenced     naive   -> ValueError: only SELECT is allowed
           guarded -> [(5,)]
cte        naive   -> ValueError: only SELECT is allowed
           guarded -> [(5,)]
like       naive   -> [(5,)]
           guarded -> [(5,)]
delete     naive   -> ValueError: only SELECT is allowed
           guarded -> DatabaseError: not authorized
salaries   naive   -> [('Ana', 91000)]
           guarded -> DatabaseError: access to salaries.employee is prohibited
two stmts  naive   -> ProgrammingError: You can only execute one statement at a time.
           guarded -> ProgrammingError: You can only execute one statement at a time.
runaway    naive   -> (not run)
           guarded -> OperationalError: interrupted
age        naive   -> [(13.1,)]
           guarded -> OperationalError: not authorized to use function: julianday
wrong      naive   -> [(0,)]
           guarded -> [(0,)]

Each layer covers what the others miss:

1. A read-only connection. SQLite's mode=ro makes the engine refuse writes ("attempt to write a readonly database"), whatever the SQL says.
2. An allow-list. The authorizer is consulted for every table and column the statement reads, and for every function it calls. Denying by default also blocks PRAGMA and ATTACH.
3. A time limit. Here it is a progress handler. In a server database, it is a statement timeout.
4. A row cap that says so. truncated: True must reach the model, or "list every open P1 ticket" gets a silently partial answer.

On PostgreSQL or MySQL, the boundary is a database role that can only SELECT from allow-listed tables or views, plus a statement timeout. A read-only transaction is a useful second layer, but the PostgreSQL docs call it "a high-level notion of read-only that does not prevent all writes to disk". And SELECT … INTO "creates a new table" (docs): a statement that starts with SELECT and writes.

Per-tenant rows need a policy the query cannot change. PostgreSQL's row-level security usually reads the tenant from a session setting that the application sets for each request. That guards against application bugs, not against model-written SQL: one generated SELECT that calls set_config('app.tenant', …) changes the setting and reads another tenant's rows, even in a read-only transaction (Infrastructure & Security measures it). Key the policy on the login instead. Your code opens the connection as the requesting tenant's own database login, chosen from the verified identity, and the policy compares each row's tenant with session_user, which the query cannot change. If the table keeps a session-setting policy for your application too, limit that policy to the application's role with TO: PostgreSQL combines permissive policies with OR, so an unscoped one reopens the hole. Create views WITH (security_invoker = true) (PostgreSQL 15+); otherwise a view reads its tables with its owner's rights, and one owned by the table owner or a superuser can read every row. When rows belong to individual users rather than tenants, prefer fixed tools that take the user from the session, as above.

The query that runs and is wrong

The last row is the dangerous one. status = 'Open' passes every guard, returns 0, and the assistant tells a manager there are no open P1 tickets in Berlin. There are five. Safety layers stop harmful queries. They do nothing for wrong ones.

What helps, in order of payoff:

• A schema card in the prompt. Give each table and column a one-line meaning, plus the allowed values of low-cardinality columns. Generate those values with SELECT DISTINCT, as ALLOWED does above, and refresh them when the data changes.
• Views with honest names. For example, open_tickets with the status filter already applied, instead of a raw table with a status column the model must spell correctly.
• Examples. Put five to ten real questions with their correct SQL in the prompt.
• Suspicion in code. A count of 0, or an empty list, on a question that presupposes rows should trigger a check of the filter values before the answer ships. Show the SQL next to the answer so a human can spot a wrong filter.

Measure it with execution accuracy: over labeled (question, correct SQL) pairs, the share where the generated SQL returns the same result as the correct SQL. Compare result sets, not SQL strings, because many different queries are right. Order matters only when the question asks for an order. Expect it to be lower than demos suggest. When the authors of Spider 2.0 (Lei et al., ICLR 2025) published their tasks from real enterprise databases in late 2024, their best agent setup, based on o1-preview, solved 21.3% of them, against 91.2% on the older Spider 1.0 and 73.0% on BIRD. Scores have climbed since: on 26 September 2026 the leaderboard for Spider 2.0-Snow, a variant that ships "well-prepared database metadata and documentation", listed its top four systems at 94–97%, all heavily engineered agents. Your schema arrives without that preparation; the schema card and the views above are how you add it. Messy schemas are where text-to-SQL gets hard, so measure execution accuracy on your own questions rather than trusting a leaderboard.

Your turn. A team lead asks, "Which Berlin P1 tickets opened this week are still open?" count_tickets cannot list tickets. What do you add, and what must its result tell the model?

Move 3: Let the model read the table of contents

Symptom: long, structured documents such as contracts, annual reports, regulations and manuals. The answer is found by following the structure: a cross-reference, a schedule, a defined term, a numbered section. The text that holds the answer does not resemble the question. That is the opening failure.

The idea is the lawyer's move, automated. PageIndex (VectifyAI, MIT licence) describes itself as "Vectorless, Reasoning-based RAG" and splits retrieval into two steps: "generate a tree-structure index for each document", then "agentically search that tree with LLM reasoning". Its README and source on GitHub (checked in September 2026, when the pageindex package on PyPI was at 0.2.x) show what that means in practice:

• In the default local indexing mode, called "flash", the tree is extracted from the document's layout without an LLM. An LLM then writes a summary for each node and runs a pass that splits over-long sections into subsections; both steps can be switched off. The README's summary: the tree "is extracted from the document layout without an LLM; the index model only summarizes and refines it". For documents whose layout yields no structure, a "standard" mode builds the tree with the model.
• Each node carries a title, a node_id, a page range (start_index, end_index), an optional summary and child nodes.
• Retrieval is a tool loop. The chat model calls a tool that returns the outline without any text, picks sections, and calls another tool that returns their pages.
• The open-source local mode handles text-based PDFs. Its hosted cloud service adds OCR for scanned documents.

The research idea is older. MemWalker (Chen et al., 2023) "first processes the long context into a tree of summary nodes"; given a query, "the model navigates this tree in search of relevant information, and responds once it gathers sufficient information". In the vocabulary of Agentic RAG Systems, this is retrieval as a tool, where the tool is "read section X" and the section reference is the bridge entity.

question ──▶ model reads the OUTLINE: ids and titles, no section text
               │ picks a node id
               ▼
            your code checks the id ──▶ not in the outline: the error, with the
               │                        valid ids, goes to the model's next call
               ▼ known id
            your code returns that section's text ──▶ evidence {id: text}
               │ model: follow a reference? read another section? answer?
               ▼
            ... until it answers or the step budget runs out

Here is the loop over the agreement from the opening. As in Agentic RAG Systems, a stub plays the model with fixed rules, so everything runs offline. It picks the outline title that shares the most words with the question, then follows every "Section" or "Schedule" reference it has not read yet. A second stub, careless_model, makes a mistake real models make: it asks for "Schedule B" by name instead of by id.

import re

REF = re.compile(r"\b(?:Section|Schedule) ([0-9]+(?:\.[0-9]+)?|[A-Z])\b")

def outline():
    """What the model sees first: ids and titles, no section text."""
    return "\n".join(f"[{sid}] {title}" for sid, (title, _) in CONTRACT.items())

def words_in(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def stub_model(question, toc, evidence, error=None):
    """Plays the LLM. Returns {"action": "read", "node": id} or {"action": "answer"}.
    It only names ids from the outline, so it never needs the error message."""
    if not evidence:                                        # step 1: choose from the outline
        titles = dict(re.findall(r"\[(\S+)\] (.+)", toc))
        best = max(titles, key=lambda sid: len(words_in(question) & words_in(titles[sid])))
        return {"action": "read", "node": best}
    for text in evidence.values():                         # then follow references
        for ref in REF.findall(text):
            if ref not in evidence and f"[{ref}]" in toc:   # only ids the outline lists
                return {"action": "read", "node": ref}
    return {"action": "answer"}

def navigate(question, model, max_steps=4):
    toc, evidence, trace, error = outline(), {}, [], None
    known = {sid.lower(): sid for sid in CONTRACT}          # compare ids case-insensitively
    for step in range(1, max_steps + 1):
        decision = model(question, toc, evidence, error)
        if decision["action"] == "answer":
            trace.append((step, "answer", "from " + ", ".join(evidence)))
            return evidence, trace
        node, error = known.get(str(decision["node"]).lower()), None
        if node is None:                                    # never trust an id the model invented
            error = f"unknown node {decision['node']!r}; valid ids: {', '.join(CONTRACT)}"
            trace.append((step, "error", error))
            continue                                        # the error goes back on the next call
        evidence[node] = CONTRACT[node][1]
        trace.append((step, f"read {node}", CONTRACT[node][0]))
    trace.append(("-", "stop", "budget spent: answer from the evidence and name the gap"))
    return evidence, trace

def careless_model(question, toc, evidence, error=None):
    """Like stub_model, but asks for "Schedule B" by name until an error shows the real ids."""
    decision = stub_model(question, toc, evidence, error)
    if decision.get("node") == "B" and error is None:
        return {"action": "read", "node": "Schedule B"}
    return decision

for llm in (stub_model, careless_model):
    print(llm.__name__)
    evidence, trace = navigate("How much interest do we owe if we pay an invoice 20 days late?", llm)
    for row in trace:
        print(*row, sep="  |  ")

stub_model
1  |  read 7.3  |  Late payment
2  |  read B  |  Schedule B: Rates and charges
3  |  answer  |  from 7.3, B
careless_model
1  |  read 7.3  |  Late payment
2  |  error  |  unknown node 'Schedule B'; valid ids: 1, 2, 3, 4, 5, 6, 7.1, 7.2, 7.3, 8, 9.1, 9.2, A, B
3  |  read B  |  Schedule B: Rates and charges
4  |  answer  |  from 7.3, B

The answer the evidence supports: "Interest of 12% a year accrues daily from the due date [7.3][B]. On a EUR 10,000 invoice paid 20 days late, that is 10,000 × 0.12 × 20/365 = EUR 65.75." Two sections went into the prompt, and they were the right two.

The careless run costs one extra step and still gets there, because navigate treats a bad id like a failed tool call in Agentic RAG Systems: it logs the error and sends it to the model on the next call, with the ids that would have worked. Drop the error instead, and the model sees exactly the same input again, so it is likely to make the same choice again, step after step, until the budget runs out.

Predict: what does the stub open first for "Can the provider suspend the service if we pay 50 days late?" What would a capable model open?

The real call

With a real model, each step is one call. The sketch below uses Anthropic's Messages API (structured outputs, prompt caching). It needs the anthropic package and an API key, and it plugs into navigate in place of stub_model. The model id is an example as of September 2026; pick a current one.

import json
import anthropic

MODEL = "claude-opus-5-5"        # example id as of 2026-09; pick a current model
client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

INSTRUCTIONS = ("You find evidence in one contract. You see its outline and the sections read so far. "
                "Reply with the next section to read, or answer when the evidence is enough. "
                "Follow references such as 'Schedule B' or 'Section 4'.")

def claude_model(question, toc, evidence, error=None):
    """Same signature as stub_model: one model call per navigation step."""
    schema = {"type": "object",
              "properties": {"action": {"type": "string", "enum": ["read", "answer"]},
                             "node": {"type": "string", "enum": list(CONTRACT) + ["none"]},
                             "reason": {"type": "string"}},
              "required": ["action", "node", "reason"], "additionalProperties": False}
    read_so_far = "\n".join(f"[{i}] {text}" for i, text in evidence.items()) or "(nothing yet)"
    rejected = f"\n\nYour last choice was rejected: {error}" if error else ""
    response = client.messages.create(
        model=MODEL, max_tokens=4000,
        system=[{"type": "text", "text": INSTRUCTIONS},
                {"type": "text", "text": "Outline:\n" + toc,
                 "cache_control": {"type": "ephemeral"}}],      # the same outline on every call
        messages=[{"role": "user",
                   "content": f"Question: {question}\n\nSections read:\n{read_so_far}{rejected}"}],
        output_config={"effort": "low",
                       "format": {"type": "json_schema", "schema": schema}})
    if response.stop_reason != "end_turn":                      # max_tokens, refusal, ...
        raise RuntimeError(f"navigation step stopped: {response.stop_reason}")
    decision = json.loads(next(b.text for b in response.content if b.type == "text"))
    return {"action": decision["action"], "node": decision["node"]}

Three details carry the weight:

• The schema's enum lists the real node ids. With structured outputs the reply must match the schema, so the model cannot invent an id such as "7.3a". The docs list exceptions: a refusal, a reply cut off at max_tokens (the code checks stop_reason for both), and enum values whose capitalization may differ from the schema. That is why navigate compares ids case-insensitively and still checks every id in code, which also protects you when you switch models or providers.
• The outline is the stable prefix. It is re-sent on every step of every question about this document, so it is marked for caching. Caching starts only once the prefix passes the model's minimum cacheable length, which is hundreds to a few thousand tokens depending on the model. This toy outline is far below that; a 400-section outline is not.
• Low effort. Choosing the next section is a small decision. Spend reasoning on the answer, and measure whether lower effort hurts node selection before you keep it.

The bill: calls, not searches

Navigation replaces one retrieval with several model calls. Here is the arithmetic for a 300-page agreement, with round-number assumptions (the price is an assumption for the exercise, not a quote):

PRICE_IN, PRICE_OUT = 3.00, 15.00  # USD per million input / output tokens
CACHE_READ = 0.10                  # cache reads at 10% of input, or less on some models (writes ignored)
TTFT, SPEED = 0.6, 100             # s to first token, output tokens per s (short prompts)
DOC = 300 * 500                    # 300 pages at ~500 tokens per page
OUTLINE = 400 * 20                 # 400 headings at ~20 tokens each (id + title)
SECTION, CHUNK = 700, 500          # tokens per section read / per retrieved chunk
BASE = 300                         # instructions + question
STEP_OUT, ANSWER_OUT = 80, 250     # output tokens per navigation decision / per answer

def usd_per_1k(fresh, cached=0, out=0):
    return 1000 * (fresh * PRICE_IN + cached * PRICE_IN * CACHE_READ + out * PRICE_OUT) / 1e6

def navigation(reads, cache_outline=False):
    calls = reads + 1                                  # one call per read, one to answer
    fresh = cached = 0
    for c in range(calls):
        fresh += BASE + c * SECTION                    # sections read so far ride along
        if cache_outline:
            cached += OUTLINE
        else:
            fresh += OUTLINE
    out = reads * STEP_OUT + ANSWER_OUT
    return fresh + cached, usd_per_1k(fresh, cached, out), calls * TTFT + out / SPEED

rows = {
    "chunk RAG, 6 chunks":       (BASE + 6 * CHUNK, usd_per_1k(BASE + 6 * CHUNK, 0, ANSWER_OUT),
                                  TTFT + ANSWER_OUT / SPEED),
    "navigate, 2 reads":         navigation(2),
    "navigate, 2 reads, cached": navigation(2, cache_outline=True),
    "navigate, 4 reads":         navigation(4),
    "whole document":            (BASE + DOC, usd_per_1k(BASE + DOC, 0, ANSWER_OUT), None),
    "whole document, cached":    (BASE + DOC, usd_per_1k(BASE, DOC, ANSWER_OUT), None),
}
print(f"{'design':26} {'input tokens':>12} {'USD/1k':>8} {'model s':>8}")
for name, (tokens, usd, seconds) in rows.items():
    print(f"{name:26} {tokens:12,} {usd:8.2f} {'measure' if seconds is None else f'{seconds:.1f}':>8}")

design                     input tokens   USD/1k  model s
chunk RAG, 6 chunks               3,300    13.65      3.1
navigate, 2 reads                27,000    87.15      5.9
navigate, 2 reads, cached        27,000    22.35      5.9
navigate, 4 reads                48,500   154.05      8.7
whole document                  150,300   454.65  measure
whole document, cached          150,300    49.65  measure

Two reads cost about 6.4 times a six-chunk RAG answer (87.15 against 13.65 USD per 1,000 questions) and take about twice as long before the answer is complete (5.9 s against 3.1 s of model time). A cached outline brings the cost to 1.6 times RAG. The whole document is the most expensive design uncached (454.65) and competitive cached (49.65). Its time is marked "measure" because time to first token grows with a 150,000-token prompt. So navigation fits questions where a wrong answer costs far more than nine cents, over documents too long, or too numerous, to send whole.

Where navigation breaks

⚠️ No usable structure. Scanned PDFs without a text layer, flat exports and wikis with three headings give the model nothing to navigate. Build the structure first (Document Processing), or chunk.

⚠️ Titles that lie. When "Miscellaneous" holds the liability cap, titles alone mislead the model. Node summaries help, at an LLM cost per node at indexing time. PageIndex's README puts its own local indexing at about USD 0.001 per page with its default index model, a vendor figure to check against your pages and prices.

⚠️ A wrong first turn. A navigator that opens the wrong chapter reads the wrong subtree. Errors compound across steps as they do across hops in Agentic RAG Systems: at 90% per step, three steps succeed 0.9³ ≈ 73% of the time.

⚠️ Answers spread everywhere. "List every notice obligation in this contract" sends the navigator into most sections. You pay long-context prices plus the extra calls. Extract the obligations into a table at ingest instead (Move 2).

⚠️ Many documents. Navigation works inside one document. Across 20,000 contracts you first need to pick the document: from metadata the user already chose, with BM25 over titles and descriptions, or with a dense search over document summaries. PageIndex's hosted service adds a file-level tree for this. "Vectorless per document" does not require "vectorless per corpus".

⚠️ A different path every run. Log the node ids per question, so you can replay a bad answer and see which turn went wrong.

Try the cheap fix first. When the only problem is that a chunk lost its heading ("90 days' written notice" without "Termination for convenience" above it), prefixing each chunk with its heading path fixes it for plain vector or BM25 retrieval (Smart Chunking). Navigation earns its extra calls when the answer needs a hop that no single chunk contains, like the one from 7.3 to Schedule B.

What the evidence says. PageIndex's README reports 98.7% accuracy on FinanceBench (Islam et al., 2023, questions over public companies' financial filings) and says this is "vastly outperforming vector-based RAG". That is a vendor's result for its own system. Treat it as a reason to run the comparison on your documents, not as the outcome of that comparison.

Measure it. Label 100 questions with the ids of the sections that hold their answers. Record:

• node recall: the gold sections the navigator opened, divided by all gold sections for the question, averaged over questions (recall@k with "opened" in place of "in the top k");
• answer accuracy;
• reads per question;
• cost per 1,000 questions;
• p95 latency.

Run the same set through chunk RAG with heading paths, and through the whole document if it fits. Ship navigation only where it wins by more than it costs.

Your turn. Trace "What happens to our data if we terminate for convenience?" through stub_model. Which sections does it read, and what is wrong with the answer it can give? What is the smallest number of model calls a good navigator needs?

Move 4: Search like a developer

Symptom: the corpus is a code base, a config repository or a log directory. It changes with every commit, questions name identifiers, and an index built last night is already out of date.

Coding agents often skip the index and search the files directly. Anthropic's Claude Code, for example, uses glob- and grep-style tools. Boris Cherny of the Claude Code team explained the choice on the Latent Space podcast (May 2025). His first reason was quality: early versions had used an embedding index, and agentic search "outperformed everything. By a lot." The evidence, in his words: "some internal benchmarks also, but mostly vibes", none of it published. Then the index itself: "the code drifts out of sync", the index becomes a security liability, and agentic search "sidesteps all of that… at the cost of latency and tokens". Cursor reported the opposite in November 2025 (Improving agent with semantic search). On its own benchmark of questions about codebases, its agent reached "on average 12.5% higher accuracy" (6.5% to 23.5%, depending on the model) when semantic search was available next to grep, and Cursor concluded that "the combination of these two leads to the best outcomes". These are two vendors describing their own products, and their accuracy claims, both from internal evaluations, point in opposite directions. They agree on the mechanism: grep is exact and always current, and it only finds the words someone thought to type.

The tools are small. Here are two, over an in-memory stand-in for a repository:

import re
from pathlib import PurePosixPath

REPO = {
    "payments/config.py": "WEBHOOK_MAX_ATTEMPTS = 5\nWEBHOOK_BACKOFF_SECONDS = [1, 5, 30, 120]\nCARD_RETRY_LIMIT = 3\n",
    "payments/webhooks.py": ("from payments.config import WEBHOOK_MAX_ATTEMPTS, WEBHOOK_BACKOFF_SECONDS\n\n"
                             "def deliver(event):\n"
                             "    for attempt in range(WEBHOOK_MAX_ATTEMPTS):\n"
                             "        if post(event):\n"
                             "            return True\n"
                             "        if attempt < len(WEBHOOK_BACKOFF_SECONDS):\n"
                             "            sleep(WEBHOOK_BACKOFF_SECONDS[attempt])\n"
                             "    dead_letter(event)\n"),
    "payments/cards.py": "from payments.config import CARD_RETRY_LIMIT\n\ndef charge(card):\n    # retry soft declines only\n    ...\n",
    "billing/retry.py": "# Retry policy for invoice emails\nMAX_RETRIES = 2\n",
    "notifications/mailer.py": "def send(msg):\n    throttle(per_minute=120)  # provider allows 2 mails/s\n    ...\n",
    "docs/runbook.md": "## Webhooks\nFailed webhook deliveries go to the dead-letter queue after the last attempt.\n",
}

def grep(pattern, path="", max_hits=5):
    """Regex search. One line per hit, 'path:line: text', capped like any tool result."""
    rx = re.compile(pattern, re.IGNORECASE)
    hits = [f"{p}:{n}: {line.strip()}"
            for p, text in REPO.items() if p.startswith(path)
            for n, line in enumerate(text.splitlines(), 1) if rx.search(line)]
    more = [f"... {len(hits) - max_hits} more hits; narrow the pattern"] if len(hits) > max_hits else []
    return hits[:max_hits] + more

def read(path, start=1, end=40):
    """Numbered lines of one file. Refuses anything outside the repository."""
    p = PurePosixPath(path)
    if p.is_absolute() or ".." in p.parts or path not in REPO:
        raise ValueError(f"no such file in repo: {path!r}")
    return [f"{n}: {line}" for n, line in enumerate(REPO[path].splitlines()[start - 1:end], start)]

# An agent's calls for "How often do we retry a failed payment webhook?"
for call in [lambda: grep("retr"), lambda: grep("webhook.*(retr|attempt)"),
             lambda: read("payments/webhooks.py", 3, 9)]:
    print("\n".join(call()), end="\n\n")

payments/config.py:3: CARD_RETRY_LIMIT = 3
payments/cards.py:1: from payments.config import CARD_RETRY_LIMIT
payments/cards.py:4: # retry soft declines only
billing/retry.py:1: # Retry policy for invoice emails
billing/retry.py:2: MAX_RETRIES = 2

payments/config.py:1: WEBHOOK_MAX_ATTEMPTS = 5
payments/webhooks.py:1: from payments.config import WEBHOOK_MAX_ATTEMPTS, WEBHOOK_BACKOFF_SECONDS
payments/webhooks.py:4: for attempt in range(WEBHOOK_MAX_ATTEMPTS):
payments/webhooks.py:8: sleep(WEBHOOK_BACKOFF_SECONDS[attempt])
docs/runbook.md:2: Failed webhook deliveries go to the dead-letter queue after the last attempt.

3: def deliver(event):
4:     for attempt in range(WEBHOOK_MAX_ATTEMPTS):
5:         if post(event):
6:             return True
7:         if attempt < len(WEBHOOK_BACKOFF_SECONDS):
8:             sleep(WEBHOOK_BACKOFF_SECONDS[attempt])
9:     dead_letter(event)

The first search is the trap. "retr" finds card retries (3) and invoice-email retries (2), and an agent that stops there answers "3" with confidence. The webhook code never says "retry": it says "attempts". The second search names webhooks, and the read shows what the constant's name hides. range(5) means five attempts: the first delivery plus four retries, after waits of 1, 5, 30 and 120 seconds, then the dead-letter queue. The answer came from reading code around a hit, not from the hit.

Where it goes wrong, and what the fix costs:

• Vocabulary. grep("rate.?limit") returns nothing, because mailer.py calls it throttle(per_minute=120). The agent must guess synonyms (throttle|per.?minute|quota finds it), and every guess is a model call. This is where a semantic index earns its upkeep.
• Floods. A common term matches thousands of lines. Cap the hits and say how many were cut, because every result is re-sent on each later model call (Agentic RAG Systems prices that).
• Reach. Confine read to the repository root (the .. check above), deny secrets files, and keep every tool read-only. Retrieved code is untrusted input: a comment can carry a prompt injection.
• Regex cost. Python's re backtracks, and a pattern such as (a+)+$ can take exponential time on the wrong input. Tools such as ripgrep use a regex engine that guarantees linear-time matching by default; otherwise, run each search with a timeout.

Measure it as task success on labeled questions, with tool calls and tokens per task beside it. Compare with and without a semantic search tool on your own repository, the way Cursor did on theirs.

Choosing: vectorless, vector, or both

Some claims about vectorless RAG are false, and they come up in design reviews:

• "It is faster." Local retrieval takes milliseconds either way; a hosted embedding API adds a network round trip. Navigation and grep add model calls, which take seconds.
• "It is more private." A self-hosted embedding model keeps text on your machines. A navigator or a grep agent sends document text to the LLM either way. Privacy depends on where things are deployed, not on whether there are vectors.
• "It is simpler." Each move above needs analyzers, schema cards, guards or document structure. The complexity moves; it does not vanish.
• "Vector indexes can't handle updates." They upsert one chunk at a time. The expensive event is a model change, which means re-embedding everything (Sparse vs Dense Retrieval). Live values belong in a database or an API, whichever retriever you use for text.
• "Only vectorless retrieval can be evaluated." Any retriever that returns ids can be scored with hit rate@k on labeled questions (Retrieval Metrics).

Combining them is the normal case. Give an agent search_kb (BM25 or hybrid), count_tickets, run_sql (guarded), read_section and grep, each with a description that says when to use it. Retrieval as a tool is exactly the loop of Agentic RAG Systems. Or route with rules when the shapes are easy to tell apart. Either way, measure tool-selection accuracy on labeled questions.

Final round: no label on the problem

Challenge 1: the delegation limit

An internal assistant answers questions over 12 finance policies of 20 to 80 pages each. "What is the approval limit for a department manager?" gets the answer "managers approve spend according to the delegation matrix", cited to section 5.4. The limit itself sits in a table in "Appendix 2: Delegation of authority", which section 5.4 references. About 200 questions a day. What do you try, in what order?

Challenge 2: the quarter that shrank

Finance asks the SQL assistant for last quarter's revenue. The number is 18% below the ledger. The generated SQL runs without error and filters on created_at. The ledger reports by invoice_date, and 18% of last quarter's invoices were created as drafts in the quarter before.

Challenge 3: the forum import

A BM25-only support bot answered error-code questions well. After 5,000 forum posts were imported into the same index, "What does E-4471 mean?" returns forum threads in which people paste the code while asking about something else, and the official article has dropped to third.

Cheat sheet: symptom → move → cost

Before moving on, take one assistant you know and sort 50 of its real questions by shape: single fact, exact identifier, aggregate, structural hop, fast-changing source. For each shape, name the cheapest retriever from this lesson that answers it correctly, what it gives up, and the one number you would track.

How would you measure it on your own data? Label 100–200 real questions with their shape, the expected answer, and the ids of the documents, rows or sections that hold it. Run your current system and the vectorless candidate side by side. Compare four numbers per shape: answer accuracy, the retrieval metric for that route (hit rate@k, execution accuracy or node recall), cost per 1,000 questions, and p95 latency. The Evaluation & Quality Metrics lessons cover the statistics.

Next: Graph Rag for when the relations between entities are the data, Hybrid Retrieval Systems for fusing BM25 with dense retrieval when paraphrases matter, and Agentic RAG Systems for the loop that ties the tools together.