Every RAG tutorial ends the same way: embed the corpus, embed the question, return the top ten chunks, stuff them into the prompt. It demos beautifully. Then it meets production, and the bill arrives.
The problem is not the model and usually not the embeddings. It is the contract. top_k = 10 answers the question "which ten chunks are most similar?" — and nobody actually wants to know that. What a caller wants is: give me exactly enough material to answer this, and do not spend more than N tokens doing it. Those are different questions, and the second one is the one that has a budget attached.
This post is about building for the second contract. It comes out of the retrieval layer I built for Hilum Tools, where the corpus is a multi-repository codebase and the caller is an autonomous agent that cannot look at what came back and say "hmm, that is not it".
Why top-k overpays
Three failures compound, and all three are invisible in a demo.
Similarity is not sufficiency. Ten chunks that each score 0.82 can be ten near-copies of the same paragraph. You have paid for ten and learned one thing. Meanwhile the one chunk that actually completes the answer sits at rank fourteen because it phrases the concept differently.
Chunk weight is unbounded. A "chunk" is a retrieval unit, not a cost unit. Ten chunks might be 800 tokens or 14,000 depending on how the splitter felt that day. The caller asked for ten of something and received an unpredictable invoice.
Rank ten is a guess about the future. k is chosen once, at build time, for all questions. "Where is AuthService defined?" needs one precise hit. "How does billing reconcile a failed webhook?" needs six files and a decision record. Serving both with the same k means overpaying for the first and starving the second.
The contract that fixes it
Return the smallest set of material sufficient to answer X, and stay under N tokens. If you cannot get there, say so.
Three parts, and the third is the one people skip.
The budget turns retrieval into a selection problem with a constraint, which means it can be optimised instead of tuned by feel. The sufficiency goal replaces similarity ranking with coverage: distinct facets of the question, not ten variations of the strongest one. The admission of failure matters more than it looks: an agent acting unsupervised cannot distinguish "nothing matched" from "nothing was searched" unless the answer tells it. Silence reads as absence, and absence reads as licence to invent.
Building it
1. Split where meaning changes, not at byte offsets
Fixed-size windows are the single largest source of retrieval noise. A 512-token window cuts a function in half, separates a code sample from the sentence explaining it, and orphans a table header from its rows. Every one of those becomes a chunk that is individually retrievable and individually useless.
Split on structure instead: function and class boundaries for code, heading boundaries for documentation, complete rows for tables. Then apply a size floor — a three-line chunk carries almost no signal and pollutes the index with noise that scores well on short queries. Where a natural unit exceeds the ceiling, split it and carry a header of parent context into each piece so a fragment still knows what it belongs to.
2. Score with more than one head
Pure semantic search has a specific, predictable blind spot: it is good at concepts and bad at strings. Identifiers, error codes, version numbers, config keys, and CLI flags are exactly the tokens a developer question hinges on, and they are exactly what an embedding smooths away. Ask a vector index for E_CONFLICT_1042 and it will happily return a paragraph about conflict resolution.
So run at least two heads and fuse them:
- Dense — an approximate-nearest-neighbour index (HNSW is the sane default) over embeddings, for conceptual matches and paraphrase.
- Sparse / lexical — BM25 or a learned sparse head, for exact identifiers and rare terms.
Fuse with reciprocal-rank fusion before you reach for anything cleverer. It needs no score calibration between heads, it is three lines of code, and it is very hard to beat with hand-tuned weights. Add a multi-vector head later if your material has several natural aspects — a symbol's signature, its documentation, and its call sites are three different things to be similar to.
3. Select against the budget, do not just rank
Ranking gives you an ordered list. Selection is what turns that list into a prompt, and it is a knapsack problem: maximise expected coverage of the question, subject to the token ceiling.
A greedy pass gets you most of the benefit. Walk the fused ranking and admit a candidate only if it adds a facet the current set does not already cover; charge its real token cost against the remaining budget; stop when the next admission would not repay its weight. Cheap and effective refinements on top:
- Deduplicate before charging. Two chunks with high mutual similarity should not both be admitted; keep the better-scoring one and spend the saved tokens elsewhere.
- Reserve a slice for the expensive-but-necessary. A decision record or an interface definition can be large and still be the thing that makes the answer correct. A pure value-per-token greedy will starve it. Reserve ~20% of the budget for at most one such admission.
- Prefer whole units at the margin. Half a function costs tokens and delivers a misunderstanding. If it does not fit whole, drop it and tell the caller.
4. Make the answer describe itself
This is the part that separates a retriever an agent can trust from one it cannot. Every response should carry, alongside the material:
- What it is — a selection under a budget, not the complete set. Say so explicitly.
- What it cost — tokens spent against tokens allowed.
- What was left out — how many candidates cleared the relevance bar but lost to the budget. "Three more files matched but did not fit" is actionable; silence is not.
- How fresh the index is — the age of the index and its coverage of the tree. A retriever answering confidently from an index that stopped updating two days ago is worse than one that admits it.
A human reader glances at the results and senses when something is off. An agent has no such sense. If the answer does not state its own reliability, the agent will treat a thin, stale, truncated result exactly like a complete one.
Measuring whether it worked
Relevance is a feeling until it is a number. Build an evaluation set from real questions — twenty to thirty is enough to be useful, and they must come from your actual users or your actual agent transcripts, not from your imagination. For each question, record which material a competent human considers necessary to answer it.
Then track four things:
| Metric | What it tells you |
|---|---|
| Recall at budget | Of the material a human deemed necessary, what fraction came back within N tokens? This is the headline number. |
| Tokens per answered question | The cost side. Falling tokens with flat recall is the whole point of the exercise. |
| Wasted-context ratio | Tokens delivered that the final answer never used. High values mean the selector is padding. |
| Unfounded-answer rate | How often the model answers confidently from material that did not contain the answer. This is what "I do not know" paths and self-describing responses are supposed to drive down. |
Run the baseline — plain top_k with your current chunker — on the same set, and keep it in the harness forever. Every change to chunking, scoring or selection gets measured against it. Without a baseline in the loop, retrieval work degenerates into vibes and every refactor feels like an improvement.
What this looks like in practice
On a code corpus the pattern that has held up for me: structure-aware chunking with a floor, a dense head plus a lexical head fused by rank, greedy selection against an explicit token ceiling with dedup and one reserved slot, and a response envelope that always states budget, spend, omissions and index age.
The gain is rarely a dramatic jump in recall. It is that recall stays flat while the token spend per question falls by a large factor — and that the failures become visible. A retriever that says "I found two of the four things you probably need, and the index is nine minutes stale" is one an agent can plan around. A retriever that silently returns two chunks and a confident tone is one that produces plausible, wrong work at scale.
Takeaways
top_kanswers the wrong question. Retrieval should be given a budget and asked for sufficiency.- Chunk where meaning changes; apply a size floor; carry parent context into fragments.
- Never run a single scoring head — identifiers and error codes are exactly what embeddings lose.
- Selection is a knapsack, not a slice of a ranking. Dedup, reserve for the large-and-necessary, prefer whole units.
- Every response states what it is, what it cost, what it omitted, and how fresh the index is.
- Build the evaluation set before the optimisation, and keep the baseline in the harness forever.
If you are building retrieval over your own documentation, wiki or codebase and want it evaluated rather than vibed, that is the AI development engagement — and the related read is persistent agent memory, which handles the material that should never have needed retrieving in the first place.