There is a standard recipe for letting a language model use a body of private knowledge: embed the corpus into vectors, take the user's message, find its nearest neighbours, and staple the best few to the prompt before the model answers. Retrieval-augmented generation — RAG — in its basic form is exactly this shape, and for a question-answering tool over a document pile, it is a reasonable shape.
We build a memory-first assistant, and each of the recipe's three steps embeds an assumption that fails for one. Take the user's message assumes the message is a good query. Find nearest neighbours in the corpus assumes there is one corpus, and that similarity to the message is what makes something relevant. Staple to the prompt assumes retrieval should be invisible, stateless, and free of consequence beyond this one answer. This entry follows a single question from the moment it is typed to the moment the model answers — because the pipeline is where those three assumptions break visibly, one after another, and where what replaced each of them earns its keep.
Earlier entries in this series established the parts: why retrieval is invoked deterministically rather than left to the model's discretion, how its per-turn cost is held flat, what the stores are and why the two main ones cannot see each other, and what a document must opt into before memory will volunteer it. This is the post about how the parts compose. Take those as read; the walk is the subject.
Suppose a thread has spent a few turns on a contractor's quote — call her Alice — for a studio build, wandered off to something else, and the user now types: "did she come back on the price?"
The message is a bad query
Embed that message raw and search with it, and you are asking the store for things similar to the words did she come back on the price — a string containing no name, no subject, and a verb phrase that could belong to a hundred topics. The person who typed it is not being careless. They are speaking the way people speak inside a conversation, where half of every sentence's meaning lives in the turns before it. The message was never the query. It is a pointer to a query, and something has to dereference it.
So before retrieval runs, a small, cheap model does one job: it reads the recent thread plus a persisted running summary of what the conversation is about — under twenty-five words, carried forward and refreshed every turn — and emits a plan. Search strings for the fuzzy store. Entity names for the fact graph. The refreshed summary. For our turn, that plan reads something like: search "Alice studio build quote", look up the entity Alice — the pronoun resolved, the elided subject restored, the topic recovered from three turns back. When the turn needs nothing — a greeting, a thanks, a general-knowledge question with no personal hook — the plan is legitimately empty and no lookup runs at all. And subjects already retrieved earlier in the session are suppressed from the plan, so a long conversation about one person does not pay for the same lookup on every turn.
The transferable idea costs one sentence: a small model planning the retrieval is cheaper than a large model recovering from a bad one. Query formulation is where retrieval accuracy is actually bought — before a single vector is compared — and it is bought at the cheap end of the model range, because coreference resolution — she, against three turns of context — is exactly the kind of bounded judgment small models do well.
Two stores that cannot join, and why that is the design
The plan now fans out — and not against one corpus. The search strings run as a fuzzy sweep over the filed notes: hybrid retrieval, vectors for meaning plus lexical scoring for exact tokens, returning ranked snippets. The entity names run as exact keyed lookups into the fact graph: typed rows about Alice, with validity dates, or nothing.
Neither store can consult the other. A search hit carries text and a score and no entity keys; a graph row carries typed facts and no notes. There is no query in either direction that pivots between them, so the runtime performs the union itself — two parallel lookups, stitched by the caller.
The split is deliberate: the two failure profiles are the reason both stores exist. The keyed side has near-zero false positives and is brittle to name variants: query a spelling the store never minted and you get nothing, which is why a graph miss is read as unknown, never absent. The fuzzy side is forgiving of naming and never returns nothing: it hands you its nearest neighbour with confidence precisely when no true answer exists. Each is strong exactly where the other fails, and measured on our own store the complement is real — the entity lookup that returned zero on a name variant sat next to a fuzzy sweep that surfaced the right documents at healthy scores, and vice versa. A single blended store would average the two failure modes into something uniformly mediocre and undiagnosable. Choosing to run two stores with named failure profiles, and to pay the stitching cost at one instrumented union point, is a choice — the recipe's "one corpus, one similarity" step, unmade.
What comes back enters the record, not the prompt
The recipe's third step is to paste the retrieved text invisibly ahead of the user's message, answer, and throw it away. Ours does something that looks slower and pays for itself three times over: the results are appended to the conversation itself, as a lookup that visibly happened — the queries, and what each returned, recorded the same way as any action the assistant takes.
First consequence: recall persists. The next turn's model sees what was retrieved this turn, because it is simply in the conversation — no re-deriving, no asking the user "did you mean the Alice from the studio project?" about a fact it held two turns ago. Second: the retrieval is inspectable by the person it is about. The interface shows the lookups as they happen — searching for this, checking the graph for that — which for a memory product is not decoration; a system claiming to know you owes you a view of when it is consulting what it knows. Third: the formulation pass reads the same record next turn and plans against it — what has been fetched is suppressed, and only the delta is queried.
One structure, three properties: making retrieval an event in the transcript rather than a hidden prefix is what makes it auditable, cacheable and de-duplicable at once. The hidden-prefix version can be given any one of those with bolted-on machinery. The transcript version gets all three from the decision itself.
Push, then pull — and the index deliberately left out
Everything so far is the push half: what arrives before the model answers, covering what the thread implies. It is not the whole of retrieval. Beyond it, the model fetches for itself, down the layered protocol described earlier in the series — orientation, sweep, keyed facts, graph walk, full fetch, each layer materially dearer than the last — and most turns never leave the cheap end.
But one index is missing from the push half on purpose, and the omission is load-bearing. The documents' own body index — the meaning-level search over full file contents — is never consulted by the pre-pass. Only memory is. The body index is far larger than memory, and a nearest-neighbours pass against it on every turn would push whatever happened to be closest into the window, every time, relevance decided by proximity alone — the recipe's failure mode, rebuilt inside a system designed to avoid it. Keeping the stores separate makes that flooding impossible by construction rather than by tuning: no threshold drift, no ranking regression, no bad day can cause the document pile to volunteer itself, because no code path exists from the pre-pass to that index. A document reaches unprompted recall only by being promoted into memory — a user's explicit, per-folder choice, argued in the two entries before this one. The same boundary, seen from the retrieval side.
The cost of the boundary is real: a fact mentioned in passing in a document nobody promoted is reachable the moment somebody asks, and only then. We accept that, for the reason the previous entries argue — the alternative is a memory diluted by everything it was handed.
One budget pays for all of it
None of the above floats free. Retrieval's results land in the same context window that already carries the shared system prompt, the schemas of the tools the assistant can call, and the conversation itself — and that window is a budget, paid on every turn. The design only stands because the rest of the budget is managed with the same discipline: several hundred tool definitions sit behind a search and load on demand rather than riding resident; the expensive prefix is kept stable so it caches; the far end of a long conversation is compacted rather than carried verbatim. Those are their own entries. The point that belongs here is the dependency: a fixed per-turn retrieval spend is only affordable inside a prompt whose other tenants are also fixed. Beats one through four are defensible because of beat five, not alongside it.
The fork: we wanted the elegant version
Now the part a fair number of readers will disagree with, and the reasoning deserves their strongest counter-argument first.
Agentic retrieval — give the model the search tools, let it decide when to look — is genuinely more elegant. It costs nothing on turns that need nothing. It is maximally selective: the model knows what it already knows, and looks things up the way a person does, when uncertainty bites. It has no pipeline to maintain, no thresholds to calibrate, no per-turn tax. We ran it that way first, wanted it to work, and instructed it carefully.
Measured across 277 of our own sessions — about four thousand tool calls — it did not happen. Memory writes outnumbered reads six to one; genuine content recall appeared in roughly one session in nine; and the behaviour persisted after the recall instructions were strengthened, with the model demonstrably recognising the entities it was declining to look up. And when the model did choose to retrieve, its judgment failed in the other direction too: one measured thread ran about twenty-six memory reads across eight turns, most of them redundant re-searches of facts already in context or probes into memory for changing-world facts that no personal store could hold. Discretion was wrong in both directions at once — too little when recall mattered, too much when it did not.
So the fork was taken on the numbers, not the aesthetics: deterministic invocation of a judgment step. The loop guarantees the retrieval step runs — invocation cannot be skipped, because invocation is precisely what the measurements showed failing — while the small model decides what the step is worth, including nothing. Reliability comes from code; selectivity comes from judgment; neither is asked to do the other's job. The tax is honest: a model call on every turn, paid whether or not the turn needed it. We pay it because in a product whose entire claim is recall, a silent miss is the expensive failure, and we could not buy its absence any other way.
If you have built the agentic version and it works for your product, the difference is probably in that last sentence. A coding tool that misses a lookup produces a wrong answer someone was already checking. A memory assistant that misses a lookup breaks the one promise it exists to keep, invisibly, with nobody positioned to notice but the user it forgot.
Postscript, September 2026. The seam between the two stores was on the list to close — search hits carrying entity keys, so a semantic find could be enriched by an exact graph lookup — and we took it off the list after reading a month of production recall. The semantic side returns something above the floor on nine turns in ten. The graph side answers just over half of the entity lookups it is asked, and on nearly all of its misses a semantic hit was present anyway; the graph was the only store to answer on five turns in the month. So the seam is not where turns are lost. What a graph miss costs is a typed fact with a date on it, on a turn that still got the prose: precision, not coverage. The one measurement that would justify a bridge is how many of those misses are a key spelt two ways rather than a memory never stored, and the pipeline does not log the names it looked up, only the counts. The names go into the log first; the bridge waits on what they show. The same month says where the real cost sits. Recall now takes a median of 2.8 seconds in front of a substantive turn, well above what this entry measured, and that is the number being worked on. The 277-session scan does not need re-running: with retrieval deterministic, "did the agent read" saturates by construction, and the per-turn line is its successor. Six substantive turns in a hundred come back with nothing from either store, which is the figure to watch. The similarity floor is still the permissive first setting. The hits it admits sit at a median similarity of 0.61, with a tenth of them under 0.47, so it sits beneath the population rather than cutting into it; a labelled evaluation is still what would earn a real calibration. The pipeline itself now also fronts scheduled runs and the stages of long autonomous jobs, which is where a guaranteed retrieval step matters most — nobody is present to notice a miss at all.