An assistant whose whole claim is that it knows you has a cost problem folded into the claim. A language model does not carry anything between messages; whatever it is supposed to know right now has to be placed in front of it right now, and it reads all of it again on every turn. Memory is therefore not a thing you pay for once, like disk. It is a thing you pay for on every message, for as long as the product exists.
And the memory has to grow. That is the product. Every conversation, every filed note, every fact learned about a person's work adds to what the assistant could know. So the two lines are heading in opposite directions: the store grows without limit, and the per-turn budget must not. Get that wrong in the obvious way — retrieve more as there is more to retrieve — and the economics run backwards: the longer someone uses the assistant, the more every message costs, and your cheapest user is a stranger. Get it wrong in the other direction, retrieving less to protect the budget, and the memory quietly stops being consulted, which in a memory-first product is the failure the whole thing exists to prevent.
The constraint we build against is stricter than "grows slowly": the cost of remembering has to stay flat as the memory grows. A year of accumulated context and a first week of it should cost the same per message. Everything in this post is the machinery that makes that true, and the honest edges where it is not yet proven.
A place, not a pile
The naive shape for an assistant's memory is a pile — an append-only log, injected wholesale, or trimmed to the most recent slice. A pile degrades as it fills. Every addition makes every future read slightly worse: more to scan, more near-duplicates, more chances for something adjacent-but-wrong to outrank the thing that matters. A notepad is a fine memory at ten pages and a liability at eight hundred.
Ours is organised as a place. Filed notes live spatially — areas for the domains of a person's life, topics within them, individual notes within those — and alongside them sits a separate store of structured facts: typed statements about named people, companies and projects, each carrying the time it became true and the time it stopped being. You do not read a place cover to cover. You navigate it.
Navigation is what makes the cost story work, because it happens in layers, and each layer is materially more expensive than the one before:
- Orientation. What exists — the areas, how full each one is. A directory in the lobby. Near-free.
- The semantic sweep. A search by meaning across the filed notes, returning ranked snippets. Cheap.
- Keyed facts. An exact lookup of the structured facts about a named entity, including as they stood at a past date. Cheap, and precise.
- The graph walk. Following connections outward across areas — used when a relationship is suspected but the path is unknown.
- The full fetch. An entire note, or the complete timeline of an entity. The expensive end, reached only once the earlier layers have named the target.
Most turns never leave the second layer. That single fact is the economics: the store can grow past eight hundred filed notes — ours has — while what a typical turn actually spends is one orientation-level glance and a few ranked snippets.
The turn's budget is fixed by construction
Whether the assistant looks at all is not left to its discretion — we covered that decision, and the measurements that forced it, in an earlier entry. Before the model answers each turn, a small planning pass reads the conversation and formulates what to look for, and the runtime executes the sweep and the keyed lookups itself. What concerns us here is the other half of the design: how much that guaranteed step is allowed to bring back.
The answer is: a fixed amount, regardless of how large the memory is. At most three snippets per search query and six in total, each truncated to a four-hundred-character excerpt; at most a dozen facts per named entity. Those caps are constants. When the memory doubles, recall does not return more — twice as many candidates compete for the same six slots, and the competition gets harder, not the context bigger.
This was a genuine design decision rather than an obvious one, because the alternative has a respectable argument: a richer memory could justify richer retrieval, and a fixed budget will eventually leave good material on the table. We chose the fixed budget anyway, for the reason at the top of the post — a per-turn cost that scales with corpus size, however gently, converts your most engaged users into your most expensive ones. If the caps prove too tight, the evidence will show up as misses in the evaluation set, and raising a constant is a one-line change. The reverse mistake, an unbounded budget discovered in a bill, is not.
Paying for a subject once
The caps bound a single turn. The next leak is across turns, and we shipped it before we saw it.
A conversation about one subject runs the same lookups again and again. Ask about a person, and the sweep and the keyed lookup both run and both return. Ask a follow-up, and — in the first version of this system — everything ran again: same queries, same round trips, same results, injected fresh and then discarded. The retrieval was ephemeral, handed to the model for one turn only. The visible symptom was worse than waste: the assistant could ask "did you mean X?" about a fact it had retrieved two turns earlier, because from where it stood, it had never retrieved anything.
We considered three fixes. A result cache keyed by query — cheapest to build, and it repairs the latency without repairing the amnesia, since the model still never sees yesterday's results. A stronger instruction telling the model not to re-search — unenforceable, because the model is not the one issuing these lookups, the planning pass is. A separate tracking field listing what had been queried — workable, and redundant the moment we noticed the third option contained it.
The third option: make retrieval part of the conversation record itself. Each turn's recall now lands in history as a lookup that visibly happened — the query, and what came back — exactly as if the assistant had chosen to search. Two things fall out. The model retains what was found, across turns, because it is simply in the conversation. And the planning pass can be shown what has already been fetched and emit only the delta — a subject retrieved this session is not retrieved again, and when the plan comes back empty, no lookup fires at all.
Within a rolling window, a subject is paid for once — and the cache is the transcript. Nothing was added to hold the remembered results: the conversation, which the model reads anyway, is the store, which is why the model and the deduplication logic can never disagree about what has already been looked up.
Permissive on purpose
One piece of the machinery decides more about felt quality than any other: the floor under the semantic sweep. A search by meaning always returns something — its nearest neighbour, however far away that is — so every hit is checked against a minimum similarity before it reaches the model, and where that floor sits is a choice about which failure you prefer.
We probed the store before choosing. When the answer to a query is genuinely present, it scores well clear — similarity of 0.5 and up. When the answer is not stored, the nearest wrong thing tends to land around 0.3: close enough to look plausible, which is precisely the material you do not want injected under an implicit "this is relevant" label. The floor sits between those bands, and it sits low in the band on purpose. In a product whose entire claim is recall, a miss costs more than mild noise: irrelevant context is a small tax on one answer, while a miss is the product visibly failing at the one thing it promised.
That number is a first setting, not a calibration. We swept the threshold against a labelled evaluation set and the sweep confirmed the knee where we had guessed it — but the set is fifteen cases against a development corpus, and the current scores are 12 of 14 on correctly deciding whether to retrieve and 4 of 6 on surfacing the right note when one exists. Those are sanity-check numbers, not tuning-grade numbers. What we hold onto is the discipline rather than the value: the telemetry that makes the distributions visible shipped before any knob was turned, every threshold is adjustable without a redeploy, and the evaluation set — not taste — is what will move the floor. Measure before you turn a knob, and be honest about how small the measurement still is.
What flat buys
Put the three mechanisms together — a fixed injection budget, layers that keep most turns at the cheap end, and a transcript that stops the same subject being paid for twice — and the property at the top of the post holds: per-message cost does not track the size of what is remembered. The store grows; the bill does not. An assistant that has known you for a year answers "morning — anything I should know today?" for the same price as one that met you on Tuesday.
That is what makes a real memory an ordinary feature rather than a luxury one. The alternative designs all work in a demo, because a demo's memory is small; the differences only compound where a product lives, in month six, in the eight-hundredth note. The transferable version, for anyone building retrieval into a long-lived system, is the constraint rather than our mechanisms: decide the per-turn budget first, make it a constant, and force every growth curve in the system to negotiate with that constant — because any cost line you let scale with corpus size is a tax on exactly the users who trust you most.
Postscript, August 2026. The shape held; two edges moved, both in instructive directions. A real ninety-minute session later hit the model's context ceiling, and what had filled the window was largely recall and tool machinery riding along in history — the cost of making retrieval persistent, arriving late. The fix prunes recall pairs older than the dedup window from what the model is shown (stored history untouched), so context retention and re-query suppression now share one window, pinned by a test that fails if either side's counting drifts. And the planner's "do not re-query" instruction turned out to leak — a subject was re-queried inside the window, byte-identical results — so suppression is now enforced in code after the plan comes back, with the prompt rule kept only as a token-saving first pass. A rule you can enforce structurally should not stay a request. The threshold floor is unchanged, still permissive, still waiting on a bigger evaluation set to earn a real calibration.