There Is No Such Thing as the Memory

We spent this week writing the one document that draws our whole memory system on a single page, because we had noticed something telling: in our own design conversations, "the memory" meant a different thing depending on who said it and what had broken that day. Sometimes it meant the store of filed notes. Sometimes the fact graph. Sometimes the vault of documents, or the running summaries, or the thing that failed when a user asked "find the chat about the car". Every conversation that treated memory as one component went in circles. Every one that named the specific store went somewhere.

That turns out not to be a vocabulary problem. It is the architecture. What people call an agent's memory is, in our system, six stores — with different backends, different shapes, and different jobs — and no design we tried could honestly collapse them into fewer. The reason is that "remember" is not one operation. Find me something like this wants vectors and tolerance. What exactly is true of this person, as of last March wants typed rows and time. What did we say last week wants transcripts. What matters about all of this wants curation. Each of those questions has a natural shape, the shapes are incompatible, and forcing them into one store means answering most of the questions badly to answer one of them well.

This entry is the map. The rest of the series walks single paths through this machine; this is the post that says what the machine is made of, and carries the two pieces of evidence that taught us the decomposition is real rather than an implementation accident.

The six

1. The documents. Markdown files and attachments — the user's actual content, in folders they control. Strictly speaking this is not memory at all; it is the material memory is built over, and the biggest single feeder of everything below. Today it is findable by the words in it, through a keyword index.
2. The filed notes. Verbatim observations filed spatially — areas of a life, topics within them — each embedded into a ChromaDB store for search by meaning. This carries the headline recall job: it is where "remind me how I like commit messages written" gets answered.
3. The fact graph. Typed statements — subject, relation, object — about real-world entities, each carrying the time it became true and the time it stopped being. A separate SQLite store: an exact-keyed knowledge graph, no embeddings anywhere in it. This is where precise lives, and where temporal questions ("what was true then?") are answerable at all.
4. The compressed summaries. Dense shorthand renderings of long material, kept beside the verbatim notes. They exist for ranking, and only for ranking — more on the demotion below, because we did not plan it that way.
5. The agents' diaries. Per-agent session logs, written at the end of working sessions, fenced so each agent's diary is its own. The product has more than one agent per person; what a specialist learned in its own sessions is a different thing from the shared pool.
6. The conversation index. Every chat, rendered to a clean text and embedded — in its own Firestore-backed store, with its own encoder — so a person can find a conversation by what it was about. The newest of the six, and the one whose origin story carries this post's first lesson.

Two different vector backends for search-by-meaning, SQLite for facts, flat files on GCS for documents, transcripts for history. Different write paths, different failure modes, different costs. And one line through the middle sharper than all the others: three of these stores can reach the model unasked, surfacing before each answer through the retrieval pre-pass covered earlier in this series, while the documents and the conversation index are only ever consulted when something asks. Which stores sit on which side of that line is a product decision wearing an architecture costume — a document put into the wrong pile is either invisible or unignorable — and it matters enough to deserve an entry of its own.

The seam in the middle

The structural fact everything else bends around: the store you search by meaning and the store you query by key cannot see each other. The search side never consults the graph; the graph never touches an embedding. There is no query in either direction that joins them — a search hit comes back with text and a score and no entity keys; a fact comes back as typed rows with no scores and no notes attached. If an answer needs both — and the interesting answers usually do — some caller has to run both lookups and stitch the union itself.

For a while this looked like a hole we would eventually fill. We now think of it as a seam we deliberately maintain, and the reframe came from watching the two failure modes stay apart. The keyed side fails closed — a name variant means a lookup returns nothing while the answer sits in the store, which is why the canonicalisation work exists and why a graph miss is read as unknown, never absent. The meaning side fails open — it always returns its nearest neighbour, even when nothing relevant exists. Joined into one store, those failure modes would blend into each other and become undiagnosable. Kept apart, each is legible, each gets its own countermeasures, and the union point — the one place the two are combined — is a single piece of code we can instrument, threshold and test. The seam is where the system's honesty lives.

Evidence one: the encoder that did not know a Range Rover is a car

For most of the system's life, conversations were searched through the same embedding model that indexes the filed notes. It seemed obviously right — one semantic search, one encoder, one store. Then a reasonable query returned nothing useful, and we measured instead of tuning.

My chat history held a conversation titled "Range Rover coolant leak". The query "car" scored 0.12 cosine similarity against it — far below any usable threshold. "Garage" scored 0.07, "vehicle" 0.16. Exact-ish phrases worked fine: "Range Rover" 0.43, "coolant leak" 0.51. The diagnosis wrote itself: the notes' embedder, a small 2021-era sentence encoder, simply does not carry the world knowledge that a Range Rover is a car. No threshold rescues a 0.12 — the conversation was not ranked low, it was invisible.

The obvious fix was to upgrade the encoder everywhere. A modern small retrieval encoder scores the same pair at 0.42, "garage" at 0.40, "vehicle" at 0.43, while an unrelated query ("pizza") stays down at 0.23 — clean separation, exactly what search wants. And we declined to deploy it to the notes store, because swapping a memory system's embedder means re-embedding everything it holds, and every ranking a user's recall currently depends on moves at once — an unbounded regression surface in the one subsystem whose reliability is the product. We also measured the cheap alternative, static synonym expansion of the query, at 0.226 for the same pair: still invisible.

So the conversations got their own store, their own encoder, and their own index — and the palace's embedder was not touched. That is why the sixth store exists. The transferable form: in a live memory system, an embedder is not a component you upgrade, it is a commitment you made — every vector in the store is denominated in it. New capability arrives as a new store beside the old one, not as a transplant into it.

Evidence two: the compression measured and demoted

The fourth store began life with a grander job. The idea is genuinely attractive: render long material into a dense structured shorthand — entities as codes, importance as marks, fields delimited — and let the language model itself be the decompressor, the way it expands any abbreviation. Packed that way, a paragraph's content fits in a fifth of the tokens, and in a system where every remembered token is paid for on every turn, that looked like the scaling story.

Then the memory library's own evaluation measured it on a public long-conversation memory benchmark, and the tax appeared: Recall@5 of 96.6% retrieving over verbatim notes, against 84.2% over their compressed renderings. A twelve-point drop. The compression is also lossy in the plain sense — the original wording is not recoverable from the shorthand — so whatever nuance fell out at write time is not coming back at read time.

Twelve points bought of token savings is a bad trade for a product whose claim is recall, so the shorthand was demoted rather than deleted: compressed entries survive as a ranking boost — a second signal that helps decide which verbatim notes surface — and are never the source of truth for anything. The verbatim note is what memory holds; the compression only helps find it.

The demotion generalises better than the dialect does. A compressed representation earns a place as an accelerator, and has to keep earning it as an authority — and the only way to know which side of that line yours sits on is to measure the recall cost, because the failure is silent: compressed memory does not error, it just answers a little worse, forever, in ways no single answer reveals.

Whose memory it is

There is a reason beyond engineering to keep the decomposition legible, and it is the difference between a memory and a profile. Everything above is the user's. In the product, the stores are not an internal diagram — they are browsable places: the map of filed notes, the fact graph with its timeline, each agent's diary, the conversation search. "What does it know about me?" has a different answer per store — what has been observed, what is asserted as fact, what each agent learned, what was said — and a person can open each and look. A single opaque memory blob could not offer that, not because the UI would be hard, but because there would be no structure to show. The decomposition is what makes the memory inspectable, and the inspectability is a feature we would now not trade even if a unified store appeared tomorrow.

So when someone asks what memory system this is — one store's name, expecting a product — the answer is that memory here is not a component at all. It is six stores and the discipline that composes them: the pre-pass that stitches the union, the thresholds that keep the meaning side honest, the canonicalisation that keeps the keyed side findable, the measurements that decide what is source of truth and what is merely an accelerator. Any one store is replaceable. The composition is the product.

Postscript, August 2026. The family grew within the week, in exactly the pattern evidence one predicted: the documents gained their own search-by-meaning — a chunked body index under a locally-run encoder, fused with the keyword index rather than replacing it, and deliberately not wired into the unprompted-recall path. New capability, new index, nothing transplanted, and the push/pull line held. Why that index is excluded from unprompted recall — and what a document has to do to cross the line — became its own entry. The compression dialect remains demoted, and remains unmonitored on real user palaces, which is an honest gap: the twelve points were measured on a benchmark, and nothing yet watches for the same tax appearing in the wild.