In 1689 John Locke asked who you would be if you woke up tomorrow with a stranger’s memories. His answer unsettled people then and it still does. You would be the stranger. Identity, he argued, is not the body you carry and not some fixed soul underneath. It is the continuous thread of what you remember.
We have started building systems that remember. Not inside a single conversation, across all of them. They keep a model of you that grows week by week, refer back to it before they answer and act on it without being asked. The industry calls this memory. It borrowed the word and skipped the philosophy that comes with it. If identity is continuity of memory, then a system holding a persistent and growing record of a person is not storing data about that person. It is assembling something that behaves like an identity. Its own, and a version of yours.
None of this is hypothetical. It is being built right now, mostly by teams who have not noticed they are answering a question about identity at all.
“The AI knows me” is a claim, not a feature
When a system says it remembers you, it is making an assertion about who you are. You prefer short answers. You work in finance. You asked about redundancy terms in March, so you are probably worried about your job. Each of these is a small claim, and together they form a portrait the system treats as true and acts on.
You did not write that portrait. You cannot see it. You did not agree to most of it. It was inferred from fragments, assembled in the background and promoted to fact the moment it became useful. The account manager who “knew” their customers used to hold a version of this in their head. That informal human knowledge was real work, and when it left with the person, everyone understood something had been lost. The persistent user model is that same portrait, made explicit, made permanent and handed to a system that will defend it.
A feature is something you can toggle. A claim about who someone is has to be governed.
The false memory does not stay quiet
The previous piece in this series argued that memory is reconstructive, in people and in machines. We rebuild the past and get it wrong, and so do vector stores and retrieval systems. Locke saw the harder version of this three centuries earlier. The thread that makes you yourself can carry false memories and still hold. A man certain he was present at an event he never attended is not a different man. He is himself, plus an error he cannot feel.
A persistent user model has the same property. Once a wrong inference enters it, the system does not hold the error tentatively. It holds it the way it holds everything else, as part of who you are to it. It greets you with the error, reasons from the error and grows new conclusions on top of it. The model that decided you were worried about your job keeps treating you as a flight risk long after the reorganisation passed. Nothing flagged it as wrong because, structurally, nothing about it was wrong. The system was being faithfully itself.
A model of a person that cannot tell which of its beliefs are time-bound lies with full confidence and calls the lie continuity. The temporal layer exists for exactly this. It marks when a fact was true so the system can let it expire, instead of carrying it forever as part of who you are.
Continuity is the feature and the liability
The reason persistent memory is worth building is continuity. The system that remembers last month’s context is more useful than the one that meets you fresh every morning. That exact continuity is what makes its failures matter.
Fork the deployment and there are now two versions of you, drifting apart with every session. Migrate the database and lose six months of accumulated context, and the person who logs in afterwards is, to the system, a stranger wearing your name. Give someone with write access a reason to edit the model and they can change who you are to the system without changing anything about you. Leak it and you have not leaked a log. You have leaked a dossier that claims to be a person.
Each of these is an ordinary operational event. Each is also a question Locke would recognise. The continuity that made the system valuable is the same continuity whose loss, corruption or theft is a real harm to a real person.
Whose memory is it
The model is about you. It is owned, stored and controlled by someone else. That gap is the responsibility problem this body of work keeps returning to, now pointed at identity itself.
Ask the questions that ownership forces and most deployments cannot answer them. Can the person read the model the system holds of them. Can they correct it when it is wrong. Can they delete it. Who else can write to it, and is that boundary structural or just a polite instruction. When the model and the person disagree about who the person is, which one wins.
These are not privacy questions bolted on at the end. They are architecture. A persistent identity model with no read path, no correction path and no ownership boundary is not a richer user experience. It is an unaccountable claim about a human being, running in production, getting more confident the longer it runs.
Closing
Three centuries on, Locke’s question is a database schema. The thread that makes a person continuous now lives in storage owned by whoever runs the model, describing someone who has never read it.
Three questions remain, and none of them are about storage. Is what the system remembers about a person true. Who answers for it when it is not. And does the person ever get to read their own file.
Sources
- An Essay Concerning Human Understanding — John Locke, 1689. Book II, Chapter 27 (“Of Identity and Diversity”) sets out the memory-based theory of personal identity and the prince-and-the-cobbler thought experiment that anchors this piece.
- The Illusion of Perfect Recall: Your AI Doesn’t Remember, It Reconstructs — Memory 3. Establishes that AI memory is reconstructive, not reproductive. This piece takes that finding to its personal-identity consequence.
- The Memory Illusion — Julia Shaw. The science behind Memory 3: human memory rebuilds the past rather than replaying it.
- The Forgotten Layer: Memory Is the AI Variable Most Teams Are Missing — the temporal-facts layer this piece leans on.
- When Humans Leave the Loop — the persistent user model as the designed replacement for the informal human knowledge of a customer (compensating middleware).
- Why AI Systems Need Guardrails That Are Structural, Not Prompt-Based — who can write to the user model is a structural boundary, not a prompt instruction.
Leave a Reply