What your AI remembers can be poisoned
2026-08-31 · Security
Giving an AI agent memory makes it far more useful. It also gives it something it did not have before: a place where a bad instruction can sit and wait.
This is worth understanding before you roll agent memory out across a team, because the failure is quiet. Nothing crashes. The agent simply believes something it should not, and keeps believing it.
A one-time trick becomes a standing instruction
Prompt injection is well known: hide an instruction in a web page, a PDF, or a support ticket, and an agent that reads it may follow it. Without memory, that is a single bad turn. Annoying, contained.
With memory, it can persist. If the agent writes what it read into its long-term store, the instruction is now recalled on future sessions — possibly for months, possibly by a different agent that trusts the same memory. One poisoned page becomes a standing policy nobody chose.
The dangerous part is not the dramatic version. It is the boring one: a wrong fact, absorbed once, quietly shaping answers long after anyone remembers where it came from.
Encryption does not fix this
We build encrypted memory, so let us be direct about what encryption does and does not do here.
Sealing memory under a key you hold solves custody. It means the operator cannot read your memories, a subpoena to us produces ciphertext, and erasure can be proved by destroying the key. Those are real properties and they are why the protocol is built the way it is.
None of them stop poisoning. If your own agent writes a bad instruction into memory, that instruction gets encrypted, faithfully, under your key — and decrypted just as faithfully next time. Confidentiality is not integrity. Any vendor who answers a poisoning question by talking about encryption has changed the subject.
What actually helps
The controls that work are unglamorous:
- Be deliberate about what gets written. Memory that only records what you explicitly asked it to remember has a far smaller attack surface than memory that silently absorbs everything the agent reads.
- Keep memories separately addressable. If every memory is its own cell, a bad one can be destroyed without rebuilding the rest.
- Be able to actually erase. “Deleted” that means “hidden from the interface” is not a remedy. You want the poisoned entry to become unreadable, permanently.
- Keep sharing narrow. The blast radius of a poisoned memory is exactly the set of agents allowed to read it.
- Review what is in there. Memory you cannot inspect is memory you cannot audit.
We set the whole picture out, including the failures sealing does not address, on AI agent memory security.
A habit worth forming
Treat memory as something you curate rather than something that accumulates. When an agent offers to remember something it picked up from a document, that is a decision, not a formality.
If you find a bad entry, erase it rather than correcting around it:
Recall what you know about our vendor approval process,
then forget the entry that came from the supplier PDF.
Because erasure destroys that cell’s key, the entry does not linger in a backup waiting to be recalled later.
Start free
You can evaluate this properly without a card or a wallet:
claude mcp add saihm -- npx -y @saihm/mcp-server-pro
Then say “Join SAIHM”. The quickstart shows the expected result at each step, and AI agent memory covers the four kinds and who can read each one.
Common questions
- Does encryption stop prompt injection?
- No. Encryption protects memory from people who should not read it. It does nothing about a bad instruction your own agent chose to write down, because that gets encrypted under your key like anything else.
- Can I remove a poisoned memory completely?
- Yes. Each memory is a separately keyed cell, and erasing one destroys its key, which makes that content unreadable permanently rather than hiding it from a view.
- Is agent memory too risky to use, then?
- No more than email or a shared drive is too risky. The point is that memory is an asset with a threat model attached, so it deserves the same deliberateness. Being explicit about what gets written is most of the work.
- How would I even notice a poisoned memory?
- By being able to read what is stored. Memory you can list and inspect can be reviewed; memory buried inside a vendor product cannot. That auditability is the practical control.
Independence notice. SAIHM is an Apache-2.0 protocol authored independently. It is not affiliated with any AI client vendor. Capabilities vary by product and configuration; evaluate any vendor, including SAIHM, against your own requirements. Pricing and tier details are on /pricing.