Fewer People, More Agents: Isolate Every Agent’s Memory Before You Cut the Team

Shared memory without enforced ownership turns retrieved material into authority. Isolate mutable state, constrain credentials, and preserve a staffed exception path before removing human review.

The Research Result That Started This

In the AgentPoison authors’ experiments, poisoned memory redirected agents toward attacker-chosen actions while performance on benign queries remained largely intact.[1] Routine accuracy could look healthy while triggered tasks went wrong. These were research evaluations, not a disclosed production breach; I have not reproduced them. It is still the clearest published test of memory poisoning against agents I know of.

Their attack assumed write access to part of the memory or knowledge base and white-box access to an embedder for trigger optimization. A triggered query retrieved malicious demonstrations, which then influenced action selection. The paper’s EHRAgent target included generating a database-deletion command—not evidence that a hospital lost records.

The Anti-Pattern Mechanism

I call the anti-pattern shared memory, shared authority. An untrusted entry enters a shared store, retrieval promotes it into planning context, and a broadly privileged credential turns that context into an action. Another agent can retrieve the same contamination. A successful API call measures execution, not authorization intent or truth.

OWASP’s agentic threat guide frames the memory threat; its LLM06:2025 guidance specifically recommends minimum permissions, downstream authorization and approval for high-impact actions.[2][3]

Reducing reviewers does not cause poisoning. My architectural concern is that removing review capacity without replacing its controls leaves fewer opportunities to interrupt this path. That staffing effect is a risk argument, not a measured result from AgentPoison.

The Architecture I Recommend

I would separate workload identity, mutable memory and authorization. Issue short-lived credentials per agent; bind delegated access to the initiating user and tenant. Enforce permissions in the tool gateway and data layer, not the prompt.

Use PostgreSQL row-level policies or separate stores keyed by tenant and agent. Derive that scope from authenticated identity, never model-supplied filters. Keep shared reference material read-only; promote proposed updates through review rather than letting agents write directly into collective memory.

Illustrative interface contract, not executable configuration:

identity: authenticated_agent_principal
memory_scope: [tenant_id, agent_id]
memory_required: [source_uri, author_id, recorded_at, content_hash]
write_grant: explicit
log_required: [input_ref, tool_call, output_ref, model_tag, memory_ids,
               policy_version, checkpoint_id, trace_id]
exceptions: {owner: named_oncall, age_limit: workload_SLO}
stop: [deny_new_calls, revoke_credentials, quarantine_writes]
restore: verified_memory_checkpoint

Store immutable, access-controlled input/output snapshots behind those references, with retention and redaction rules. Record exact model, retriever and policy versions. Replay reconstructs decisions from captured state; it does not promise identical stochastic generation. Provenance makes an entry traceable, not trustworthy.

Production Readiness Checklist

These are proposed acceptance tests, not measured fleet results. A failed check blocks removing the reviewer.

• Own identity: revoke the credential; only its agent must stop.
• Least privilege: an unlisted tool must fail; writes and deletes require separate grants.
• Scoped memory: plant a marker in agent A’s tenant-bound store; agent B must not retrieve it, including through caches.
• Provenance: in a proposed spot-check of five entries, trace source, author and timestamp; missing lineage fails.
• Replayable log: reconstruct a past task’s inputs, calls, outputs, model version and memory reads from retained evidence.
• Exception owner: age an unreviewed item past the workload’s declared limit; verify the named person receives the alert and has capacity to respond. After a cut, that owner is often the reviewer who stayed, now covering more agents, so measure their capacity rather than assume it.
• Kill switch and rollback: in staging, stop execution and restore a verified checkpoint; record elapsed time and reconcile already-issued actions.

What I Would Build Differently

I would retain independent approval for destructive operations even after these checks pass. Stopping an agent must also address queued calls and credentials already issued. Memory rollback needs a separate compensation plan for external effects.

Rex Circuit’s planned contention test in this series can exercise fixture isolation and log reconstruction. It cannot establish production memory isolation or human cleanup cost. This checklist has no production-fleet benchmark or independent grading; I would require adversarial testing against the deployed versions before changing staffing.

What we offer

The Agent Hire Kit brings this design checklist together with the job spec, business-case calculator, worker worksheet and planned contention harness.

Takeaway: remove shared authority before removing the people who catch its failures.

References

1. Chen et al., AgentPoison, arXiv:2407.12784, submitted July 17, 2024. https://arxiv.org/abs/2407.12784
2. OWASP, Agentic AI – Threats and Mitigations, February 17, 2025. https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/
3. OWASP, LLM06:2025 Excessive Agency, 2025 edition. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/