Project memory
Can the assistant carry forward decisions, constraints, open loops, and project state?
Results pending first run
Quarterly benchmark / launching with first run
AI Memory Failure Index
A public, repeatable test of whether AI assistants remember the project, follow durable instructions, stay in role, and retrieve the right document when the conversation gets long.
What gets tested
Memory that has to survive real work
The first quarterly run will test Claude, ChatGPT, Gemini, and local agents under a versioned protocol. Exact models, access modes, memory features, and local harnesses will be pinned in the run manifest before publication.
Instruction retention
Does it keep following durable rules after interruptions, context growth, and task changes?
Role persistence
Can it preserve an assigned role, decision boundary, and communication stance?
Document recall
Can it locate and use the correct source fact without substituting a plausible answer?
Results
First quarterly run
| Assistant | Tested configuration | Project memory | Instruction retention | Role persistence | Document recall | Failure points |
|---|---|---|---|---|---|---|
|
Results pending first run No scores, percentages, or rankings have been published. |
||||||
Higher failure points will indicate more observed memory failures. No values exist yet.
Scoring and reproducibility rules →Publication standard
Protocol first. Results second.
The methodology is public before the first result. Each quarterly release will identify the tested product, model, access mode, enabled memory features, run manifest, task set version, and any protocol deviation.
Review the public methodology →