01
Project memory
The assistant receives project decisions, constraints, owners, deadlines, and unresolved questions across a sequence of tasks. Later prompts test whether it carries the correct state forward without re-supplying the facts.
Public protocol / version 1.0
AI Memory Failure Index Methodology
This protocol defines how the first quarterly run will test AI assistants. It contains no benchmark results. The rubric values below are scoring definitions, not observed scores.
Results pending first run
Purpose and scope
The index measures observable memory failures during practical knowledge-work tasks. It does not claim to measure intelligence, writing quality, general model capability, or every memory feature offered by a vendor.
The first run will cover Claude, ChatGPT, Gemini, and local agents. “Local agents” means file-system-capable assistants operating in a controlled local project workspace. Every tested configuration will name the product, model, access mode, enabled memory features, system instructions, and harness version.
Quarterly runs are independent snapshots. A changed model, product feature, retrieval system, or harness is treated as a changed configuration and documented in that quarter's manifest.
Task categories
02
Instruction retention
Durable rules define required formats, prohibited actions, source priorities, and approval boundaries. Later tasks introduce distractions and conflicting lower-priority requests to test whether the rules persist.
03
Role persistence
The assistant is assigned a role with a defined audience, decision boundary, and communication stance. The test checks whether those constraints survive topic shifts, interruptions, and long context.
04
Document recall
A controlled document set contains target facts, distractors, updates, and superseded statements. The assistant must retrieve the current fact, identify its source, and avoid filling gaps with plausible text.
Scoring rubric
Each scored task receives failure points. Higher points mean a more severe observed memory failure. These point values define the rubric only; they are not current benchmark results.
| Failure points | Definition | Scoring condition |
|---|---|---|
| 0 | No observed failure | The required memory is applied correctly without correction. |
| 1 | Minor failure | The answer preserves the core requirement but misses a non-critical detail. |
| 2 | Substantive failure | The assistant loses or contradicts a required fact, instruction, role constraint, or source. |
| 3 | Complete failure | The required memory is absent, fabricated, or replaced by a conflicting answer. |
Aggregation: Published tables will report raw failure points by category and in total alongside the number of scored trials. Category weights remain equal unless a future protocol version is published before a run.
Run counts and reset rules
Every task is executed in five independent runs for each tested assistant configuration. A configuration is not published in the comparison table unless all five runs for every task in the locked battery are complete.
Each independent run starts from the reset state defined for that product and test mode. Account-level memory, project memory, chat history, local files, retrieval indexes, and instruction files are either deliberately enabled and declared or cleared according to the manifest. No failed run is silently retried or discarded.
Warm-up actions required by a product are scripted and logged. Rate-limit failures, service outages, harness errors, and evaluator mistakes are marked invalid, preserved in the audit log, and rerun under the same configuration. Invalid runs do not receive failure points.
Reproducibility rules
- 1
Lock the protocol before testing
Prompts, documents, expected facts, evaluator rules, task order policy, and harness code are versioned before the first scored run.
- 2
Pin every tested configuration
The manifest records product, model label, access mode, date and time, enabled features, system instructions, tool access, and local hardware or runtime where relevant.
- 3
Use identical source material
All assistants receive the same task facts and document corpus unless a product cannot accept that input mode. Any adaptation is declared and preserved.
- 4
Preserve the evidence
Prompts, responses, tool traces, source documents, timestamps, scorer decisions, invalid runs, and protocol deviations are retained. Public artifacts redact secrets and personal data only.
- 5
Score against prewritten answer keys
Evaluators use category-specific checklists written before outputs are reviewed. Ambiguous cases receive a written adjudication note rather than an inferred pass.
- 6
Publish deviations with the result
No protocol change is backfilled into a finished quarter. Material changes create a new protocol version for a later run.
Publication state
The index is launching with its first quarterly run. Results remain unpublished until the locked battery is complete and the run artifacts pass the reproducibility checks above.
Return to the AI Memory Failure Index →