---
title: "What a good AI memory benchmark measures"
description: "An AI memory benchmark must test changing facts, ordering, and abstention on complete splits. The ten abilities, and what to demand from any score."
canonical: https://past.dev/blog/ai-memory-benchmark-guide
date: 2026-09-01
category: Research
authors: The past.dev team
---
# What a good AI memory benchmark measures

An AI memory benchmark answers one question: given a large history, can the system return what is currently true, with evidence? Most published scores answer an easier question, such as whether the system can find a sentence that was stated once and never changed. This post lists what a memory benchmark has to measure to predict production behavior, and what to check before trusting any score, including ours.

## The ten abilities an AI memory benchmark should test

BEAM scores memory across ten abilities, and the list doubles as a checklist for what your workload actually needs.

1. **Knowledge update.** A fact changed. Does the system answer with the current value? This is the hardest ability at every scale we measure: past.dev scores 75.63% at 100K tokens and 62.14% at 1M, and treating those numbers as solved is how demos differ from production.
2. **Temporal reasoning.** Questions whose answer depends on dates: what was true in March, what happened before the migration. See [temporal reasoning](/glossary/temporal-reasoning).
3. **Event ordering.** Sequence questions. Structural for systems with [event time](/glossary/event-time); past.dev holds 99.3% or above at every scale.
4. **Abstention.** The data cannot answer. Does the system say so? A memory that guesses scores well on easy suites and fails audits. See [abstention](/glossary/abstention).
5. **Contradiction resolution.** Sources disagree. Low seventies for us at every scale, and unsolved across the field. See [contradictory facts](/guides/contradictory-facts).
6. **Multi-session reasoning.** Evidence spread across conversations, which stresses [entity resolution](/guides/entity-resolution).
7. **Information extraction.** Was the fact captured at ingestion at all?
8. **Instruction following.** Does recall respect the question's constraints?
9. **Preference following.** Stable personal facts, the easiest class: near saturation for us at every scale.
10. **Summarization.** Condensing a period without inventing.

An agent that files expense reports needs knowledge updates and ordering. A companion app needs preference following. Averaged scores hide which one you are buying.

## What to demand from any published score

Four checks separate a measurement from a marketing number, and they apply to every vendor including us.

- **Complete splits, visible denominators.** A score on a sampled slice with hidden question counts is unfalsifiable. Our [benchmarks page](/benchmarks) lists questions and conversations behind every figure, per scale and per ability. The [BEAM leaderboard](/benchmarks/beam) ranks every published result by history size.
- **The full scale curve.** One point at 100K predicts nothing about 10M. We publish 100K, 500K, 1M and 10M, and wrote up [how accuracy degrades with scale](/blog/memory-accuracy-at-scale).
- **Corrected answer keys.** LoCoMo's original keys contain known errors; the community-corrected keys change results materially. Any published LoCoMo score should name its key version, because a score on broken keys measures agreement with mistakes. The [LoCoMo page](/benchmarks/locomo) explains the correction.
- **The abstention trap.** A suite where every question is answerable rewards systems that never abstain. Production traffic is full of unanswerable questions, and the status a system returns there is part of its accuracy. See [evidence status](/glossary/evidence-status).

## Running your own evaluation

Public benchmarks rank systems; they do not test your data. The five-test protocol in [how to handle facts that change over time](/guides/facts-that-change-over-time) takes an afternoon: change a fact and ask for both values with dates, backfill with original timestamps, refer to one person three ways, ask the unanswerable, and price the run. The [methodology page](/benchmarks/methodology) documents our judges and configurations so the same protocol can be run against us. [How to reproduce this →](/benchmarks/methodology)