---
title: "How to benchmark agent memory"
description: "A reproducible method to benchmark agent memory systems: test cases, baselines, judges, cost reporting, and the publication checklist that makes results comparable."
canonical: https://past.dev/benchmarks/methodology
last-updated: 2026-08-31
---
# How to benchmark agent memory

Source: https://past.dev/benchmarks/methodology

Benchmark an agent memory system with tests for updates, identity resolution, historical questions and unsupported questions. Include full-context and plain-RAG baselines. Score retrieval and answers separately with more than one judge. Publish accuracy, cost, latency and the complete configuration so another team can rerun the result.

## Test the four operations that define memory

- **Updates.** State a fact, revise it later, ask for the current value. Tests [supersession](/glossary/fact-supersession) rather than retrieval.
- **Identity.** The same person as an email signature, a transcript name and a chat handle. Tests entity resolution across sources.
- **Historical questions.** The value as of a date. Tests [validity windows](/glossary/validity-window) and [point-in-time recall](/glossary/point-in-time-recall).
- **Unsupported questions.** Questions the history cannot answer. past.dev returns this result as `NoKnownSupport`.

Public suites cover parts of this: [LongMemEval](/benchmarks/longmemeval) tests updates and abstention, [LoCoMo](/benchmarks/locomo) tests long conversational recall, [MemoryBench](/benchmarks/memorybench) tests continual learning. Run several; none alone is sufficient.

## Required baselines

Use two baselines. The full-context baseline sends the complete history for every question and measures accuracy and cost without retrieval. The plain-RAG baseline retrieves passages from the same corpus. Publishing both shows whether the memory system improves on simpler methods.

## Separate retrieval and answer scores

Retrieval sufficiency measures whether the returned evidence contains the required information and ranks current facts before superseded facts. Answer accuracy also measures the downstream model. Report the metrics separately. Use more than one judge, publish judge prompts and report results for each judge because judge choice affects scores.

## The publication checklist

1. Dataset, split and answer-key version, with corrections named.
2. System version and full configuration: retrieval limits, token budgets, models.
3. History size and question counts.
4. Answer model and every judge, with prompts.
5. Accuracy, cost and latency from the same run, reported both by category and in aggregate.
6. Failed, timed-out and degraded requests, counted rather than excluded.
7. The harness, runnable by someone who is not you.

Our runs use this checklist on the [benchmarks page](/benchmarks). To evaluate your own data, use the [Memory API](/docs/memory-api/quickstart) with changed facts and unanswerable questions. Apply the same checklist to every tested system, including past.dev.

## Frequently asked questions

### Why not trust vendor-published scores?

A result with an unpublished configuration cannot be verified. Check whether the harness, data split, retrieval limits, token budget and judge configuration are available and reproducible.

### How large should the test history be?

Test at least the history size expected in production. Run the same evaluation at several sizes to measure how accuracy, latency, and cost change as history grows.

## Related

- [LongMemEval, explained](https://past.dev/benchmarks/longmemeval)
- [LoCoMo, explained](https://past.dev/benchmarks/locomo)
- [MemoryBench, explained](https://past.dev/benchmarks/memorybench)
- [What is agent memory?](https://past.dev/what-is-agent-memory)
- [Quickstart](https://past.dev/docs/memory-api/quickstart)