---
title: "MemoryBench: the memory benchmark, explained"
description: "What MemoryBench measures, how it relates to LoCoMo, LongMemEval and other agent memory benchmarks, and how to compare published results."
canonical: https://past.dev/benchmarks/memorybench
last-updated: 2026-08-31
---
# MemoryBench

Source: https://past.dev/benchmarks/memorybench

MemoryBench evaluates memory and continual learning in LLM systems. Researchers at Tsinghua's THUIR group published it with code on GitHub. Related benchmarks include LoCoMo, LongMemEval, MemBench and MemoryAgentBench. Each tests different abilities, so results are comparable only when the benchmark, split and configuration are stated.

## What MemoryBench is

MemoryBench, published at [github.com/THUIR/MemoryBench](https://github.com/THUIR/MemoryBench), frames memory as a continual-learning problem: a system receives information over time and is evaluated on how well later behavior uses earlier information. The repository hosts the benchmark interface and baseline implementations, with reproduction code linked from it.

Many retrieval evaluations use a static corpus. Continual-learning benchmarks add information over time and measure later use of earlier information. They can test updates, incremental ingestion and larger histories.

## The memory benchmark family

| Benchmark | Focus | Source |
| --- | --- | --- |
| MemoryBench | Memory and continual learning in LLM systems | [THUIR/MemoryBench](https://github.com/THUIR/MemoryBench) |
| LoCoMo | Very long conversations: single-hop, multi-hop, open-domain and temporal questions | [snap-research/locomo](https://github.com/snap-research/locomo) |
| LongMemEval | Long-term interactive memory: updates, temporal reasoning, abstention | [xiaowu0162/longmemeval](https://github.com/xiaowu0162/longmemeval) |
| MemoryAgentBench | Incremental multi-turn interactions for memory agents | [HUST-AI-HYZ/MemoryAgentBench](https://github.com/HUST-AI-HYZ/MemoryAgentBench) |
| BEAM | Recall over growing event histories, including updates, identity and abstention | [our benchmarks page](/benchmarks) |

No single benchmark covers every memory function. A conversational-recall score does not measure backdated ingestion, entity resolution across sources or point-in-time queries unless the suite includes those tests. [What is agent memory?](/what-is-agent-memory) describes the main functions.

## How to compare memory benchmark scores

- **Which split, which keys.** Some suites have documented answer-key errors and community corrections; a score is only comparable when the key version is stated.
- **Who ran it.** Vendor-run scores of the vendor's own system deserve the same scrutiny as any self-reported metric. Ask for the harness, the prompts and the judge configuration.
- **Cost and latency.** Report token cost and latency from the same run as accuracy.
- **Retrieval or answers.** Retrieval sufficiency measures the memory; answer accuracy also measures the downstream model. Mixing the two hides which component failed.

## How past.dev benchmarks memory

The [benchmarks page](/benchmarks) reports current BEAM results and the planned evaluation scope. [How to benchmark agent memory](/benchmarks/methodology) contains the method and publication checklist. Use the [quickstart](/docs/memory-api/quickstart) to test the API on your own data.

## Frequently asked questions

### Is MemoryBench the standard memory benchmark?

No single standard exists. MemoryBench tests continual learning; LoCoMo tests long conversations; LongMemEval tests updates, temporal reasoning and abstention. A complete evaluation uses several suites and publishes each configuration.

### Why do vendors report such different numbers on the same benchmark?

Retrieval limits, token budgets, judge models and answer-key versions affect scores. Compare runs only when these settings match. A published harness makes the configuration verifiable.

## Related

- [LongMemEval, explained](https://past.dev/benchmarks/longmemeval)
- [LoCoMo, explained](https://past.dev/benchmarks/locomo)
- [How to benchmark agent memory](https://past.dev/benchmarks/methodology)
- [What is agent memory?](https://past.dev/what-is-agent-memory)
- [Quickstart](https://past.dev/docs/memory-api/quickstart)