---
title: "LongMemEval: what it tests and how to run it"
description: "LongMemEval evaluates long-term memory in chat assistants across five abilities, including knowledge updates, temporal reasoning, and abstention. Definition and reproducible evaluation steps."
canonical: https://past.dev/benchmarks/longmemeval
last-updated: 2026-10-15
---
# LongMemEval

Source: https://past.dev/benchmarks/longmemeval

LongMemEval evaluates long-term memory in interactive assistants. Published at ICLR 2025, it contains 500 questions over multi-session chat histories. The suite tests information extraction, knowledge updates, temporal reasoning, multi-session synthesis and abstention on unanswerable questions. A comparable result must state the memory procedure, model, prompts and judge.

## What LongMemEval contains

The suite, published at [github.com/xiaowu0162/longmemeval](https://github.com/xiaowu0162/longmemeval), embeds evidence across long multi-session conversation histories and asks 500 questions against them. Question types cover information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention: questions whose correct answer is that the history does not say.

Evidence is distributed across sessions, so some questions require several passages. The update and abstention categories also test whether a system distinguishes current facts from previous facts and unsupported questions.

## Why the update and abstention categories matter most

Knowledge-update questions ask for the current value after a change. A store without [supersession](/glossary/fact-supersession) may retrieve both values without identifying which one is current. Temporal-reasoning questions require event dates and ordering.

Abstention questions have no answer in the supplied history. The evaluator checks whether the answer layer declines to invent an answer from the returned documents.

## Running LongMemEval reproducibly

- State the variant and history length; compressed and full-length versions are not comparable.
- Fix and publish retrieval limits and token budgets; the token budget is the knob that silently buys accuracy.
- Report per-category results so update and abstention behavior remains visible alongside the aggregate.
- Name the judge model and prompts. Judge choice moves scores.
- Report cost and latency next to accuracy for the same run.

## LongMemEval and our results

Our published results cover BEAM at every size and LoCoMo with corrected keys. We do not publish a LongMemEval score. The [benchmarks page](/benchmarks) reports the method, results, per-category scores and cost. [How to benchmark agent memory](/benchmarks/methodology) contains the publication checklist. The [quickstart](/docs/memory-api/quickstart) shows how to test recall on your own data.

## Frequently asked questions

### How is LongMemEval different from LoCoMo?

LoCoMo tests long conversational recall with single-hop, multi-hop, open-domain and temporal questions. LongMemEval adds explicit knowledge-update and abstention categories over multi-session histories.

### What is a good LongMemEval score?

A score is only meaningful with its configuration: variant, retrieval limits, token budget, judge. Compare systems on identical configurations, and weigh the update, temporal and abstention categories over the aggregate if your workload has changing facts.

## Related

- [The BEAM leaderboard](https://past.dev/benchmarks/beam)
- [LoCoMo, explained](https://past.dev/benchmarks/locomo)
- [MemoryBench, explained](https://past.dev/benchmarks/memorybench)
- [How to benchmark agent memory](https://past.dev/benchmarks/methodology)
- [Fact supersession](https://past.dev/glossary/fact-supersession)
- [Quickstart](https://past.dev/docs/memory-api/quickstart)