---
title: "LongMemEval: what it tests and how to run it"
description: "LongMemEval evaluates long-term memory in chat assistants across five abilities, including knowledge updates, temporal reasoning, and abstention. Definition and reproducible evaluation steps."
canonical: https://past.dev/benchmarks/longmemeval
last-updated: 2026-08-31
---
# LongMemEval

Source: https://past.dev/benchmarks/longmemeval

LongMemEval evaluates long-term memory in interactive assistants. Published at ICLR 2025, it contains 500 questions over multi-session chat histories. The suite tests information extraction, knowledge updates, temporal reasoning, multi-session synthesis and abstention on unanswerable questions. A comparable result must state the memory procedure, model, prompts and judge.

## What LongMemEval contains

The suite, published at [github.com/xiaowu0162/longmemeval](https://github.com/xiaowu0162/longmemeval), embeds evidence across long multi-session conversation histories and asks 500 questions against them. Question types cover information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention: questions whose correct answer is that the history does not say.

Evidence is distributed across sessions, so some questions require several passages. The update and abstention categories also test whether a system distinguishes current facts from previous facts and unsupported questions.

## Why the update and abstention categories matter most

Knowledge-update questions ask for the current value after a change. A store without [supersession](/glossary/fact-supersession) may retrieve both values without identifying which one is current. Temporal-reasoning questions require event dates and ordering.

Abstention questions have no answer in the supplied history. past.dev represents this result as `NoKnownSupport`.

## Running LongMemEval reproducibly

- State the variant and history length; compressed and full-length versions are not comparable.
- Fix and publish retrieval limits and token budgets; the token budget is the knob that silently buys accuracy.
- Report per-category results so update and abstention behavior remains visible alongside the aggregate.
- Name the judge model and prompts. Judge choice moves scores.
- Report cost and latency next to accuracy for the same run.

## LongMemEval in our evaluation

Our harness includes LongMemEval, LoCoMo with corrected keys and BEAM at larger scales. The [benchmarks page](/benchmarks) reports the method, results, per-category scores and cost. [How to benchmark agent memory](/benchmarks/methodology) contains the publication checklist. The [quickstart](/docs/memory-api/quickstart) shows how to test recall on your own data.

## Frequently asked questions

### How is LongMemEval different from LoCoMo?

LoCoMo tests long conversational recall with single-hop, multi-hop, open-domain and temporal questions. LongMemEval adds explicit knowledge-update and abstention categories over multi-session histories.

### What is a good LongMemEval score?

A score is only meaningful with its configuration: variant, retrieval limits, token budget, judge. Compare systems on identical configurations, and weigh the update, temporal and abstention categories over the aggregate if your workload has changing facts.

## Related

- [LoCoMo, explained](https://past.dev/benchmarks/locomo)
- [MemoryBench, explained](https://past.dev/benchmarks/memorybench)
- [How to benchmark agent memory](https://past.dev/benchmarks/methodology)
- [Fact supersession](https://past.dev/glossary/fact-supersession)
- [Quickstart](https://past.dev/docs/memory-api/quickstart)