---
title: "Measuring memory honestly"
description: "How past.dev benchmarks memory: corrected answer keys, strong baselines, multiple judges, and cost reported next to accuracy."
canonical: https://past.dev/blog/measuring-memory-honestly
date: 2026-08-25
category: Research
authors: The past.dev team
---
# Measuring memory honestly

Memory benchmarks are easy to win and hard to trust. Most published numbers compare a memory system against a weak retriever, on answer keys with known errors, with a single judge, and without saying what the run cost. We publish ours differently, and this post says exactly how.

## Corrected keys, or the benchmark grades itself

LoCoMo, the most cited long-conversation benchmark, ships with answer keys the community has documented as partially wrong. Scoring against a wrong key punishes a system for being right. We run LoCoMo on the community-corrected keys and say so, because a number is only comparable when the key behind it is stated.

## Baselines that are allowed to win

Every run includes two baselines: full context, where the whole history is resent to the model on every question, and plain RAG over the same corpus. If a memory system cannot beat resending everything, it has no reason to exist, so the baseline must be allowed to win. On the BEAM 100k split, retrieval sufficiency reaches 93.3% across all ten abilities.

## More than one judge

LLM-as-judge is convenient and biased. We score with multiple judges and report them, so a flattering judge cannot carry a result on its own.

## Cost next to accuracy

An answer that costs too much is not an answer you can ship. Every benchmark table reports cost and latency next to accuracy, and every /answer response reports its own cost in dollars at runtime. Answering from memory comes out 23x cheaper per question than resending 1M tokens of history on every call.

## Run it on your own data

The numbers above are reproducible, and the harness, splits and judges are documented on the [benchmarks page](/benchmarks). The stronger test is your own history: evaluate past.dev with updates, identity resolution, historical queries and unsupported questions from your own data, where /answer abstaining honestly matters more than any leaderboard.