---
title: "DolphinBench: past.dev at 81.7% of 600 tasks"
description: "past.dev passed 490 of DolphinBench's 600 tasks with Hermes Agent and GPT 5.6 Luna, the highest score reported on the benchmark so far."
canonical: https://past.dev/blog/dolphinbench-results
date: 2026-10-19
category: Research
authors: The past.dev team
---
# DolphinBench: past.dev at 81.7% of 600 tasks

Mem0 released DolphinBench in September 2026. It tests whether an agent with memory does the right thing, rather than whether it can repeat what it was told. With past.dev as its memory, Hermes Agent, an open-source agent from Nous Research, passed 490 of the 600 tasks (81.7%) on GPT 5.6 Luna. That is the highest score reported on the benchmark so far. With our own agent in place of Hermes, on the same tasks and model, the same memory passed 520 tasks for $15.77 in total, with a median of 9.8 seconds per task.

![DolphinBench tasks passed on Hermes Agent with GPT 5.6 Luna, as bars running from zero to 600: past.dev 490, Mem0 424, Hindsight 417, Honcho 411, Hermes built-in memory 394, Supermemory 355.](/blog/dolphinbench/field.svg)

*Not run directly by Mem0, but available for anyone to test.*

## What is DolphinBench

[DolphinBench](https://dolphinbench.ai) is Mem0's open benchmark for agent memory. The code is [on GitHub](https://github.com/mem0ai/dolphinbench) under the Apache License 2.0, and the method is described in their [paper](https://arxiv.org/abs/2609.24971). It gives an agent months of a person's work history, then asks it to do tasks that depend on that history.

There are three simulated users: Morgan runs a startup, Alex works on infrastructure and Riley manages a product. Each one has about 500,000 tokens of dated history from a few months of work: the messages, emails and documents they sent and received.

The agent reads the whole history and saves what it needs to its memory. Then the memory is locked, so nothing new goes in, and the agent gets 200 tasks per user. Each task starts a new conversation, so the agent has none of the history in front of it.

The tasks are jobs: send an email, add a calendar event, review a pull request, write a document. Each one depends on something the user said earlier. The benchmark checks the result in two ways. Automatic checks look at which app the agent used and what it sent, and a model grades anything the agent wrote. Tasks an agent can pass without the history are left out, so all 600 test memory.

A recall benchmark can miss the mistakes that matter here. An agent can quote a conversation back word for word and still book the meeting in the wrong timezone. On DolphinBench, that task fails.

## The DolphinBench results

Six memories ran on the same agent and model, Hermes Agent with GPT 5.6 Luna: past.dev, Mem0, Hindsight, Honcho, Supermemory and the memory built into Hermes. past.dev passed 490 of the 600 tasks (81.7%), the highest of the six. Mem0's own memory follows at 424.

| Memory | Tasks passed | Share | Cost | Median latency |
|---|---|---|---|---|
| past.dev, self-run | 490 / 600 | 81.7% | $196.56 | 46.3 s |
| [Mem0](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/mem0) | 424 / 600 | 70.7% | $96.21 | 37.7 s |
| [Hindsight](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/hindsight) | 417 / 600 | 69.5% | $84.65 | 55.3 s |
| [Honcho](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/honcho) | 411 / 600 | 68.5% | $142.85 | 44.7 s |
| [Hermes built-in memory](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/builtin) | 394 / 600 | 65.7% | $61.48 | 44.4 s |
| [Supermemory](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/supermemory) | 355 / 600 | 59.2% | $345.64 | 52.7 s |

Mem0 ran the other five themselves, with version `81cb6f8` of the DolphinBench code, and published every agent conversation. We ran past.dev ourselves on the same 600 tasks with the same checks. Mem0 hasn't published our result yet; the reason is further down.

Task by task, past.dev passes 108 tasks that Mem0 fails, and Mem0 passes 43 that past.dev fails. Against each of the other four memories, past.dev passes between 114 and 161 tasks that they fail, and they pass between 27 and 36 that past.dev fails. A standard statistical test (exact McNemar) puts the chance of a gap this large between two equally good memories below one in seven million, in every comparison.

### Where the lead comes from

DolphinBench's tasks fall into two groups. 261 ask the agent to write a document. The other 339 ask for a different action: an email, a calendar event, a review, an order.

![Share of tasks passed as vertical bars, two per memory, actions then documents: past.dev 82.0% and 80.8%, Mem0 69.3% and 72.4%, Hindsight 69.0% and 70.1%, Honcho 67.0% and 70.5%, Hermes built-in memory 59.6% and 73.6%, Supermemory 57.8% and 60.9%.](/blog/dolphinbench/split.svg)

past.dev comes first on both kinds, by 43 tasks on actions and 22 on documents, and it is the only memory that passes a larger share of the actions than of the documents. That fits what each kind of task needs. You can write a good document from a rough idea of a project. To send an order update to the right person, the agent needs the exact fact, and it has to be the latest version of it.

## Cost and latency

With Hermes, our run cost more and was slower than Mem0's. It cost $196.56 against $96.21, and our median task took 46.3 seconds against 37.7. Most of the cost came from loading the history: sending the 13,539 history messages through Hermes cost $168.73 on its own.

To see how much of that comes from Hermes itself, we ran the same 600 tasks with our own agent in place of Hermes, on the same model and the same memory. It passed 520 of 600 (86.7%) for $15.77 in total, with a median of 9.8 seconds per task. So most of the cost and time in the Hermes run comes from Hermes, rather than from the memory.

![Tasks passed against total cost on a log scale: past.dev with its own agent 520 for $15.77, past.dev with the DolphinBench setup 490 for $196.56, Mem0 424 for $96.21, Hindsight 417 for $84.65, Honcho 411 for $142.85, Hermes built-in memory 394 for $61.48, Supermemory 355 for $345.64.](/blog/dolphinbench/cost.svg)

![Median seconds per task as columns: past.dev with its own agent 9.8, set apart from the runs on Hermes Agent: Mem0 37.7, Hermes built-in memory 44.4, Honcho 44.7, past.dev 46.3, Supermemory 52.7, Hindsight 55.3.](/blog/dolphinbench/latency.svg)

That run cost less, and had a shorter median task, than every one of the 13 official DolphinBench results, on any agent and model. It uses a different agent from Hermes, so it isn't a like-for-like comparison with the table above. Our [DolphinBench page](/benchmarks/dolphinbench) lists it separately.

## Why Mem0 hasn't published our results yet

We ran both of our results ourselves, and Mem0 hasn't checked them yet. The [official leaderboard](https://dolphinbench.ai/leaderboard/) and its [results file](https://dolphinbench.ai/leaderboard/results.json) list 13 results, and none of them is past.dev.

We have tried to get them published. The Hermes run can't be uploaded: its records come to 1,341 MiB of JSON, and DolphinBench's packaging tool accepts at most 512 MiB per file. We submitted the own-agent result on 6 October. DolphinBench's own checker code accepts it at 520 of 600 when we run it, but the check on their site ran out of memory, and the submission is still waiting in their queue for an administrator. We've also asked Mem0 to run past.dev on Hermes themselves. Until one of these happens, our numbers are self-reported. As of 19 October, their [Run and submit](https://dolphinbench.ai/run/) page also says: "The submission service is temporarily unavailable. Try again."

## Thank you to Mem0

Mem0 built a benchmark that someone else can come first on. They released the code under the Apache License 2.0, published their own results with every agent conversation and wrote the paper, knowing that companies like ours would run it and publish the results.

Before DolphinBench, a company could claim an agent memory result and nobody could check how it was measured, let alone reproduce it. Mem0 have been shaping this field since before most people had a word for it, and we're grateful to have them in it. Congratulations to [Soumil Rathi](https://www.linkedin.com/in/soumilrathi/), [Deshraj Yadav](https://www.linkedin.com/in/deshrajdry/) and [Taranjeet Singh](https://www.linkedin.com/in/taranjeet7114/), and thank you for the work. Their launch post, [Introducing DolphinBench: Mapping the Pareto Frontier of Agent Memory](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory), explains the design in the authors' own words.

Mem0's own memory has the best of the five official results with Hermes and GPT 5.6 Luna. With Claude Code and Claude Sonnet 5, Honcho comes first. We trust a benchmark more when its author doesn't come first every time.

## Why this benchmark now

A year ago, agent memory was tested by asking a model questions about a transcript. That made sense for the agents of the time.

Agents now do more. [Instinct](https://instinct.com) takes an instruction by text or phone and does the task on a computer. Meta's [Muse](https://muse.ai) runs as a personal agent on its own virtual machine. [Town](https://www.town.com) runs assistants across more than fifty of a company's tools. These products act in people's real accounts, on facts that were true last Tuesday and may have changed since.

DolphinBench is the first public benchmark we know of that tests this kind of work. We see it as a companion to recall benchmarks, and we publish on both.

BEAM is a recall benchmark: it asks questions about very long histories. past.dev has the highest published BEAM scores at every history size from 100K to 10M tokens, including 85.03% at ten million, where memory systems usually lose most of their accuracy. We ran the public BEAM prompts ourselves, and each competitor's figure comes from that vendor's own publication, with different models and judges. The [BEAM leaderboard](/benchmarks/beam) lists every size with its source.

BEAM tests whether a system still knows what's true after ten million tokens, and DolphinBench tests whether an agent can act on it. Every figure we publish, with its run date and the source of each competitor's number, is on [benchmarks](/benchmarks). Every DolphinBench result, for each agent and model Mem0 tested, is on [our DolphinBench page](/benchmarks/dolphinbench).