--- title: "DolphinBench leaderboard: agent memory scored on actions" description: "Every published DolphinBench result in one place, from Mem0's official runs to past.dev's self-run 489 of 600, each figure linked to its source." canonical: https://past.dev/benchmarks/dolphinbench last-updated: 2026-10-20 --- # DolphinBench leaderboard > DolphinBench scores agent memory on what an agent does, across 600 tasks. past.dev reports 490 of 600 with Hermes Agent and GPT 5.6 Luna, against 424 for Mem0's own memory. That run is self-administered and absent from DolphinBench's leaderboard. Source: https://past.dev/benchmarks/dolphinbench DolphinBench is Mem0's benchmark. This page is past.dev's own publication of the results. It is not an official leaderboard. ## Hermes Agent with GPT 5.6 Luna The configuration with the most entrants. Five rows are Mem0's own runs at commit 81cb6f8; past.dev's row is our own run. | Memory | Tasks passed | Cost | Median latency | Run by | Source | | --- | --- | --- | --- | --- | --- | | past.dev | 490 / 600 (81.7%) | $196.56 | 46.3 s | past.dev, self-run | [past.dev harness](https://github.com/pastdotdev/benchmarks) | | Mem0 | 424 / 600 (70.7%) | $96.21 | 37.7 s | Mem0 | [Mem0 run records](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/mem0) | | Hindsight | 417 / 600 (69.5%) | $84.65 | 55.3 s | Mem0 | [Mem0 run records](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/hindsight) | | Honcho | 411 / 600 (68.5%) | $142.85 | 44.7 s | Mem0 | [Mem0 run records](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/honcho) | | Hermes built-in memory | 394 / 600 (65.7%) | $61.48 | 44.4 s | Mem0 | [Mem0 run records](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/builtin) | | Supermemory | 355 / 600 (59.2%) | $345.64 | 52.7 s | Mem0 | [Mem0 run records](https://github.com/mem0ai/dolphinbench/tree/81cb6f8405b40a9e76089cef650806a80af06ea2/results/hermes/gpt-5.6-luna/supermemory) | ## Hermes Agent with MiniMax M3 A weaker model, and the ordering is unchanged. past.dev has not run this configuration. Supermemory is absent from Mem0's published table for it. | Memory | Tasks passed | Cost | Median latency | Run by | Source | | --- | --- | --- | --- | --- | --- | | Mem0 | 47.8% | $148.02 | 47.0 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | | Honcho | 47.3% | $211.85 | 80.3 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | | Hindsight | 41.0% | $158.33 | 144.3 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | | Hermes built-in memory | 26.5% | $107.03 | 51.4 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | ## Claude Code with Claude Sonnet 5 Here Honcho leads and Mem0 does not. past.dev has not run this configuration. Hindsight and Supermemory are absent from Mem0's published table for it. | Memory | Tasks passed | Cost | Median latency | Run by | Source | | --- | --- | --- | --- | --- | --- | | Honcho | 35.8% | $1,565.82 | 37.9 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | | Mem0 | 32.3% | $1,830.57 | 42.3 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | | Built-in memory | 26.3% | $1,132.75 | 32.2 s | Mem0 | [Mem0 launch post](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory) | Cost is the agent model plus memory processing with grading excluded, on DolphinBench's own basis. Latency is the median task. Retrieved October 19, 2026, at DolphinBench commit 81cb6f8. ## past.dev's own agent A different agent from Hermes, so it stands as its own configuration rather than a row above. | Configuration | Tasks passed | Cost | Median latency | | --- | --- | --- | --- | | past.dev's own agent, GPT 5.6 Luna, past.dev memory | 520 / 600 (86.7%) | $15.77 | 9.8 s | Wilson 95% interval 83.7% to 89.2%. Submitted to the DolphinBench site on October 6, 2026. Their hosted validation ran out of memory on the evidence file, so the submission shows as queued and awaiting administrator attention. ## Tasks by kind DolphinBench's 600 tasks split into 261 that ask for a document and 339 that expect another action. Hermes Agent with GPT 5.6 Luna: | Memory | Other actions (339) | Documents (261) | | --- | --- | --- | | past.dev | 278 | 211 | | Mem0 | 235 | 189 | | Hindsight | 234 | 183 | | Honcho | 227 | 184 | | Hermes built-in memory | 202 | 192 | | Supermemory | 196 | 159 | ## How to read a single run - Every figure on this page is one run of 600 tasks. Between past.dev's own two passes on the same frozen memory, 58 tasks changed outcome. - The Wilson 95% interval on past.dev's 81.7% runs from 78.4% to 84.6%. Read the large gaps between rows and treat the small ones as noise. - Matched task by task, past.dev passes 108 tasks that Mem0 fails and fails 43 that Mem0 passes, at an exact McNemar p below 1.3e-7. The same comparison against the other official rows runs from 114 to 36 up to 161 to 27. ## What past.dev's run does not have past.dev's Hermes result is self-administered. DolphinBench has not run or verified it and it is absent from their leaderboard. Its records come to 1,341 MiB of JSON, over the 512 MiB per file that DolphinBench's packaging tool accepts, so it cannot be uploaded to their site. past.dev has asked Mem0 to run it as an official configuration. In its place past.dev publishes the harness, the per task records in the format of the official runs, the raw evidence archives and a disclosures list longer than the results. Two items a careful reader should take first: the reported run is a second pass on the same frozen memory whose first pass passed 363 of 600, and its toolset configuration was chosen after side experiments on these same tasks with no held-out set. ## What DolphinBench measures DolphinBench scores agent memory on the action an agent takes. Three simulated users carry about 500,000 tokens of dated history each: Morgan runs a startup, Alex works on infrastructure, Riley manages a product. A configuration sends that whole history through its agent and its memory, freezes the memory, then runs 200 tasks per user in fresh conversations with no history in the context. The tasks are jobs: send the email, add the calendar entry, review the pull request, write the document. Grading is on what the agent did. Deterministic checks read which app it called and with what arguments, and a model judge reads anything written. A task an agent can pass without the history is excluded, so all 600 are memory tasks. The paper: [https://arxiv.org/abs/2609.24971](https://arxiv.org/abs/2609.24971). The benchmark: [https://github.com/mem0ai/dolphinbench](https://github.com/mem0ai/dolphinbench). Mem0's launch post: [Introducing DolphinBench](https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory). ## Attribution and licence DolphinBench was created by Mem0 and is published under the Apache License 2.0 at https://github.com/mem0ai/dolphinbench, read here at commit 81cb6f8. The paper is Rathi, Yadav and Singh, DolphinBench: Mapping the Pareto Frontier of Agent Memory, 2026, under CC BY 4.0. Every official figure on this page is Mem0's published result, reproduced with attribution and linked to its source. past.dev's harness extends DolphinBench code, and those adapted portions stay under the Apache License 2.0 as recorded in the harness NOTICE. Nothing on this page implies that Mem0 endorses past.dev. Hermes Agent is Nous Research's work, under the MIT License: https://github.com/NousResearch/hermes-agent. ## Related - [past.dev benchmark results across every benchmark](https://past.dev/benchmarks) - [BEAM leaderboard: long-term memory at 100K to 10M tokens](https://past.dev/benchmarks/beam) - [How to benchmark agent memory](https://past.dev/benchmarks/methodology) - [LoCoMo, explained](https://past.dev/benchmarks/locomo)