--- title: "Hanabi: AI agents that reproduce a biology study" description: "Team Hanabi tested AI agents that reproduce a biology study from its raw data. In the swarm version, the agents share one memory on past.dev." canonical: https://past.dev/blog/projects/hanabi last-updated: 2026-11-18 --- # Hanabi: AI agents that reproduce a biology study Projects, November 18, 2026. The past.dev team. Team Hanabi is Timothé Chaumont, Sofia Bennani, Georges Khalil and Anthony Chassagne. At the Iterate AI Research Hackathon, they worked on AI agents that must reproduce a published biology study from its raw data. In one version, a swarm of agents shares one memory on past.dev. Demo video: [Demo of team Hanabi](https://www.youtube.com/watch?v=dGMmqsiH1js). The demo video of team Hanabi for the hackathon. The video does not show the results with past.dev. These results come from the repository of the team. - Built by: Timothé Chaumont ([GitHub](https://github.com/timothe-chaumont)), Sofia Bennani ([GitHub](https://github.com/sofiabennani)), Georges Khalil ([GitHub](https://github.com/GeorgesCKhalil)), Anthony Chassagne ([GitHub](https://github.com/anthony-chassagne)) - Event: [Iterate AI Research Hackathon](https://iterate.inc/ai-research), Wefunder, San Francisco, November 14 and 15, 2026 - Code on GitHub: https://github.com/timothe-chaumont/BixBench3 ## The project [BixBench3](https://arxiv.org/abs/2608.25286) is a benchmark for AI agents in biology. The agent gets the raw sequencing data of a published study, and it must rebuild the result tables of the study on its own. A grader compares each table with the published table and gives it a score from 0 to 1. The team used task 9, a study about a type of brain cancer. The agent must write 14 tables, and the score of a run is the mean of their 14 scores. The team used one model, GPT-5.4 Mini, for all the tests. It changed only the harness: the prompts, the tools and the checks around the model. Each version of the harness ran 5 times, because the same version can score 0.33 in one run and 0.93 in the next. | Version | What it adds | Mean score of 5 runs | |---|---|---| | Baseline | One agent with the official prompt of BixBench3 | 0.566 | | Environment | The exact tools of the study, installed before the run | 0.690 | | Environment and reviewer | A second call to the model checks the format of each table | 0.830 | | Environment and swarm | An orchestrator agent gives each part of the work to a sub-agent | 0.640 | | Environment, swarm and past.dev | The agents of the swarm share one memory on past.dev | 0.681 | In the swarm, an orchestrator agent plans the analysis. For each table or step, it writes a brief: a short text with the work to do and the files to use. Then it starts a sub-agent with this brief. Each sub-agent starts with an empty context. A sub-agent knows only its brief, so it must read the files again to learn what earlier sub-agents found. In one run of an earlier swarm version, the team saw these problems: - All 17 sub-agents read the same list of installed tools. - About 200 of the 810 tool calls read known facts again. - A convention is a rule for several tables, such as the source of the gene names. One convention did not reach the sub-agents of 2 tables, so the swarm had to build these tables again. ## How the memory works The memory code is in one Python file, [run_memory.py](https://github.com/timothe-chaumont/BixBench3/blob/543d986eaf796669d89159e622cc50bb5d77a468/src/bixbench3/inspect_harness/run_memory.py). The agents do not call past.dev: the harness calls the API for them. At the start of a run, the harness calls `PUT /api/v1/audiences/{slug}` to create one audience with one reader identity. Each record goes to this audience, so a run cannot read the records of another run. ### The harness writes four types of record The harness sends each record to `POST /api/v1/ingest`. No record needs an extra call to the model. | Record | Comes from | What it holds | |---|---|---| | Observation | The harness | Each command of an agent that only reads a file or lists a folder, with its output | | File | The harness | Each file that a sub-agent made, with its size and its first line | | Lesson | Each sub-agent | Conventions, commands that work, problems and fixes, and facts, from the end of the report of the sub-agent | | Convention | The orchestrator | A rule for several tables, from the plan of the orchestrator | This is a simplified version of one write and one read in run_memory.py: ```python BASE = "https://api.past.dev/api/v1" # Each record goes to the audience of the run post(f"{BASE}/ingest", { "id": f"{run_id}-{kind}-{key_hash}", # same key, same id "content": f"[{kind}] {title}\n{text}", "label": title, "timestamp": now, "metadata": {"kind": kind, "author": author, "run": run}, "audience": audience, }) # Before each sub-agent starts: its brief is the query post(f"{BASE}/recall", { "query": brief, "identity": reader, "sort": "relevance", }) ``` ### Each sub-agent starts with the records that match its brief 1. Before a sub-agent starts, the harness sends its brief as the `query` of `POST /api/v1/recall`. 2. past.dev returns the records of the run, the most relevant first. The harness adds them to the brief, up to 8,000 characters. 3. The harness also adds every convention of the run, without a ranking. 4. While it works, any agent can ask a question with a `recall` tool, for example "columns of the sample sheet". The team tells each sub-agent to reuse these records, and to read a file only when the records do not have the information. ## What memory changed The team compared the swarm without and with past.dev, with 5 runs each and no other change. The memory logs of the team show that past.dev answered in every run. BixBench3 also has a judge: a model that lists the failures in the log of a run, such as an inefficient analysis. In the table, each value is the mean of 5 runs, except for the spread. | Result | Swarm without past.dev | Swarm with past.dev | Change | |---|---|---|---| | Score | 0.640 | 0.681 | 6% higher | | Spread of the score (standard deviation) | 0.091 | 0.025 | 3.6 times smaller | | Failures that the judge found in a run | 1.8 | 1.2 | 0.6 fewer | | Tool calls in a run | 913 | 742 | 19% fewer | | Model calls in a run | 834 | 702 | 16% fewer | | OpenAI cost of a run | $3.60 | $3.21 | 11% lower | **Note:** The team writes that these differences are not statistically significant on their own. The reason is that each version has only 5 runs. The team also writes that all the differences favor the swarm with past.dev. ### The swarm with past.dev was the most stable version - 0.460 to 0.699: Lowest and highest score of the 5 runs without past.dev - 0.649 to 0.723: Lowest and highest score of the 5 runs with past.dev Scores of the swarm runs, from results/task9/data/runs.jsonl in the repository. With past.dev, the spread of the score was 3.6 times smaller. It is the smallest spread of the five versions. But the swarm with past.dev did not get the best mean score. The single agent with a reviewer had a mean score of 0.830, at $2.24 for each run. ### The swarm used fewer calls and cost less With past.dev, the swarm used 19% fewer tool calls and 16% fewer model calls. The orchestrator started 41 sub-agents on average, against 49 without memory. The cost of a run was 11% lower: $3.21, against $3.60. This cost is what the team paid OpenAI for the model calls. ### The judge found fewer failures The official judge reads only the messages of the orchestrator. So the team ran the judge on the full log of each swarm run. With past.dev, the judge found 1.2 failures in each run on average, against 1.8 without past.dev. ### Memory did not help the single agent The team also gave the reviewer of the single agent a past.dev memory of its accepted tables. The mean score fell from 0.830 to 0.663. Here the memory went only to the reviewer, and the agent that did the analysis did not get it. The team thinks that the causes are a quick implementation and the large variation between runs, with only 5 runs for each version. ### The team thinks that memory helps most when one context is too small The team thinks that a shared memory helps most when a task needs more context than one model can hold and still work well. A swarm splits the work over dozens of empty contexts: the harness stored 176 to 282 records in each swarm run with past.dev. A single agent holds all of task 9 in one context: with the reviewer, the harness stored only 14 records in each run. ## Try Hanabi 1. Read the write-up and the figures in the [README of the repository](https://github.com/timothe-chaumont/BixBench3). 2. Read each past.dev call in [run_memory.py](https://github.com/timothe-chaumont/BixBench3/blob/543d986eaf796669d89159e622cc50bb5d77a468/src/bixbench3/inspect_harness/run_memory.py). 3. Open [results/task9](https://github.com/timothe-chaumont/BixBench3/tree/main/results/task9) for the data of each run. To build agents that share one memory, start with the [quickstart of the Memory API](/docs/memory-api/quickstart). Thank you to Timothé, Sofia, Georges and Anthony for building with past.dev at the hackathon.