---
title: "Testing entity resolution in agent memory"
description: "A working method for testing entity resolution: a gold set built from your identifier matrix, five standard cases, and an end-to-end assertion."
canonical: https://past.dev/guides/testing-entity-resolution
last-updated: 2026-09-02
---
# Testing entity resolution in agent memory

Source: https://past.dev/guides/testing-entity-resolution

Testing entity resolution means checking that a memory system links records of the same subject and keeps different subjects apart. The method: list every way each person actually appears in your data (full name, first name, email, handle, nickname, misspelling), build a gold set of pairs from that matrix, and rerun it on every model or pipeline change. Failures are invisible in demos and expensive in production, because a wrong link corrupts stored history rather than a single response.

## Failures are invisible until production

Entity resolution has a testing problem: correct behavior and broken behavior look identical in a demo. Demo data tends to mention each person one way. Production data mentions the same person as a full name in a contract, a first name in chat, an email address in a ticket and a nickname in a transcript.

Failures also surface far from their cause. A resolver quietly merged two customers named Chris Lee on March 4, 2026. The visible symptom arrived in June, when a refund conversation cited the other Chris Lee's order history. By then the merged entity had accumulated three months of interleaved facts. That distance in time between defect and symptom is the reason resolution needs its own test suite rather than being covered incidentally by application tests.

## Build a gold set from your identifier matrix

A gold set is a versioned list of resolution cases with known correct outcomes. Build it from your own data rather than a public corpus: the variants that matter are the ones your users actually produce.

Start with an identifier matrix. For each of a dozen real people (internal accounts, or customers who consented), list every way that person appears in your sources.

```anidentifiermatrixentry
person: dana-reyes
appears as:
  full name      Dana Reyes
  first name     Dana
  email          dana@northwind.com
  handle         @dreyes
  nickname       Dee
  misspelling    Dana Rayes
```

From the matrix, generate positive pairs (two appearances of the same person) and negative pairs (appearances of different people). Include hard negatives: a second Dana Reyes who works at another company, and a `d.reyes@` address that belongs to someone else. Record the expected outcome for every pair and keep the set in version control next to the pipeline.

## Coreference resolution and cross-record resolution

Coreference resolution is the NLP task of deciding which mentions inside one document refer to the same entity: pronouns (she, they), definite references (the CFO, the vendor) and repeated names. Cross-record entity resolution decides whether records in different sources describe the same subject. A memory pipeline needs both, in sequence.

|  | Coreference resolution | Cross-record entity resolution |
| --- | --- | --- |
| Scope | Mentions within one document | Records across sources |
| Signals | Grammar, discourse structure, proximity | Identifiers, field similarity, shared keys |
| Typical failure | A pronoun attached to the wrong speaker | Two sources kept apart or wrongly joined |

The two errors compound. If coreference attaches `she approved the budget` to the wrong participant in a transcript, entity resolution then files a wrong fact under a correctly resolved entity. Test the stages separately, then test them together.

## Five standard test cases

1. **Three identifiers, one person.** Ingest an email from `dana@northwind.com`, a chat message from `@dreyes` and a ticket filed by Dana Reyes. Assert that one entity holds all three facts.
2. **Two people, one name.** Ingest records for Chris Lee at Acme and Chris Lee at Meridian. Assert two entities, and assert that a question about one never cites the other.
3. **A changed email.** Dana's address moves from `dana@northwind.com` to `dana@meridian.com` after a job change dated March 1, 2026. Assert one continuous entity, with the change on the timeline rather than a second person.
4. **A nickname introduced mid-history.** Early sources say Robert Alvarez; later chat says Bob. Assert that the nickname joins the existing entity.
5. **A role-only reference.** A memo dated June 2025 mentions only the CFO. Assert that it resolves to the person who held the role in June 2025, even after a new CFO starts in 2026. Role facts change over time; see [facts that change over time](/guides/facts-that-change-over-time).

## Precision and recall as concepts

Precision measures whether the links the system made are correct. Recall measures whether the links that truly exist were made. Under-merging is a recall failure. Over-merging is a precision failure.

The two failures cost differently in an assistant. A recall miss degrades an answer and is repaired by adding a link. A precision miss moves facts between people, can leak private context and requires a split to repair. Resolution for memory should therefore be tuned to favor precision, and the gold set should carry enough hard negatives to measure that choice. Track both directions of failure on your gold set; the [benchmark methodology](/benchmarks/methodology) describes scoring end-to-end recall on full question-answer tasks rather than isolated pair decisions.

## The regression protocol

1. **Rerun the gold set on every change** to the model, the prompt, the matching thresholds or the ingestion pipeline. Resolution behavior shifts with any of them.
2. **Treat gold-pair violations as release blockers.** A link between two people the gold set marks as different is a precision regression, whatever else improved.
3. **Investigate cluster growth.** A cluster that absorbs records from two different gold-set people signals a bad transitive edge.
4. **Refresh the matrix on a schedule.** New channels introduce new identifier shapes; a matrix built before your SMS launch covers no phone numbers.

> **Blocker**
>
> A link between two gold-set people fails the release. Ship improvements that keep the gold pairs intact.

## The end-to-end assertion

Pair-level tests check decisions. One more test checks the outcome that matters: ask a question whose answer requires joining all the identifiers, with dates. Ingest three sources: an email from `dana@northwind.com` dated March 12, 2026 approving the budget; a chat message from `@dreyes` dated March 19 moving the deadline to April 30; a ticket by Dana Reyes dated April 2 cutting the pilot to one office. Ask what Dana decided in spring 2026. A passing answer cites all three facts with their dates. An answer with two locates the broken edge.

past.dev resolves entities at ingestion and labels each recall with a status value (Supported, Conflicted, NoKnownSupport or UnknownBecauseDegraded), so an end-to-end test can assert both the joined content and the status. The [Memory API quickstart](/docs/memory-api/quickstart) shows an ingest and recall pair that this kind of harness can script.

## Frequently asked questions

### How do I test entity resolution without labeled data?

Build your own labels: take a dozen real people from your data, list every way each appears, and generate positive and negative pairs from that matrix. A small gold set built from real variants beats a large public corpus that misses your formats.

### What is the difference between coreference resolution and entity resolution?

Coreference resolution links mentions inside one document, such as a pronoun to a name. Entity resolution links records across sources, such as an email address to a CRM contact. Memory pipelines need both.

### What is a gold set for entity resolution?

A versioned collection of record pairs with known correct outcomes, including hard cases such as two people who share a name. It is rerun on every pipeline change to catch regressions.

### Why does entity resolution work in testing but fail in production?

Test data usually mentions each person one way, while production data holds nicknames, misspellings, role references and old email addresses. Failures also appear long after the wrong link is made, in answers rather than in logs.

### What does precision and recall mean in entity matching?

Precision asks whether the links the system made are correct, and recall asks whether all true links were made. Over-merging two people is a precision failure; splitting one person across profiles is a recall failure.

## Related

- [Entity resolution for agent memory](https://past.dev/guides/entity-resolution)
- [How to merge duplicate profiles](https://past.dev/guides/merge-duplicate-profiles)
- [Identity resolution across channels](https://past.dev/guides/identity-resolution)
- [Benchmark methodology](https://past.dev/benchmarks/methodology)
- [Memory API quickstart](https://past.dev/docs/memory-api/quickstart)