How should we test an AI agent’s memory?

When we talk about an AI agent’s memory, we often ask what it can recall. In real work, a more useful question is whether relevant history helps it complete a task—and what that improvement costs in time and computation.

DolphinBench explores this with long histories and several simulated personas, comparing tasks with and without access to relevant history. It moves beyond simple recall questions by measuring task performance alongside cost.

A benchmark still cannot stand in for real use. Results depend on the personas, tasks, and how histories are constructed. For a product, I would look beyond accuracy: what does the memory retain, how can it be deleted, and can the user correct it?