---
title: "Cutting LLM token costs with memory"
description: "LLM token costs scale with how much history you send per question. The arithmetic of context stuffing, what caching changes, and what memory changes."
canonical: https://past.dev/blog/llm-token-costs
date: 2026-09-01
category: Research
authors: The past.dev team
---
# Cutting LLM token costs with memory

LLM token costs follow one rule: you pay for every input token, on every call. An agent that answers questions by re-sending its history pays for that history each time it is asked anything. At 1M tokens of accumulated history, every question starts at 1M input tokens before the question itself arrives. This post walks the arithmetic, what prompt caching does and does not change, and what replacing re-sent history with recalled evidence changes.

## The arithmetic of context stuffing

Take an input price of p dollars per million tokens. An agent whose context holds H tokens of history answers each question for roughly H/1,000,000 x p dollars of input, plus the question and the output. Three consequences follow directly.

- **Cost per question grows with history, forever.** The tenth question over a 1M-token history costs the same as the first. At an illustrative $3 per million input tokens, each question starts at $3.00. A hundred questions a day is $300 a day before any output token is counted.
- **The ceiling arrives before the bill.** Context windows top out. An agent that accumulates 50K tokens a day crosses a 1M window inside a month, and after that something must be dropped regardless of budget. See [context window](/glossary/context-window).
- **Quality falls before the ceiling.** Long-context models score worse on content buried mid-context, an effect measured across published long-context evaluations and separate from price. See [context rot](/glossary/context-rot).

## What prompt caching changes

Prompt caching reprices repeated prefixes; cached input tokens are billed at a discount on supported models. It genuinely helps a stable, append-only transcript. It does none of the following: cached tokens still cost money on every read, the cache expires and repopulates at full price, any edit above the change invalidates everything after it, and the model still attends over the same haystack, so the quality problem is untouched. Caching lowers the slope. It does none of the structural work.

## What memory changes

A memory layer inverts the shape of the call. History is written once, structured at ingestion, and each question retrieves only the evidence relevant to it. The model reads a few thousand tokens of dated, attributed evidence instead of the full transcript, whatever the total history size. Cost per question then tracks the answer's evidence rather than the archive's size, which is the difference between paying for what you ask and paying for what you have.

Two properties matter more than raw size reduction. Retrieved evidence carries [provenance](/glossary/data-provenance), so the smaller context is also a checkable one. And a structured store handles [facts that change over time](/guides/facts-that-change-over-time): a transcript contains every value a fact has ever had, and re-sending all of them leaves the disambiguation to the model at inference time.

On our benchmark workload, retrieving from memory costs 20x less per question than sending 1M tokens of history. The workload, judges and configurations behind that sentence are documented on the [methodology page](/benchmarks/methodology), and accuracy at each history size is published on the [benchmarks page](/benchmarks): cheaper answers are only interesting if they stay correct. The [quickstart](/docs/memory-api/quickstart) shows the two calls involved. [How to reproduce this →](/benchmarks/methodology)