Token Drift Explained: Why Your Agent Gets Slower and More Expensive

What Token Drift Actually Means

Most conversational agents build each request from several sources: a system prompt, tool definitions, recent messages, retrieved context, durable memory, and sometimes a summary of older work. Whether the application sends that state on every request or a provider manages part of it, the model still has an effective context to process.

Consider an illustrative session:

The exact price and latency depend on the model, provider, cache behavior, and workload. The important signal is the trend: later calls repeatedly process a larger context.

Why Agents Accumulate Tokens So Quickly

Conversation history is only one source of growth. Production agents often accumulate tokens in several places at once:

• Repeated transcripts: Every prior user and assistant message remains in context.
• Tool schemas: Large tool descriptions and JSON schemas may be attached to every model call.
• Tool results: Search results, stack traces, database rows, and API responses can be much larger than the user's request.
• Retrieved documents: RAG pipelines sometimes add too many chunks or keep stale retrieval results across turns.
• Retries and repairs: Failed tool calls and validation errors may be appended even after they stop being useful.
• Duplicated memory: A summary, structured state, and raw transcript may all contain the same facts.

This is why limiting the number of chat messages is useful but incomplete. A ten-message conversation can still be expensive if one tool returned 30,000 tokens of JSON.

The Cost Curve

Suppose a request has a fixed cost of s tokens for system instructions and tools, and each turn adds roughly m tokens of new history. The input on turn t is approximately:

input(t) = s + (t * m)

Per-turn input grows roughly linearly. However, the cumulative input processed across n turns is approximately:

session input = (n * s) + m * n * (n + 1) / 2

That second term is quadratic. This distinction matters: the final request is not necessarily exponentially larger, but repeatedly resending a growing transcript can make the total session cost rise much faster than the number of turns.

Prompt caching can reduce the cost of repeated prefixes on providers that support it. It does not remove context-window limits, irrelevant-history problems, or the need to control tool and retrieval payloads.

Measure Drift Before You Trim

Use the token usage returned by the model API whenever possible. Character-based estimates are acceptable for admission control, but they are not a reliable billing metric across models or languages. Track usage per model call, not only per user request. One agent turn may contain planning, tool calls, retries, and a final response.

type UsageSample = {
  turn: number;
  step: string;
  inputTokens: number;
  outputTokens: number;
  cachedInputTokens?: number;
  durationMs: number;
};

function averageInputGrowth(samples: UsageSample[]): number {
  if (samples.length < 2) return 0;
  const byTurn = new Map<number, number>();
  for (const sample of samples) {
    byTurn.set(sample.turn, (byTurn.get(sample.turn) ?? 0) + sample.inputTokens);
  }
  const totals = [...byTurn.entries()]
    .sort(([a], [b]) => a - b)
    .map(([, tokens]) => tokens);
  if (totals.length < 2) return 0;
  const deltas = totals.slice(1).map((value, index) => {
    return value - totals[index];
  });
  return Math.round(
    deltas.reduce((sum, delta) => sum + delta, 0) / deltas.length
  );
}

A useful trace should also record the model, operation, tool name, retry count, retrieval chunk count, and whether cached input was used. Avoid recording sensitive prompt content unless your privacy policy explicitly permits it.

Use a Context Budget, Not a Message Limit

A sliding window is a reasonable fallback, but a token budget is the stronger control because message sizes vary dramatically. Reserve space for the output and for any tool calls the model may make, then allocate the remainder to fixed context and recent turns. Treat a complete agent turn as the unit of removal. Dropping an individual tool result while keeping the assistant message that refers to it can leave an invalid or confusing transcript.

type AgentMessage = {
  role: 'user' | 'assistant' | 'tool';
  content: string;
};

type ConversationTurn = {
  id: string;
  messages: AgentMessage[];
  tokenCount: number;
};

type ContextPlan = {
  turns: ConversationTurn[];
  omittedTurnIds: string[];
  estimatedInputTokens: number;
};

type ContextBudget = {
  maxContextTokens: number;
  reservedOutputTokens: number;
  systemPromptTokens: number;
  toolSchemaTokens: number;
  summaryTokens: number;
  retrievedContextTokens: number;
};

function planContext(
  turns: ConversationTurn[],
  budget: ContextBudget
): ContextPlan {
  const fixedTokens =
    budget.systemPromptTokens +
    budget.toolSchemaTokens +
    budget.summaryTokens +
    budget.retrievedContextTokens;
  let remaining =
    budget.maxContextTokens - budget.reservedOutputTokens - fixedTokens;
  if (remaining <= 0) {
    throw new Error('Fixed context exceeds the available input budget');
  }
  const selected: ConversationTurn[] = [];
  for (let index = turns.length - 1; index >= 0; index -= 1) {
    const turn = turns[index];
    if (turn.tokenCount > remaining) break;
    selected.unshift(turn);
    remaining -= turn.tokenCount;
  }
  if (selected.length === 0 && turns.length > 0) {
    throw new Error('The latest turn is larger than the context budget');
  }
  const selectedIds = new Set(selected.map((turn) => turn.id));
  const omittedTurnIds = turns
    .filter((turn) => !selectedIds.has(turn.id))
    .map((turn) => turn.id);
  return {
    turns: selected,
    omittedTurnIds,
    estimatedInputTokens:
      fixedTokens + selected.reduce((sum, turn) => sum + turn.tokenCount, 0),
  };
}

The token counts in this example should come from the provider's tokenizer or a compatible tokenizer for the chosen model. Recalculate the plan whenever tools, retrieval results, or the summary changes.

Separate Conversation From State

Not every fact belongs in the transcript. Long-running agents become more reliable when they store important state explicitly:

type TaskState = {
  objective: string;
  constraints: string[];
  decisions: Array<{ choice: string; reason: string }>;
  openQuestions: string[];
  artifactIds: string[];
};

Structured state is compact, inspectable, and easier to validate than a prose summary. It also prevents a critical decision from disappearing simply because an old message fell outside the recent window.

A practical context often has four layers:

• Stable instructions: The system prompt and safety constraints.
• Structured task state: Goals, decisions, identifiers, and unresolved work.
• Compressed history: A summary of older turns.
• Recent turns: The verbatim messages and tool exchanges needed for continuity.

This layered design is usually more useful than importance-scoring individual messages. Heuristic scores can be difficult to explain, and they may discard a quiet but essential constraint.

Summarize Deliberately

Summarization is valuable when sessions last long enough that a recent window loses necessary context. It should not run blindly on every turn. Create or refresh a summary only when older turns cross a threshold. Ask the summarizer to preserve facts, decisions, constraints, failures, unresolved questions, and references to external artifacts. Store the summary separately from the raw transcript so it can be reviewed or regenerated.

Summaries are lossy and can introduce errors. Keep canonical values such as account IDs, approval state, financial amounts, and file paths in structured storage rather than trusting generated prose.

Bound Tools and Retrieval Too

Many apparent memory problems are actually payload problems. A useful context policy should also:

• Send only the tool definitions available in the current state.
• Replace large tool results with a compact, typed projection.
• Store full results externally and keep an ID or URL in context.
• Limit retrieval by both relevance and token budget.
• Deduplicate overlapping chunks before adding them to the prompt.
• Remove failed retries once their diagnostic value has expired.

For example, an agent rarely needs an entire database response. It may need five selected fields, the result count, and an identifier it can use to fetch more data later.

Choose a Strategy by Workload

Most systems do not need a sophisticated memory-ranking algorithm on day one. They need visibility, a budget, and clear rules for what may enter context.

Production Quality Gates

Context reduction is successful only if it lowers resource use without damaging task quality. Compare the policy against representative sessions and track:

• Input tokens per turn and per completed task
• Cached versus uncached input tokens
• Model-call count, retries, and tool failures
• Median and tail latency
• Task completion and human correction rates
• Missing facts or constraints after compaction
• Summary refresh frequency and cost

Run deterministic regression cases for critical facts and tool sequences. For example, verify that an agent still remembers an approval constraint after the originating turn has moved into summarized history.

A Practical Baseline

Start with a simple policy: measure real usage, reserve output headroom, cap the input context, keep structured task state, retain a small window of complete recent turns, and summarize older history only when necessary. Bound tool and retrieval payloads independently.

Token drift is not mysterious model degradation. It is accumulated application state. Once that state is measured and budgeted, cost and latency become predictable engineering trade-offs instead of late-session surprises.