HQAI Engineering HQ
Search
Checking sign-in...

Context compaction

Concept8 min readintermediate

What you’ll understand

  • Explain why long-running AI interactions need context compaction
  • Compare trimming, summarization, extraction, and retrieval-based compaction
  • Design a compaction process that preserves decisions, constraints, and unresolved work

Helpful first: Context engineering · Tokens in large language models

An AI application cannot keep adding information to a model request forever. Conversation turns, documents, tool results, plans, and generated output all consume tokens. Eventually they approach the model’s context limit or make each request slow and expensive.

Context compaction reduces older or less useful information into a smaller form while trying to preserve what the model still needs. It is a practical part of context engineering for long conversations and multi-step agent tasks.

Compaction is not simply deleting the oldest messages. The goal is to preserve the task’s working state: important facts, decisions, constraints, completed steps, failures, and unresolved questions.

Turning growing history into compact working context A growing context contains stable rules, the goal, many conversation turns, repeated tool output, logs, decisions, and unresolved work. A compaction step keeps exact rules and constraints, extracts durable state, summarizes replaceable history, removes duplicates, and archives source records. The next model request receives a smaller active context with the goal, constraints, decisions, completed work, current step, and selected details. Archived source records remain available for retrieval. GROWING CONTEXT System rules + goal Conversation turns Repeated tool output Logs + retries Decisions + constraints Unresolved work Compact deliberately keep exact rules extract durable state summarize old turns remove duplicates archive source records validate the next step still works NEXT MODEL REQUEST Rules + current goal Decisions + constraints Completed + unresolved work Current step + selected details Archived source records retrieve exact details when needed Compaction shrinks the active window without making the original record disappear.
Keep the working state in the active context and move replaceable detail into summaries or retrievable source records.

Why context grows

A basic chat may begin with a short system instruction and one user message. A longer workflow can accumulate:

  • Every user and assistant turn.
  • Retrieved documents and citations.
  • Tool calls and their results.
  • Plans, intermediate drafts, and corrections.
  • Error messages and retry details.
  • Application state that changes during the task.

Sending all of that on every model call has costs. Inference takes longer because the model must process more input. The request costs more when providers bill by input token. Important facts can also become harder to find inside a large amount of stale or repeated text.

Compaction is useful before the hard limit is reached. Waiting until the request no longer fits forces an emergency decision, such as dropping content without checking whether it matters.

Four compaction techniques

Applications usually combine several techniques.

Trimming

Trimming removes items that are no longer useful. Examples include duplicate tool output, successful retry attempts, old status updates, and verbose logs whose result is already known.

Trimming is cheap and predictable, but it can lose information if the removal rule is too broad.

Summarization

Summarization rewrites many messages into a shorter account. A useful summary records facts and decisions, not just the general topic.

Weak summary:

The user and assistant discussed a deployment problem.

Useful summary:

Goal: restore the payments API.
Observed: version 42 fails health checks after 30 seconds.
Ruled out: database connectivity and expired credentials.
Decision: compare version 42 with version 41 configuration.
Unresolved: whether the new startup probe is too strict.

Structured extraction

Structured extraction stores important state in named fields instead of prose:

goal: restore payments API
completed:
  - checked database connectivity
  - checked credentials
current_hypothesis: startup probe is too strict
next_step: compare deployment configuration
constraints:
  - do not change production data

This form is easier for software to validate and update. It also reduces the chance that a later summary hides a critical constraint inside a paragraph.

Retrieval

Some details do not need to stay in every request. The application can store older messages, documents, or tool results outside the active context and retrieve them only when they become relevant again.

This is different from forgetting them. The information remains available, but the model does not pay the token cost on every turn.

A practical compaction process

A safe process separates durable task state from replaceable history:

  1. Reserve space for the next model response and likely tool results.
  2. Keep stable instructions and active user constraints.
  3. Extract decisions, facts, completed work, and unresolved items.
  4. Remove duplicate or superseded details.
  5. Summarize the remaining older conversation.
  6. Store source messages outside the active window when they may be needed later.
  7. Validate that the compacted state still supports the next step.

The application should record when compaction happened and which messages were replaced. That makes failures easier to debug.

What must survive

Good compaction preserves information that changes what the system should do:

  • The user’s current goal.
  • Safety, privacy, and permission constraints.
  • Decisions that have already been made.
  • Facts supported by tool results or trusted sources.
  • Work that is complete, failed, or still pending.
  • Identifiers, file paths, dates, and other exact values needed later.
  • Open questions and the expected next action.

Exact source material should remain exact when wording matters. Do not summarize a command, legal clause, code patch, or error message if the exact text may be needed. Keep it directly or store it for retrieval.

What can usually shrink

Lower-value details often include:

  • Repeated acknowledgements.
  • Earlier plans that were replaced.
  • Full tool output after the relevant result was extracted.
  • Duplicate copies of the same document.
  • Reasoning about options that are no longer possible.
  • Formatting and conversational phrasing that do not affect the task.

The distinction is task-specific. A stack trace may be disposable after a bug is fixed, but essential while the investigation is still active.

Compaction can introduce errors

A summary is generated data. It can omit a constraint, merge two separate facts, or state an uncertain idea as if it were confirmed. Repeated summarization can make this worse because each new summary is based on an older summary rather than the original source.

Reduce that risk by:

  • Keeping important state in structured fields.
  • Linking compacted facts back to source turns or tool results.
  • Marking uncertainty explicitly.
  • Rebuilding summaries from original records when practical.
  • Testing compaction with long, realistic tasks.
  • Allowing the model or user to request older source material.

For high-impact workflows, treat compacted context as an optimization, not the only record of truth.

Evaluating compaction

Do not evaluate compaction only by counting saved tokens. A smaller prompt is useful only if the system can still complete the task correctly.

Useful checks include:

  • Retention tests: can the model still answer questions about key decisions and constraints?
  • Continuation tests: can it choose the correct next action after compaction?
  • Contradiction tests: did old information survive after being replaced by newer information?
  • Token reduction: how much input was removed?
  • Latency and cost: did the smaller context improve production performance?

Save representative long-running traces and replay them with different compaction strategies. This makes compaction part of system evaluation rather than an emergency cleanup rule.

The key idea

Context compaction turns a growing interaction into a smaller working state. It should remove repetition while preserving the facts, decisions, constraints, and unresolved work that determine what happens next.

Saved in this browser only. Not synced to your account.