HQAI Engineering HQ
Search
Checking sign-in...

Prompt caching

Concept8 min readintermediate

What you’ll understand

  • Explain how prompt caching reuses work for repeated input prefixes
  • Structure model requests to improve cache reuse without changing meaning
  • Distinguish prompt caching from semantic caches and the KV cache

Helpful first: Tokens in large language models · Context engineering

Many AI requests repeat a large amount of input. A coding assistant may send the same system instructions and tool descriptions on every turn. A document workflow may reuse the same policy manual for many questions.

Prompt caching lets a model provider reuse some of the computation for an identical or matching prompt prefix. When a request hits the cache, the repeated input can be cheaper or faster to process than reading it from the beginning again.

The exact behavior differs by model and provider. Some caches are automatic. Others require explicit cache controls. Minimum token lengths, expiration times, eligible models, and pricing rules also vary.

Stable prefix, changing suffix

Prompt caching usually works best when repeated content appears first and changing content appears later.

Stable prefix:
  system instructions
  safety rules
  tool definitions
  large reference document

Changing suffix:
  recent conversation
  current user request
  latest tool result

If two requests begin with the same eligible tokens, the provider may reuse the cached work for that prefix. The model still processes the changing suffix and generates a new answer.

This means request order matters. Putting a timestamp, request ID, or changing user data near the beginning can prevent the rest of the request from matching the cached prefix.

How a repeated prompt prefix produces a cache hit The first request contains a stable prefix of system instructions, tool definitions, and a reference document, followed by user question A. The provider performs prefill and stores reusable state for the stable prefix. The second request begins with the exact same stable prefix but has user question B. The provider reuses the cached prefix work and processes only the new suffix before generating a new answer. A third example puts a changing timestamp before the stable content, causing a cache miss because the prefix no longer matches. FIRST REQUEST Stable prefix system + tools + reference document Changing suffix user question A Provider prefill cache stable prefix state SECOND REQUEST Same stable prefix exact matching tokens New suffix user question B Cache hit reuse prefix; process suffix generate a new answer CACHE-BREAKING ORDER new timestamp otherwise stable system + tools + document miss: prefix changed early
Prompt caching rewards an exact, stable prefix followed by request-specific content; an early change can invalidate everything after it.

What a cache hit saves

During inference, the model first processes the input tokens in the prefill phase. Long inputs require more prefill work.

A prompt-cache hit can reuse provider-side state for the matching prefix. Depending on the API, this can reduce:

  • Billed input cost for cached tokens.
  • Time to first token.
  • Compute spent processing repeated instructions or documents.

It does not mean the whole response is reused. The model still considers the current suffix and generates a new output. A cache hit also does not guarantee lower end-to-end latency if output generation or network time dominates the request.

A worked example

Suppose an application sends this input on every request:

System instructions:       2,000 tokens
Tool definitions:          3,000 tokens
Product documentation:    20,000 tokens
Current user request:         80 tokens

The stable prefix is 25,000 tokens. Without prompt caching, each request processes all 25,080 input tokens.

If later requests reuse the same 25,000-token prefix, only the changing suffix needs fresh prefill work. The provider may still report all input tokens, but usually separates cached and uncached usage so billing and performance can be measured correctly.

The exact savings depend on provider prices and cache policy. Treat the numbers as an engineering measurement, not a promise.

Designing for cache reuse

Cache-friendly prompt design is mostly careful context engineering:

  • Put stable instructions before changing messages.
  • Keep tool definitions in a consistent order.
  • Use deterministic serialization for JSON schemas and configuration.
  • Avoid timestamps, random IDs, and per-request metadata in the prefix.
  • Reuse exact document text instead of reformatting it each time.
  • Append new conversation turns rather than rebuilding older turns differently.

Do not reorder information only for caching if the new order makes instructions less clear or changes model behavior. Correctness comes first.

What invalidates a cached prefix

A cache may miss when:

  • One token near the beginning changes.
  • Tool definitions are reordered.
  • Whitespace or formatting changes enough to alter tokenization.
  • The provider expires or evicts the cached entry.
  • The request uses a different model or model version.
  • The repeated prefix is shorter than the provider’s minimum.
  • Account, region, or deployment boundaries prevent sharing the entry.

Applications should work correctly on both hits and misses. A cache is an optimization, not a required source of state.

Prompt cache vs other caches

Several AI systems use the word “cache,” but they reuse different things.

Prompt caching reuses provider computation for repeated input tokens. The model still generates a new answer.

A semantic cache reuses an earlier final response when a new request is similar enough to a previous request. This happens at the application layer and can skip a model call entirely.

The KV cache stores attention keys and values during one active generation. It speeds token-by-token decoding by avoiding recomputation of earlier positions. Some serving systems can reuse prefix KV state across requests, but API-level prompt caching is a provider feature with its own eligibility and billing rules.

These techniques can be combined, but they solve different problems.

Privacy and isolation

Repeated prompts may contain private documents, user data, or internal instructions. Read the provider’s cache retention and isolation guarantees before enabling explicit caching.

Questions to check include:

  • Is cached state isolated by account, project, or organization?
  • How long can it remain available?
  • Can it be manually expired?
  • Is cached content used for training?
  • Are there regional or compliance restrictions?

Do not assume that “cached” means stored as readable text, or that it means nothing is retained. Use the provider’s precise terms.

Measuring prompt caching

Production telemetry should separate:

  • Total input tokens.
  • Cached input tokens.
  • Uncached input tokens.
  • Cache hit rate.
  • Time to first token.
  • Input cost and total cost.

A high hit rate is not automatically good. If the cached prefix contains irrelevant material, the application may be saving money while still sending poor context.

Test cache behavior with realistic request sequences. Change one part at a time and inspect the usage fields returned by the API.

Common mistakes

  • Assuming similar text will hit the cache. Prompt caching usually requires an exact matching prefix at the token level.
  • Putting changing data before stable content.
  • Treating cached input as free. Providers may charge a lower rate rather than zero.
  • Relying on the cache for correctness or memory.
  • Ignoring expiration and eviction.
  • Confusing a provider prompt cache with an application response cache.

The key idea

Prompt caching reuses work for repeated input prefixes. Keep stable content stable and early, measure real cache usage, and treat every hit as a performance optimization rather than a change to the model’s behavior.

Saved in this browser only. Not synced to your account.