HQAI Engineering HQ
Search
Checking sign-in...

Semantic cache

Concept8 min readintermediate

What you’ll understand

  • Explain how a semantic cache answers similar prompts by comparing embeddings instead of exact text
  • Tune a similarity threshold and describe what goes wrong when it is too high or too low
  • Keep cached answers correct by including chat context in the cache key, and keep the cache from growing forever

Helpful first: Vector representations for AI systems · Vector distance and similarity metrics

A semantic cache stores past prompts and the answers an LLM gave, then reuses those answers when a new prompt means roughly the same thing. It matches by meaning, not by exact text, using a vector similarity search.

LLM calls are often the slowest and most expensive step in an AI app. Every call is billed and timed by tokens, and the work of inference takes longer as the input grows. A cache lets you skip the call when you already have a good answer.

Why LLM calls are worth caching

A short prompt is cheap. Real requests are not short:

  • LLMs do not remember earlier turns. To hold a conversation, the app sends part of the chat history with each new prompt. This sliding context window adds tokens and latency. See context engineering.
  • In retrieval-augmented generation (RAG), documents fetched from a database are added to the prompt too. Payloads of thousands of tokens and several seconds of waiting are common.

At high volume, many users ask the same thing in different words. A normal cache cannot help with that, because “How do I reset my password?” and “I forgot my password, how do I change it?” are different strings.

How a semantic cache works

A traditional cache looks up a key by exact string match. A semantic cache turns the lookup into a vector query:

  1. Convert the incoming prompt into an embedding.
  2. Search the stored prompt embeddings (the cache keys) for the closest ones.
  3. If a match is close enough, return its stored completion. This is a cache hit.
  4. If not, call the LLM, return its answer, and store the new prompt embedding and completion. This is a cache miss.
Semantic cache lookup flow A user prompt is embedded and used in a vector search over cached prompts. If the best similarity score meets the threshold, the stored completion is returned as a cache hit. If not, the app calls the LLM, returns the new completion, and writes the prompt embedding and completion into the cache. User prompt new question Embed text to vector Vector search over cached keys Good match? score vs threshold Hit stored answer Miss call the LLM Store prompt + answer yes no
A hit skips the LLM entirely. A miss pays for one LLM call and teaches the cache for next time.

Because the lookup is a query, not a single-key fetch, it can also return several close matches, ranked from most to least similar. An app can then pick the best one, or offer a few. This can reduce how many answers the LLM has to generate.

The similarity threshold

Every vector query returns a similarity score for each result, from 0 (nothing alike) to 1 (an exact match). Scores like cosine similarity are one common choice; see vector distance metrics for others. The cache uses a threshold on this score to decide what counts as a hit.

  • Threshold too high: only near-identical prompts hit. Most requests miss, and the cache fills with many answers to similar questions.
  • Threshold too low: loosely related prompts hit. Users get answers that do not match what they asked.

There is no universal right value. It depends on your embedding model and your domain, so expect to tune it using real prompts and check the answers it returns.

Choosing a similarity threshold A scale of similarity from 0 on the left to 1 on the right. A threshold set near 1 gives few hits and a bloated cache. A threshold set near 0 gives many hits but irrelevant answers. A middle value balances hits and relevance. The positions are illustrative, not recommended values. SIMILARITY SCORE 0 (unrelated) 1 (exact match) Too low many hits, wrong answers Tuned hits stay on topic Too high few hits, cache bloats
The threshold trades hit rate against relevance. Positions are illustrative; tune on your own data.

Include the conversation, not just the last prompt

Chat prompts depend on earlier turns, so the last message alone is a bad cache key. Here is the failure:

  1. User A asks, “What is the largest lake in North America?” The answer is Lake Superior. Then A asks, “What is the second largest?” The answer is Lake Huron. Both pairs are cached.
  2. Later, User B asks, “What is the largest stadium in North America?” and then, “What is the second largest?”
  3. A cache keyed only on the last prompt finds an exact match for “What is the second largest?” and returns “Lake Huron”. That is wrong for a conversation about stadiums.
Why the cache key must include chat context Two conversations end with the same prompt, "What is the second largest?". One is about lakes and one is about stadiums. A cache keyed on the last prompt only returns Lake Huron for both, which is wrong for the stadium conversation. A cache keyed on the prompts in the context window plus the last prompt keeps the conversations apart and returns Lake Huron only for the lake conversation. Conversation A (lakes) Conversation B (stadiums) 1. What is the largest lake in North America? 2. What is the second largest? 1. What is the largest stadium in North America? 2. What is the second largest? Key = last prompt only Both conversations match the same key and get "Lake Huron". Wrong for B. Key = context window + last prompt The keys differ, so A gets "Lake Huron" and B misses or gets its own stadium answer.
Cache the whole sequence of prompts in the context window, so follow-up questions only match the same line of conversation.

The fix is to embed the context window and the last prompt together and search on that. The cache then returns an answer only when a similar sequence of questions was asked before, which is the same context the LLM would have had.

Keeping the cache healthy

Like any cache, a semantic cache can grow without limit. Common controls:

  • Time to live (TTL): expire entries after a set time, so stale answers disappear.
  • Size limits: cap the number of items or total storage.
  • Hit-count pruning: count how often each entry is hit. Let rarely used entries expire, and extend the TTL of popular ones.
  • A good threshold: a lower threshold means more hits and fewer near-duplicate entries, at the cost of relevance.
  • Recency filters: you may want to keep old entries for analysis but only serve fresh ones. Filter the query on a lastUpdated field, or sort by it.

Cached prompts and answers also show how users actually use your app, so they are worth analyzing before you delete them.

Pitfalls

  • Stale answers: a cached answer to a question about changing facts can be wrong later. Use a short TTL for that content.
  • Personal or private data: an answer written for one user must not be served to another. Keep such requests out of a shared cache, or scope the cache per user or tenant.
  • Near misses: a threshold that is slightly too loose can return an answer to a subtly different question, such as one with a different number or name. Test with prompts that look alike but differ in meaning.
  • Not every request fits: answers that depend on live tools or data are poor candidates for caching.

Saved in this browser only. Not synced to your account.