A semantic cache stores past prompts and the answers an LLM gave, then reuses those answers when a new prompt means roughly the same thing. It matches by meaning, not by exact text, using a vector similarity search.
LLM calls are often the slowest and most expensive step in an AI app. Every call is billed and timed by tokens, and the work of inference takes longer as the input grows. A cache lets you skip the call when you already have a good answer.
Why LLM calls are worth caching
A short prompt is cheap. Real requests are not short:
- LLMs do not remember earlier turns. To hold a conversation, the app sends part of the chat history with each new prompt. This sliding context window adds tokens and latency. See context engineering.
- In retrieval-augmented generation (RAG), documents fetched from a database are added to the prompt too. Payloads of thousands of tokens and several seconds of waiting are common.
At high volume, many users ask the same thing in different words. A normal cache cannot help with that, because “How do I reset my password?” and “I forgot my password, how do I change it?” are different strings.
How a semantic cache works
A traditional cache looks up a key by exact string match. A semantic cache turns the lookup into a vector query:
- Convert the incoming prompt into an embedding.
- Search the stored prompt embeddings (the cache keys) for the closest ones.
- If a match is close enough, return its stored completion. This is a cache hit.
- If not, call the LLM, return its answer, and store the new prompt embedding and completion. This is a cache miss.
Because the lookup is a query, not a single-key fetch, it can also return several close matches, ranked from most to least similar. An app can then pick the best one, or offer a few. This can reduce how many answers the LLM has to generate.
The similarity threshold
Every vector query returns a similarity score for each result, from 0 (nothing alike) to 1 (an exact match). Scores like cosine similarity are one common choice; see vector distance metrics for others. The cache uses a threshold on this score to decide what counts as a hit.
- Threshold too high: only near-identical prompts hit. Most requests miss, and the cache fills with many answers to similar questions.
- Threshold too low: loosely related prompts hit. Users get answers that do not match what they asked.
There is no universal right value. It depends on your embedding model and your domain, so expect to tune it using real prompts and check the answers it returns.
Include the conversation, not just the last prompt
Chat prompts depend on earlier turns, so the last message alone is a bad cache key. Here is the failure:
- User A asks, “What is the largest lake in North America?” The answer is Lake Superior. Then A asks, “What is the second largest?” The answer is Lake Huron. Both pairs are cached.
- Later, User B asks, “What is the largest stadium in North America?” and then, “What is the second largest?”
- A cache keyed only on the last prompt finds an exact match for “What is the second largest?” and returns “Lake Huron”. That is wrong for a conversation about stadiums.
The fix is to embed the context window and the last prompt together and search on that. The cache then returns an answer only when a similar sequence of questions was asked before, which is the same context the LLM would have had.
Keeping the cache healthy
Like any cache, a semantic cache can grow without limit. Common controls:
- Time to live (TTL): expire entries after a set time, so stale answers disappear.
- Size limits: cap the number of items or total storage.
- Hit-count pruning: count how often each entry is hit. Let rarely used entries expire, and extend the TTL of popular ones.
- A good threshold: a lower threshold means more hits and fewer near-duplicate entries, at the cost of relevance.
- Recency filters: you may want to keep old entries for analysis but only serve fresh ones. Filter the query on a
lastUpdatedfield, or sort by it.
Cached prompts and answers also show how users actually use your app, so they are worth analyzing before you delete them.
Pitfalls
- Stale answers: a cached answer to a question about changing facts can be wrong later. Use a short TTL for that content.
- Personal or private data: an answer written for one user must not be served to another. Keep such requests out of a shared cache, or scope the cache per user or tenant.
- Near misses: a threshold that is slightly too loose can return an answer to a subtly different question, such as one with a different number or name. Test with prompts that look alike but differ in meaning.
- Not every request fits: answers that depend on live tools or data are poor candidates for caching.