A cached answer can be fast and still be wrong for the person asking. In a multi-tenant LLM application, two customers may use almost identical wording while referring to different contracts, prices or policies.
Semantic caching can reuse a stored response when a new request is sufficiently similar. Safe reuse also depends on authorization, source freshness and application state. These seven practices help teams decide when a cache hit is eligible to become an answer.
TL;DR
- Treat similarity as a candidate match, not permission to return an answer.
- Partition reusable content by tenant and effective access scope.
- Invalidate answers when their evidence, policies or configuration change.
- Bypass response reuse for live transactions and actions that must execute again.
- Measure incorrect reuse alongside hit rate and cost.
Understand the cache types
| Cache type | What is reused | Typical concern |
|---|---|---|
| Exact response cache | Answer to an identical eligible request | Key omits changing context |
| Semantic response cache | Answer to a similar eligible request | Similar wording hides a different meaning |
| Prefix cache | Computed model states for shared input | Compatibility and isolation |
| Retrieval cache | Search results or document chunks | Stale content or permissions |
The Redis semantic cache documentation describes reuse based on similarity. That mechanism still needs application rules that determine whether the stored answer is valid for the current request.
1 Define which requests may use cached answers
Begin with stable information such as a public product explanation or an approved policy summary. Classify balance enquiries, availability checks and other live data requests separately. A prior answer about an account balance should not substitute for a fresh authorized lookup.
Pros: Eligibility rules prevent unnecessary generation for predictable content. Cons: Too many bypasses reduce the hit rate. Keep correctness as the first constraint, then look for repeated eligible traffic worth caching.
2 Isolate tenants before similarity search
Use tenant identity from the authenticated server context. Apply the tenant boundary during cache lookup, rather than retrieving a global nearest neighbor and trusting a later check.
Microsoft's multitenancy guidance specifically discusses partitioning semantic cache entries by tenant. If information is intentionally shared across tenants, give it an explicit public or shared classification instead of silently falling back to a global partition.
Pros: Clear isolation reduces accidental cross-customer reuse. Cons: Separate partitions may reduce cache efficiency and require additional lifecycle management.
3 Include the effective permission scope
Tenant membership alone is insufficient when users have different access to matters, departments or documents. Associate each entry with the access scope and evidence needed to reuse it. Reauthorize at lookup, particularly after a role change or revoked access.
A safe cache design can include user or role scope, policy version and source identifiers as eligibility metadata. Do not allow a client supplied field to elevate that scope.
Pros: Answers remain aligned with the reader's permissions. Cons: Scope changes create invalidation and testing work.
4 Version the answer context
Store the model and prompt versions, relevant source versions, locale and application configuration with the response. Set a time to live based on the volatility of the underlying information, and invalidate affected entries when evidence changes.
Microsoft's Redis semantic cache tutorial includes partitioning considerations such as tenant, user, model and prompt version. A time limit is useful, but it cannot guarantee that a source did not change earlier.
Pros: Versioned context makes stale answers easier to identify. Cons: More precise invalidation requires an inventory of dependencies and reliable change events.
5 Calibrate similarity on near misses
Test pairs that look similar but need different answers. Examples include a refund policy in two countries, a price before and after an effective date, or two customers with different service levels.
Use separate validation data to choose a threshold and any additional verification step. A universal similarity threshold cannot be assumed to work across embedding models and workloads. The Redis semantic cache guidance provides implementation context for this pattern.
Pros: Near-miss tests expose dangerous false hits. Cons: Stricter matching can lower reuse. Review false misses too, but do not optimize hit rate by accepting incorrect answers.
6 Separate cached explanations from actions
An answer that explains how to reschedule an appointment can be reusable. A message saying the appointment has been rescheduled cannot replace executing and verifying the authorized action.
Keep transaction identifiers, approvals and idempotency checks in the live workflow. If the system reuses explanatory text around an action, populate current facts only after the backend operation succeeds. Record cache provenance without retaining unnecessary personal information.
Pros: Reuse does not silently skip business operations. Cons: Application code must distinguish generated explanation from action execution and confirmation.
7 Monitor incorrect reuse and total cost
Track eligible requests, hits, misses, bypass reasons, stale answers and incorrect reuse discovered through review. Include embedding requests, vector lookup, cache storage, invalidation work and fallback generation in cost calculations.
Compare against an uncached baseline at the same workload and quality level. A high hit rate can be misleading if cache entries are often wrong, or if inexpensive requests dominate the measurement.
Pros: The team can judge whether caching creates useful savings. Cons: Evaluation and invalidation have ongoing operating costs.
A tenant isolation example
Imagine two companies asking, “How long do employees have to claim travel expenses?” Company A uses a 30-day policy; Company B uses a 60-day policy. These values are illustrative.
The application authenticates the user, resolves the company and permitted policy version, then searches only eligible cache entries. A match from Company A must never satisfy Company B's request. If Company B changes its policy, the affected cached answer is invalidated before reuse. The example shows why semantic similarity alone is not enough.
Minimum acceptance tests
| Test | Expected behavior |
|---|---|
| Same question from another tenant | No private cross-tenant answer |
| Access revoked after cache population | Previously accessible answer is withheld |
| Source policy updated | Prior version is invalidated or rejected |
| Similar question with a different date | Answer uses the correct effective policy |
| Request performs a booking | Live operation runs and is verified |
| Cache service unavailable | Controlled fallback respects normal access rules |
Build these checks into the same evaluation process used for generation. For related controls, read permission-aware retrieval and the LLM gateway comparison.
Frequently asked questions
Should every answer be cached
No. Start with stable and repeated information where eligibility can be defined. Bypass requests involving live data or sensitive context when safe reuse cannot be established.
Does encryption solve cross-tenant reuse
No. Encryption protects data in transit or at rest, while authorization determines who can receive an answer. Both need deliberate implementation.
Is a short time to live enough
No. A permission or policy can change before the cache expires. Use event driven invalidation or eligibility checks where freshness is important.
What is the best success metric
Use cost and latency at an acceptable answer quality level. Incorrect reuse, stale responses and access failures matter more than hit rate alone.
Start with one eligible workload
Pilot one stable information flow, test near misses and validate isolation before expanding reuse. Explore LuMay AI engineering services for help integrating caching with retrieval, identity and evaluation.
Sources and further reading
- Multitenancy and Foundry Models - Azure Architecture Center | Microsoft Learn
- Tutorial: Use Azure Managed Redis as a semantic cache - Azure Managed Redis | Microsoft Learn
- Redis semantic cache | Docs
Sources and linked destinations checked on 1 November 2026. Product capabilities and documentation can change.
