Semantic Caching for Self-Hosted LLMs: What Actually Happens When You Build One

The last post here covered why Sheria needed an agentic RAG pipeline and a citation graph instead of plain vector search. This one is about a smaller, more mundane piece of the same system: the semantic cache sitting in front of it. It's less architecturally flashy than a citation graph, but it's the part that actually keeps the platform cheap to run — and writing about it honestly meant finding a few things I'd gotten wrong along the way, including inside my own commit history.

Why cache LLM responses at all

Here's the blunt version of self-hosting your own inference stack: there's no elastic API scaling to hide behind. Sheria's inference runs on a local 3-node Ollama GPU cluster serving qwen3:8b — not a cloud API with someone else's capacity behind it. Every query costs real, fixed compute against hardware I own. There's no billing dashboard creeping upward, but there's also no scaling your way out of a slow model when traffic spikes.

Legal research questions repeat constantly in practice, just phrased differently each time:

• "What is the test for adverse possession under Kenyan law?"
• "How do I claim land via adverse possession in the High Court?"
• "Adverse possession, 12-year rule, Limitation of Actions Act — requirements?"

An exact-match cache — a plain key lookup, say — misses all three of these; the strings don't match even though the underlying question is identical. Saving GPU cycles without hurting the response quality means matching on meaning instead of text, which is what a semantic cache buys you.

The design: Qdrant as the cache, not just the retrieval store

The cache lives in its own Qdrant collection, semantic_cache, entirely separate from kenya_law_reports, the collection holding the case-law embeddings used for retrieval. An incoming query gets embedded with Ollama's nomic-embed-text (768 dimensions), then searched against semantic_cache by cosine similarity with a threshold of 0.95. Clear that bar and the cached answer returns directly, skipping the LangGraph planner/retriever/responder pipeline — and the generation pass — entirely.

[ Incoming Legal Query ]
        │
        ▼
[ Embed Query ]  (nomic-embed-text, 768-dim, via Ollama)
        │
        ▼
┌──────────────────────────────────────────────┐
│ Qdrant: `semantic_cache`                      │
│ cosine similarity >= 0.95, age <= 30 days     │
└──────────────────────┬───────────────────────┘
                        │
          ┌─────────────┴─────────────┐
          ▼                           ▼
    [ Cache HIT ]                [ Cache MISS ]
  Return cached answer      Reuse query_vector to search
  (skip LangGraph + LLM)    `kenya_law_reports`, run the
                            agent, then cache the answer

It's worth being explicit about why this is Qdrant alone rather than Qdrant plus Redis, since Redis does exist elsewhere in the stack — it backs JWT token blacklisting and health checks. A semantic cache is fundamentally a vector-similarity lookup, and Redis without a search layer only does exact-key matching, which is precisely the thing that doesn't work here. Adding it to this path wouldn't add capability, just another system to operate.

Soft TTL instead of real expiry

Vector stores aren't key-value caches, and Qdrant points don't expire on their own — there's no native TTL. The workaround is a soft TTL enforced at query time. Every cached point carries a created_at timestamp in its payload:

{
  "query": "What is adverse possession in Kenya?",
  "answer": "Under the Limitation of Actions Act (Cap 22)...",
  "created_at": 1741558640
}

and the lookup adds a range filter alongside the similarity threshold — anything with a created_at older than the configured max age (30 days by default) is excluded from the results entirely, which behaves exactly like a cache miss from the caller's side. It's not free expiry the way a key-value store's EXPIRE would give you — the storage stays occupied until something eventually cleans up old points — but application-side, it reliably keeps stale answers from reaching users.

The triple-embedding fix

The most useful part of building this kind of infrastructure yourself is catching the silent inefficiencies that creep in early. The original version of this pipeline re-embedded the same query text up to three times per request:

1. Embed the query to check semantic_cache.
2. On a miss, embed the same query again to search kenya_law_reports for retrieval.
3. Embed it a third time when writing the new answer back into semantic_cache.

Generating a 768-dim vector via Ollama isn't free — tens of milliseconds each time — and paying for it three times over on every cache miss worked against the whole point of caching. The fix is to compute the vector once and pass it forward instead of recomputing it at each step. get_cached_response embeds the query and returns (answer, vector) on both a hit and a miss, so the caller always has the embedding on hand. On a miss, that vector goes into the agent's state as query_vector, and the retriever node checks for it before doing anything: if it's already populated, retrieval skips its own embed call and searches kenya_law_reports directly with the vector it was handed. The same vector then gets passed into set_cached_response, which only embeds the query itself if none was supplied.

# Simplified from services/api/app/routes/chat.py
cached_answer, query_vector = await cache.get_cached_response(message)

if cached_answer:
    return cached_answer  # cache hit — pipeline stops here

state = AgentState(current_query=message, query_vector=query_vector or [])
# retriever skips its own embed call when query_vector is already populated
...
background_tasks.add_task(
    cache.set_cached_response, message, final_answer, vector=query_vector
)

One embed call now feeds all three steps — cache lookup, retrieval, and the write back into the cache — instead of three separate calls for the same piece of text.

What the metrics actually say

There's a Prometheus counter tracking this in production:

from prometheus_client import Counter

CACHE_HITS = Counter(
    "rag_semantic_cache_total",
    "Semantic cache hit and miss counts",
    ["result"],  # "hit" | "miss"
)

CACHE_HITS.labels(result="hit" if cached_answer else "miss").inc()

I'll be honest that I don't yet have enough production traffic to quote a hit rate that would mean anything — the point of putting the counter in now is that when usage grows, I'll be able to answer that question with a number instead of a guess.

The open problem: staleness isn't really solved

There's a real gap I'm not going to pretend is handled: nothing invalidates a cached answer when the underlying case law it was based on changes. If a case gets overruled, the citation graph from the last post would know it — the semantic cache wouldn't. A cached answer sitting inside its 30-day window has no way to hear about that. Tying cache invalidation to citation-graph updates (so an OVERRULES edge can evict anything cached from the case it affects) is the obvious next piece, and it isn't built yet.

Where this leaves things

None of this is as architecturally interesting as an agentic pipeline or a citation graph, but it's the part of Sheria that quietly keeps repeated queries cheap on hardware I own, rather than hardware I rent. The next post in this series will probably be about closing that staleness gap — making the cache aware of the same case relationships the citation graph already knows about.