DEV Community

Cover image for Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer
AS
AS

Posted on • Edited on

Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer

#kotlin #ai #llm #opensource

Exposes the hidden danger of similarity thresholds

An exact-match cache misses "how do I reverse a list in Python" when it has already answered "python list reverse". A semantic cache doesn't: it embeds the prompt, finds the closest one it has seen, and replays that answer instead of calling the model. Fewer API calls, lower latency, same answers.

Except for the part where it hands back the wrong one.

"Convert 100 USD to EUR"
"Convert 250 USD to EUR"      cosine similarity: ~0.99
Enter fullscreen mode Exit fullscreen mode

Every mainstream embedding model scores that pair around 0.99. No threshold separates it from a genuine paraphrase, because on the similarity axis the near miss sits closer than most paraphrases do. Raise the threshold and you lose real hits before you lose that one.

So a cache built on a threshold alone will tell someone that 250 dollars is 92 euros. Quickly, with no error, and nothing in the logs.

What Kmemo does about it

Kmemo treats that as the main event rather than an edge case. Similarity is only the first filter. Candidates that clear it get read as text by a chain of ten guards looking for concrete evidence that the two answers must differ: swapped numbers, mismatched units, different entities, different time references, negation, flipped antonyms, reversed comparisons, or a different kind of answer being asked for.

The defaults follow from the cost asymmetry. A wrong rejection costs one API call. A wrong acceptance costs a wrong answer. So the guards abstain rather than guess.

Quick start

Requires JDK 17+.

dependencies {
    implementation("io.github.nacode-studios:kmemo-core:1.0.0")
}
Enter fullscreen mode Exit fullscreen mode

kmemo-core declares kotlinx-coroutines-core as its only dependency. You bring the embedding source, which is any function from String to FloatArray. Kmemo ships none and depends on no provider SDK.

val cache = SemanticCache(
    embedder = Embedder { text -> openAi.embed(text) },
    store = InMemoryStore(maxEntries = 10_000, ttl = 1.hours),
)

val answer = cache.getOrPut(prompt) { llm.complete(it) }
Enter fullscreen mode Exit fullscreen mode

getOrPut embeds the prompt once and reuses the vector for both the lookup and the write. Concurrent callers asking the same thing get coalesced: the first one computes, the rest wait and are served its answer.

Every miss tells you why

A cache with a 4% hit rate is untunable unless you know what caused the misses, because the fix is opposite for a threshold miss and a guard rejection:

when (val result = cache.lookup(prompt)) {
    is CacheLookup.Hit  -> result.response
    is CacheLookup.Miss -> when (result.reason) {
        MissReason.BELOW_THRESHOLD   -> // traffic repeats less than you assumed, or the threshold is too tight
        MissReason.REJECTED_BY_GUARD -> // a guard found a concrete difference; result.detail says which
        else -> null
    }
}
Enter fullscreen mode Exit fullscreen mode

There's also cache.explain(prompt), a read-only companion that shows every candidate with every guard's verdict. It's what you reach for when a hit you expected didn't happen.

Scopes

Anything that changes what a correct answer looks like belongs in the scope: model, temperature, system prompt, tenant, language. Leave it out and the cache will serve one model's answer to another model's caller.

cache.getOrPut(prompt, scope = "gpt-4o|t=0.0|v3") { llm.complete(it) }
Enter fullscreen mode Exit fullscreen mode

Choosing how strict to be 

SemanticCache(embedder)                                    // MatchGuards.standard()
SemanticCache(embedder, guards = MatchGuards.strict())     // trades hit rate for margin
SemanticCache(embedder, guards = MatchGuards.none())       // the naive similarity-only baseline
Enter fullscreen mode Exit fullscreen mode

The guards work outside English too. Curated packs ship for Italian, Spanish, German and French, each measured against a localized near-miss corpus:

SemanticCache(embedder, guards = MatchGuards.standard(Locale.ITALIAN))
Enter fullscreen mode Exit fullscreen mode

What lexical guards can't see

About a third of near misses need world knowledge. Deworming a puppy is not the same as deworming an adult dog. The boiling point of ethanol is not the boiling point of methanol. No amount of token comparison catches those.

For that there's an optional Verifier, typically a cheap model call. It runs only on candidates that already cleared the threshold and every guard, so it costs nothing in the common case, and it fails closed: a timeout or an error rejects rather than serving something unconfirmed.

The numbers

The guards are judged against three labelled corpora with a blind validation split that no guard was tuned against, run as a CI regression gate on every build.

On the blind split, near misses are rejected 67% of the time and paraphrases are kept 88% of the time.

Neither number is 100%, and I would rather publish them than a marketing claim. The near misses that get through are mostly the world-knowledge cases the verifier covers. Reproduce them yourself:

./gradlew :kmemo-core:test --tests '*CorpusTest*'
Enter fullscreen mode Exit fullscreen mode

How the blind splits grow without getting contaminated is written up in docs/CORPUS.md.

Calibrate the threshold, don't copy it

ThresholdCalibrator measures the right threshold for your embedding model. The value you found in a blog post was tuned for somebody else's.

Stores, resilience, observability

Embedder and CacheStore are one-method seams, so you can start in memory and move to a vector database without touching the match logic. Redis (RediSearch KNN) and Postgres (pgvector) stores ship, plus an opt-in in-process HNSW store for when the exact scan stops scaling.

The embedder is a network call on every lookup, so Kmemo lets you own its failure:

val cache = SemanticCache(
    embedder = myEmbedder.retrying(maxAttempts = 4),
    embedFailurePolicy = EmbedFailurePolicy.FALL_BACK_TO_COMPUTE,
    negativeCacheSize = 10_000,
)
cache.warm(faqPairs.map { WarmEntry(it.question, it.answer) })
Enter fullscreen mode Exit fullscreen mode

For dashboards and logs, subscribe to the event stream instead of polling stats(). It costs nothing when unused:

val metrics = KmemoMetrics().also { it.bindTo(meterRegistry) }   // kmemo-micrometer
val cache = SemanticCache(embedder, listeners = listOf(metrics, Slf4jCacheListener()))
Enter fullscreen mode Exit fullscreen mode

Integrations

  • A Spring Boot starter that auto-configures a SemanticCache bean
  • A Spring AI caching Advisor for ChatClient
  • A LangChain4j caching ChatModel wrapper
  • A Ktor server plugin

examples/ is a runnable demo that needs no API key. It shows a guard catching a live near miss, with a docker-compose for the Redis store.

Try it

1.0 is stable under SemVer. If you have run a semantic cache in production and hit a false hit I haven't thought about, I want to hear about it. Open an issue with the pair that broke it.

If you'd want something like this to exist, a star is what makes it findable for
the next person looking.

Top comments (9)

Collapse
 
max_quimby profile image
Max Quimby •

The 100 USD vs 250 USD example is the whole argument in one line — the near-miss sits closer on the similarity axis than an honest paraphrase, so no threshold can separate them, and raising it just costs you real hits first. That's the part people miss when they bolt a semantic cache on and celebrate the hit rate: the failure isn't a lower hit rate, it's a confident wrong answer with nothing in the logs.

Grounding the defaults in cost asymmetry — one wrong rejection is one API call, one wrong acceptance is a wrong answer, so abstain — is exactly the right prior. It's the same reasoning that makes argument-validation gates worth their tedium: the expensive side is silent.

Two things I'm curious about. First, do the guards look at the candidate answer text as well as the two prompts? A swapped number in the stored response is the real tell, and it survives even when the prompts paraphrase cleanly. Second, how does this behave with stateful/multi-turn prompts where the "same" question depends on conversation history the embedding never sees — does context get folded into the key, or is that explicitly out of scope?

Collapse
 
tonytonycoder11 profile image
AS •

No on the first one, and you've put your finger on something. The guards only
ever see the two prompts. The response is sitting right there on the entry and
nothing in the match path touches it, not even the optional Verifier. So the
case you're describing is exactly the one they can't catch: if both prompts read
as clean paraphrases there's nothing for NumericGuard to compare, and the number
that actually matters is off in the stored answer where nobody's looking. I
think that interface needs to get wider.

The multi-turn one I've left alone on purpose, and I'm not convinced folding
context into the key is the right fix anyway. The only lever is scope, which is
an exact match rather than part of the similarity search, so you can put a
conversation id in there but then you've got a cache per conversation and you've
given up most of what you wanted. Embedding the history has the opposite
problem: every follow-up drifts further from its neighbours and the hit rate
quietly goes. What I'd do instead is rewrite the prompt into something
self-contained before it ever reaches the cache, so "and in euros?" becomes
"convert 100 USD to EUR", and cache that. You probably need that rewrite for
retrieval anyway.

Collapse
 
tonytonycoder11 profile image
AS •

Opened an issue for the first one, with you credited in it:
github.com/NaCode-Studios/Kmemo/is...

Writing it up surfaced the thing that actually blocks it, which hadn't occurred
to me until I started: the labelled corpus is prompt pairs only, so the 67/88
numbers are measured on a shape with no answer text in it at all. A
response-aware guard can't be reported honestly until the corpus schema grows an
answer field first. Slower than just widening the interface, but I'd rather not
ship a guard I can't put a number on.

Collapse
 
jugeni profile image
Mike Czerwinski •

The guard-then-verifier split maps cleanly onto a distinction that deserves its own name: guards catch what's derivable from the text alone, swapped numbers, negation, mismatched entities, no world model required. The verifier exists for exactly the cases where two strings are lexically indistinguishable and only differ by a fact about the world, ethanol versus methanol, puppy versus adult dog. That's a real fault line, syntactic difference versus semantic difference, and it explains why guards can be cheap, pure text comparison, while the verifier can't, it needs to know something, not just compare something.

Worth being precise about what the 67%/88% numbers include, since the post reads ambiguously on this point: are those guard-only rejection rates, or guard-plus-verifier? If the world-knowledge third of near misses is exactly the residual the verifier exists to catch, the honest number to publish alongside 67% is the verifier's own catch rate on that specific third, since that's the piece actually closing the gap the guards structurally can't. Otherwise 67% quietly reads as the whole system's performance when it might be describing only the free layer, with the paid layer's contribution unmeasured in the same table.

The negative-cache and embed-failure-policy design is the part I'd bet earns its keep in production faster than the guards do, mostly because embedding-call failures are boring and constant where near-misses are rare and dramatic. Curious whether you're tracking which of the two actually drives more cache misses in real traffic once this ships somewhere.

Collapse
 
tonytonycoder11 profile image
AS •

Guard-only, and you're right that the post doesn't say so. The corpus test runs
MatchGuards.standard() and nothing else, so the verifier isn't in those numbers
at all. Which makes the README line about it covering the rest an unmeasured
claim sitting next to two measured ones, and that's the wrong thing for this
project of all things to be sloppy about.

The set you'd need is small and already sitting there: the held-out near misses
the guards don't reject, about 28 of 86. That's exactly the population the
verifier claims, so running one against those 28 gives the companion number you
asked for on the same split. Opened it as
github.com/NaCode-Studios/Kmemo/is... and credited you.

On the third point, I think you're right and the telemetry half exists:
MissReason already separates REJECTED_BY_GUARD from REJECTED_BY_VERIFIER. Embed
failures aren't in there though, because FALL_BACK_TO_COMPUTE bypasses the cache
rather than producing a miss, so nobody's counting the boring failure. Nothing's
in real traffic yet, but that's the counter I'd want on day one.

Collapse
 
jugeni profile image
Mike Czerwinski •

Guard-only numbers next to an unmeasured verifier claim is exactly the gap worth catching before it ships anywhere serious, and running the verifier against precisely the 28 held-out near misses the guards already let through is the most direct test, since that's not a fresh sample, it's the exact population the verifier exists to cover. Whatever number comes back there is the real claim the README line was gesturing at.

The embed-failure gap is the more useful find of the two, honestly, because it's invisible in a way the guard/verifier split isn't. A rejected candidate produces a MissReason and gets counted somewhere. A fallback-to-compute produces nothing, a cache miss that never registers as a miss, so the failure mode with the least drama, an embedding call timing out, is the one with no telemetry pointing at it at all. Worth a counter before day one specifically because it's boring: nobody's going to notice it's missing until a postmortem needs it and it isn't there, same shape as most of what fails quietly in this thread all week.

Thread Thread
 
tonytonycoder11 profile image
AS •

Thanks, this is the kind of comment that actually moves something.

And it's worse than I described: there's no event either. CacheEvent only has
Hit, Miss, Write and Eviction, so a fallback never even reaches the listener
seam. Opened it as github.com/NaCode-Studios/Kmemo/is... and
credited you there.

Collapse
 
dropzilla_site_bee900de05 profile image
dp •

Nice project! I really like the idea of using semantic guards instead of relying only on similarity scores. The explanation about the 100 USD vs. 250 USD example makes the problem very clear. I'll definitely test this approach in one of my AI projects. We also share free developer resources and tutorials on Codecan.net, so this was an interesting read. Thanks for making it open source!

Collapse
 
tonytonycoder11 profile image
AS •

Thanks, glad that example landed. It's the one that made me build the thing in
the first place. If you do try it on one of your projects I'd love to hear how
it goes, especially what your near misses look like: the guards are tuned on a
corpus that's mostly numbers, units and entities, and I have no idea yet how
well that carries into other domains. And if you like where it's heading, a star
genuinely helps people stumble on it.