Semantic Cache

The semantic cache speeds up repeated or near-identical SEMANTIC_MATCH queries by serving results from a local cache instead of re-embedding the query text and re-running HNSW traversal. It is available on any table with an embedding column and requires no application changes.

What it does

When a SEMANTIC_MATCH query arrives, GalaxDB embeds the query text and checks the cache for a stored result whose query embedding is within the configured cosine similarity threshold of the incoming query. If a match is found within the TTL window, the cached result set is returned directly - the sidecar is not called and HNSW traversal is skipped entirely.

The cache key includes the query embedding, the search threshold, the LIMIT value, and the model version. A different model, threshold, or limit always produces a cache miss. This prevents stale or mismatched results from being served.

Note

A cache hit returns exactly the same result set as a cache miss would. The cache is transparent to the application - results are never approximate or stale due to caching alone.

Setup

Configure a cache on a table with a single DDL statement:

SQL

-- Queries within cosine similarity 0.97 of a cached query are served from cache.
-- Cache entries expire after 5 minutes (300 seconds).
CREATE SEMANTIC CACHE FOR TABLE docs SIMILARITY 0.97 TTL 300;

Parameters:

SIMILARITY - cosine similarity threshold (0.0–1.0). Queries whose embedding is within this distance of a cached query embedding are treated as cache hits. Higher values mean stricter matching; lower values serve more queries from cache but risk returning slightly mismatched results.
TTL - time-to-live in seconds. Cache entries older than this are expired and a fresh search is run.

SQL

-- Remove the semantic cache for a table
DROP SEMANTIC CACHE FOR TABLE docs;

Tip

A SIMILARITY of 0.97 is a good starting point for most RAG workloads. Users asking the same question in slightly different words typically produce embeddings with cosine similarity above 0.95.

Invalidation

The semantic cache is invalidated automatically whenever the underlying table changes. Any INSERT, UPDATE, or DELETE to the table flushes the entire cache for that table. This ensures cached results never reflect rows that have since been modified or deleted.

Additional invalidation triggers:

Model version change - a different model produces different embedding spaces, making all cached keys invalid.
DROP SEMANTIC CACHE FOR TABLE t - explicitly removes the cache and all entries.

Note

Because the cache is flushed on any write, it works best for tables with a high read-to-write ratio - typical for document retrieval, FAQ lookups, and knowledge-base queries in production RAG pipelines.

Monitoring

The cache exposes a dedicated counter in the /metrics endpoint:

bash

curl http://localhost:9090/metrics | grep semantic_cache
# galaxdb_semantic_cache_hits_total 94

galaxdb_semantic_cache_hits_total increments once per cache hit. Use it to measure cache effectiveness, estimate sidecar load reduction, and implement usage-based billing for cache acceleration.

Example Prometheus query to compute cache hit rate over 5-minute windows:

# Cache hits per minute
rate(galaxdb_semantic_cache_hits_total[5m]) * 60

# Hit rate (cache hits / total vector queries)
rate(galaxdb_semantic_cache_hits_total[5m])
  / rate(galaxdb_vector_ops_total[5m])

RAG & Vector Indexing

Serializable Isolation