Title:Leveraging Approximate Caching for Faster Retrieval-Augmented Generation

Abstract:Retrieval-augmented generation (RAG) improves the reliability of large language model (LLM) answers by integrating external knowledge. However, RAG increases the end-to-end inference time since looking for relevant documents from large vector databases is computationally expensive. To address this, we introduce Proximity, an approximate key-value cache that optimizes the RAG workflow by leveraging similarities in user queries. Instead of treating each query independently, Proximity reuses previously retrieved documents when similar queries appear, substantially reducing reliance on expensive vector database lookups. To scale efficiently, Proximity employs a locality-sensitive hashing (LSH) scheme that enables fast cache lookups while preserving retrieval accuracy. We evaluate Proximity using the MMLU and MedRAG question answering benchmarks. Our experiments demonstrate that Proximity with our LSH scheme and a realistically skewed MedRAG workload reduces database calls by 78.9% while maintaining database recall and test accuracy. We experiment with different similarity tolerances and cache capacities, and show that the time spent within the Proximity cache remains low and constant (4.8 microseconds) even as the cache grows substantially in size. Our work highlights that approximate caching is a viable and effective strategy for optimizing RAG-based systems.