-
CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction
Authors:
Harsh Maheshwari,
Srikanth Tenneti,
Alwarappan Nakkiran
Abstract:
Retrieval Augmented Generation (RAG) has emerged as a powerful application of Large Language Models (LLMs), revolutionizing information search and consumption. RAG systems combine traditional search capabilities with LLMs to generate comprehensive answers to user queries, ideally with accurate citations. However, in our experience of developing a RAG product, LLMs often struggle with source attrib…
▽ More
Retrieval Augmented Generation (RAG) has emerged as a powerful application of Large Language Models (LLMs), revolutionizing information search and consumption. RAG systems combine traditional search capabilities with LLMs to generate comprehensive answers to user queries, ideally with accurate citations. However, in our experience of developing a RAG product, LLMs often struggle with source attribution, aligning with other industry studies reporting citation accuracy rates of only about 74% for popular generative search engines. To address this, we present efficient post-processing algorithms to improve citation accuracy in LLM-generated responses, with minimal impact on latency and cost. Our approaches cross-check generated citations against retrieved articles using methods including keyword + semantic matching, fine tuned model with BERTScore, and a lightweight LLM-based technique. Our experimental results demonstrate a relative improvement of 15.46% in the overall accuracy metrics of our RAG system. This significant enhancement potentially enables a shift from our current larger language model to a relatively smaller model that is approximately 12x more cost-effective and 3x faster in inference time, while maintaining comparable performance. This research contributes to enhancing the reliability and trustworthiness of AI-generated content in information retrieval and summarization tasks which is critical to gain customer trust especially in commercial products.
△ Less
Submitted 11 June, 2025; v1 submitted 22 April, 2025;
originally announced April 2025.
-
Contradiction Detection in RAG Systems: Evaluating LLMs as Context Validators for Improved Information Consistency
Authors:
Vignesh Gokul,
Srikanth Tenneti,
Alwarappan Nakkiran
Abstract:
Retrieval Augmented Generation (RAG) systems have emerged as a powerful method for enhancing large language models (LLMs) with up-to-date information. However, the retrieval step in RAG can sometimes surface documents containing contradictory information, particularly in rapidly evolving domains such as news. These contradictions can significantly impact the performance of LLMs, leading to inconsi…
▽ More
Retrieval Augmented Generation (RAG) systems have emerged as a powerful method for enhancing large language models (LLMs) with up-to-date information. However, the retrieval step in RAG can sometimes surface documents containing contradictory information, particularly in rapidly evolving domains such as news. These contradictions can significantly impact the performance of LLMs, leading to inconsistent or erroneous outputs. This study addresses this critical challenge in two ways. First, we present a novel data generation framework to simulate different types of contradictions that may occur in the retrieval stage of a RAG system. Second, we evaluate the robustness of different LLMs in performing as context validators, assessing their ability to detect contradictory information within retrieved document sets. Our experimental results reveal that context validation remains a challenging task even for state-of-the-art LLMs, with performance varying significantly across different types of contradictions. While larger models generally perform better at contradiction detection, the effectiveness of different prompting strategies varies across tasks and model architectures. We find that chain-of-thought prompting shows notable improvements for some models but may hinder performance in others, highlighting the complexity of the task and the need for more robust approaches to context validation in RAG systems.
△ Less
Submitted 31 March, 2025;
originally announced April 2025.
-
Robust Audio Anomaly Detection
Authors:
Wo Jae Lee,
Karim Helwani,
Arvindh Krishnaswamy,
Srikanth Tenneti
Abstract:
We propose an outlier robust multivariate time series model which can be used for detecting previously unseen anomalous sounds based on noisy training data. The presented approach doesn't assume the presence of labeled anomalies in the training dataset and uses a novel deep neural network architecture to learn the temporal dynamics of the multivariate time series at multiple resolutions while bein…
▽ More
We propose an outlier robust multivariate time series model which can be used for detecting previously unseen anomalous sounds based on noisy training data. The presented approach doesn't assume the presence of labeled anomalies in the training dataset and uses a novel deep neural network architecture to learn the temporal dynamics of the multivariate time series at multiple resolutions while being robust to contaminations in the training dataset. The temporal dynamics are modeled using recurrent layers augmented with attention mechanism. These recurrent layers are built on top of convolutional layers allowing the network to extract features at multiple resolutions. The output of the network is an outlier robust probability density function modeling the conditional probability of future samples given the time series history. State-of-the-art approaches using other multiresolution architectures are contrasted with our proposed approach. We validate our solution using publicly available machine sound datasets. We demonstrate the effectiveness of our approach in anomaly detection by comparing against several state-of-the-art models.
△ Less
Submitted 3 February, 2022;
originally announced February 2022.
-
Transient Dynamics of Epidemic Spreading and Its Mitigation on Large Networks
Authors:
Chul-Ho Lee,
Srinivas Tenneti,
Do Young Eun
Abstract:
In this paper, we aim to understand the transient dynamics of a susceptible-infected (SI) epidemic spreading process on a large network. The SI model has been largely overlooked in the literature, while it is naturally a better fit for modeling the malware propagation in early times when patches/vaccines are not available, or over a wider range of timescales when massive patching is practically in…
▽ More
In this paper, we aim to understand the transient dynamics of a susceptible-infected (SI) epidemic spreading process on a large network. The SI model has been largely overlooked in the literature, while it is naturally a better fit for modeling the malware propagation in early times when patches/vaccines are not available, or over a wider range of timescales when massive patching is practically infeasible. Nonetheless, its analysis is simply non-trivial, as its important dynamics are all transient and the usual stability/steady-state analysis no longer applies. To this end, we develop a theoretical framework that allows us to obtain an accurate closed-form approximate solution to the original SI dynamics on any arbitrary network, which captures the temporal dynamics over all time and is tighter than the existing approximation, and also to provide a new interpretation via reliability theory. As its applications, we further develop vaccination policies with or without knowledge of already-infected nodes, to mitigate the future epidemic spreading to the extent possible, and demonstrate their effectiveness through numerical simulations.
△ Less
Submitted 20 May, 2019; v1 submitted 1 March, 2019;
originally announced March 2019.
