-
Towards Understanding Bias in Synthetic Data for Evaluation
Authors:
Hossein A. Rahmani,
Varsha Ramineni,
Nick Craswell,
Bhaskar Mitra,
Emine Yilmaz
Abstract:
Test collections are crucial for evaluating Information Retrieval (IR) systems. Creating a diverse set of user queries for these collections can be challenging, and obtaining relevance judgments, which indicate how well retrieved documents match a query, is often costly and resource-intensive. Recently, generating synthetic datasets using Large Language Models (LLMs) has gained attention in variou…
▽ More
Test collections are crucial for evaluating Information Retrieval (IR) systems. Creating a diverse set of user queries for these collections can be challenging, and obtaining relevance judgments, which indicate how well retrieved documents match a query, is often costly and resource-intensive. Recently, generating synthetic datasets using Large Language Models (LLMs) has gained attention in various applications. While previous work has used LLMs to generate synthetic queries or documents to improve ranking models, using LLMs to create synthetic test collections is still relatively unexplored. Previous work~\cite{rahmani2024synthetic} showed that synthetic test collections have the potential to be used for system evaluation, however, more analysis is needed to validate this claim. In this paper, we thoroughly investigate the reliability of synthetic test collections constructed using LLMs, where LLMs are used to generate synthetic queries, labels, or both. In particular, we examine the potential biases that might occur when such test collections are used for evaluation. We first empirically show the presence of such bias in evaluation results and analyse the effects it might have on system evaluation. We further validate the presence of such bias using a linear mixed-effects model. Our analysis shows that while the effect of bias present in evaluation results obtained using synthetic test collections could be significant, for e.g.~computing absolute system performance, its effect may not be as significant in comparing relative system performance. Codes and data are available at: https://github.com/rahmanidashti/BiasSyntheticData.
△ Less
Submitted 11 June, 2025;
originally announced June 2025.
-
Advancing Digital Accessibility In Digital Pharmacy, Healthcare, And Wearable Devices: Inclusive Solutions for Enhanced Patient Engagement
Authors:
Vishnu Ramineni,
Balaji Shesharao Ingole,
Nikhil Kumar Pulipeta,
Balakrishna Pothineni,
Aditya Gupta
Abstract:
Modern healthcare facilities demand digital accessibility to guarantee equal access to telemedicine platforms, online pharmacy services, and health monitoring devices that can be worn or are handy. With the rising call for the implementation of robust digital healthcare solutions, people with disabilities encounter impediments in their endeavor of managing and getting accustomed to these modern te…
▽ More
Modern healthcare facilities demand digital accessibility to guarantee equal access to telemedicine platforms, online pharmacy services, and health monitoring devices that can be worn or are handy. With the rising call for the implementation of robust digital healthcare solutions, people with disabilities encounter impediments in their endeavor of managing and getting accustomed to these modern technologies owing to insufficient accessibility features. The paper highlights the role of comprehensive solutions for enhanced patient engagement and usability, particularly, in digital pharmacy, healthcare, and wearable devices. Besides, it elucidates the key obstructions faced by users experiencing auditory, visual, cognitive, and motor impairments. Through a kind consideration of present accessibility guidelines, practices, and emerging technologies, the paper provides a holistic overview by offering innovative solutions, accentuating the vitality of compliance with Web Content Accessibility Guidelines (WCAG), Americans with Disabilities Act (ADA), and other regulatory structures to foster easy access to digital healthcare services. Moreover, there is due focus on using AI-driven tools, speech-activated interfaces, and tactile feedback in wearable health devices to assist persons with disabilities. The outcome of the research explicates the necessity of prioritizing accessibility for individuals with disabilities and cultivating a culture where healthcare providers, policymakers, and officials build a patient-centered digital healthcare ecosystem that is all-encompassing in nature.
△ Less
Submitted 29 May, 2025;
originally announced May 2025.
-
Advancing Digital Accessibility: Integrating AR/VR and Health Tech for Inclusive Healthcare Solutions
Authors:
Vishnu Ramineni,
Shivareddy Devarapalli,
Balakrishna Pothineni,
Prema Kumar Veerapaneni,
Aditya Gupta,
Pankaj Gupta
Abstract:
Modern healthcare domain incorporates a feature of digital accessibility to ensure seamless flow of online services for the patients. However, this feature of digital accessibility poses a challenge particularly for patients with disabilities. To eradicate this issue and provide immersive and user-friendly experiences, evolving technologies like Augmented Reality (AR) and Virtual Reality (VR) are…
▽ More
Modern healthcare domain incorporates a feature of digital accessibility to ensure seamless flow of online services for the patients. However, this feature of digital accessibility poses a challenge particularly for patients with disabilities. To eradicate this issue and provide immersive and user-friendly experiences, evolving technologies like Augmented Reality (AR) and Virtual Reality (VR) are integrated in medical applications to enhance accessibility. The present research paper aims to study inclusivity and accessibility features of AR/VR in revolutionizing healthcare practices especially in domains like telemedicine, patient education, assistive tools, and rehabilitation for persons with disabilities. The current trends of advancements and case studies are also analyzed to measure the efficacy of AR/VR in healthcare. Moreover, the paper entails a detailed analysis of the challenges of its adoption particularly technical limitations, implementation costs, and regulatory aspects. Finally, the paper concludes with recommendations for integrating AR/VR to foster a more equitable and inclusive healthcare system and provide individuals with auditory, visual, and motor impairments with digital healthcare solutions.
△ Less
Submitted 29 May, 2025;
originally announced May 2025.
-
Bridging the Gap: Enhancing Digital Accessibility for Medicaid Populations in Telehealth Adoption
Authors:
Vishnu Ramineni,
Aditya Gupta,
Balakrishna Pothineni,
Isan Sahoo,
Shivareddy Devarapalli,
Balaji Shesharao Ingole
Abstract:
The swift evolution of telehealth has revolutionized how medical professionals deliver healthcare services and boost convenience and accessibility. Yet, the Medicaid population encounters several impediments in utilizing facilities especially owing to poor internet connectivity, less awareness about digital platforms, and a shortage of assistive technologies. The paper aims to explicate key factor…
▽ More
The swift evolution of telehealth has revolutionized how medical professionals deliver healthcare services and boost convenience and accessibility. Yet, the Medicaid population encounters several impediments in utilizing facilities especially owing to poor internet connectivity, less awareness about digital platforms, and a shortage of assistive technologies. The paper aims to explicate key factors behind digital accessibility for Medicaid populations and expounds robust solutions to eradicate these challenges. Through inclusive design ideas, AI-assisted technologies, and all-encompassing policies by the concerned authorities, healthcare professionals can enhance usability and efficacy and thus better serve the needy. This revolution not only enhances convenience but also expands access, mainly for underserved groups such as rural populations or those with mobility issues, thereby ensuring inclusivity and flexibility in the healthcare domain. Besides, the paper highlights the vitality of collaboration between healthcare professionals, policymakers, and tech developers in unveiling the accessibility and usability impediments. What else helps in minimizing healthcare differences and enhancing patient outcomes is guaranteeing equitable access to telehealth for Medicaid beneficiaries. The paper systematically offers major recommendations to increase digital accessibility in telehealth, thereby creating a patient-oriented and all-encompassing healthcare system.
△ Less
Submitted 29 May, 2025;
originally announced May 2025.
-
The Dual Impact of Artificial Intelligence in Healthcare: Balancing Advancements with Ethical and Operational Challenges
Authors:
Balaji Shesharao Ingole,
Vishnu Ramineni,
Nikhil Kumar Pulipeta,
Manoj Jayntilal Kathiriya,
Manjunatha Sughaturu Krishnappa,
Vivekananda Jayaram
Abstract:
The synchronic and diachronic study of the evolution of Artificial Intelligence (AI) unveils one prominent fact that its effect can be traced in almost all fields such as healthcare industry. The growth is perceived holistically in software, hardware implementation, or application in these various fields. As the title suggests, the review will highlight the impact of AI on healthcare possibly in a…
▽ More
The synchronic and diachronic study of the evolution of Artificial Intelligence (AI) unveils one prominent fact that its effect can be traced in almost all fields such as healthcare industry. The growth is perceived holistically in software, hardware implementation, or application in these various fields. As the title suggests, the review will highlight the impact of AI on healthcare possibly in all dimensions including precision medicine, diagnostics, drug development, automation of the process, etc., explicating whether AI is a blessing or a curse or both. With the availability of enough data and analysis to examine the topic at hand, however, its application is still functioning in quite early stages in many fields, the present work will endeavour to provide an answer to the question. This paper takes a close look at how AI is transforming areas such as diagnostics, precision medicine, and drug discovery, while also addressing some of the key ethical challenges it brings. Issues like patient privacy, safety, and the fairness of AI decisions are explored to understand whether AI in healthcare is a positive force, a potential risk, or perhaps both.
△ Less
Submitted 8 November, 2024;
originally announced November 2024.
-
AI-driven innovation in medicaid: enhancing access, cost efficiency, and population health management
Authors:
Balaji Shesharao Ingole,
Vishnu Ramineni,
Manjunatha Sughaturu Krishnappa,
Vivekananda Jayaram
Abstract:
The U.S. Medicaid program is experiencing critical challenges that include rapidly increasing healthcare costs, uneven care accessibility, and the challenge associated with addressing a varied set of population health needs. This paper investigates the transformative potential of Artificial Intelligence (AI) in reshaping Medicaid by streamlining operations, improving patient results, and lowering…
▽ More
The U.S. Medicaid program is experiencing critical challenges that include rapidly increasing healthcare costs, uneven care accessibility, and the challenge associated with addressing a varied set of population health needs. This paper investigates the transformative potential of Artificial Intelligence (AI) in reshaping Medicaid by streamlining operations, improving patient results, and lowering costs. We delve into the pivotal role of AI in predictive analytics, care coordination, the detection of fraud, and personalized medicine. By leveraging insights from advanced data models and addressing challenges particular to Medicaid, we put forward AI-driven solutions that prioritize equitable care and improved public health outcomes. This study underscores the urgency of integrating AI into Medicaid to not only improve operational effectiveness but also to create a more accessible and equitable healthcare system for all beneficiaries.
△ Less
Submitted 11 October, 2024;
originally announced October 2024.
-
Advancements In Heart Disease Prediction: A Machine Learning Approach For Early Detection And Risk Assessment
Authors:
Balaji Shesharao Ingole,
Vishnu Ramineni,
Nikhil Bangad,
Koushik Kumar Ganeeb,
Priyankkumar Patel
Abstract:
The primary aim of this paper is to comprehend, assess, and analyze the role, relevance, and efficiency of machine learning models in predicting heart disease risks using clinical data. While the importance of heart disease risk prediction cannot be overstated, the application of machine learning (ML) in identifying and evaluating the impact of various features on the classification of patients wi…
▽ More
The primary aim of this paper is to comprehend, assess, and analyze the role, relevance, and efficiency of machine learning models in predicting heart disease risks using clinical data. While the importance of heart disease risk prediction cannot be overstated, the application of machine learning (ML) in identifying and evaluating the impact of various features on the classification of patients with and without heart disease, as well as in generating a reliable clinical dataset, is equally significant. This study relies primarily on cross-sectional clinical data. The ML approach is designed to enhance the consideration of various clinical features in the heart disease prognosis process. Some features emerge as strong predictors, adding significant value. The paper evaluates seven ML classifiers: Logistic Regression, Random Forest, Decision Tree, Naive Bayes, k-Nearest Neighbors, Neural Networks, and Support Vector Machine (SVM). The performance of each model is assessed based on accuracy metrics. Notably, the Support Vector Machine (SVM) demonstrates the highest accuracy at 91.51%, confirming its superiority among the evaluated models in terms of predictive capability. The overall findings of this research highlight the advantages of advanced computational methodologies in the evaluation, prediction, improvement, and management of cardiovascular risks. In other words, the strong performance of the SVM model illustrates its applicability and value in clinical settings, paving the way for further advancements in personalized medicine and healthcare.
△ Less
Submitted 16 October, 2024;
originally announced October 2024.
-
Understanding the Role of User Profile in the Personalization of Large Language Models
Authors:
Bin Wu,
Zhengyan Shi,
Hossein A. Rahmani,
Varsha Ramineni,
Emine Yilmaz
Abstract:
Utilizing user profiles to personalize Large Language Models (LLMs) has been shown to enhance the performance on a wide range of tasks. However, the precise role of user profiles and their effect mechanism on LLMs remains unclear. This study first confirms that the effectiveness of user profiles is primarily due to personalization information rather than semantic information. Furthermore, we inves…
▽ More
Utilizing user profiles to personalize Large Language Models (LLMs) has been shown to enhance the performance on a wide range of tasks. However, the precise role of user profiles and their effect mechanism on LLMs remains unclear. This study first confirms that the effectiveness of user profiles is primarily due to personalization information rather than semantic information. Furthermore, we investigate how user profiles affect the personalization of LLMs. Within the user profile, we reveal that it is the historical personalized response produced or approved by users that plays a pivotal role in personalizing LLMs. This discovery unlocks the potential of LLMs to incorporate a greater number of user profiles within the constraints of limited input length. As for the position of user profiles, we observe that user profiles integrated into different positions of the input context do not contribute equally to personalization. Instead, where the user profile that is closer to the beginning affects more on the personalization of LLMs. Our findings reveal the role of user profiles for the personalization of LLMs, and showcase how incorporating user profiles impacts performance providing insight to leverage user profiles effectively.
△ Less
Submitted 22 June, 2024;
originally announced June 2024.
