-
DisasterQA: A Benchmark for Assessing the performance of LLMs in Disaster Response
Authors:
Rajat Rawat
Abstract:
Disasters can result in the deaths of many, making quick response times vital. Large Language Models (LLMs) have emerged as valuable in the field. LLMs can be used to process vast amounts of textual information quickly providing situational context during a disaster. However, the question remains whether LLMs should be used for advice and decision making in a disaster. To evaluate the capabilities…
▽ More
Disasters can result in the deaths of many, making quick response times vital. Large Language Models (LLMs) have emerged as valuable in the field. LLMs can be used to process vast amounts of textual information quickly providing situational context during a disaster. However, the question remains whether LLMs should be used for advice and decision making in a disaster. To evaluate the capabilities of LLMs in disaster response knowledge, we introduce a benchmark: DisasterQA created from six online sources. The benchmark covers a wide range of disaster response topics. We evaluated five LLMs each with four different prompting methods on our benchmark, measuring both accuracy and confidence levels through Logprobs. The results indicate that LLMs require improvement on disaster response knowledge. We hope that this benchmark pushes forth further development of LLMs in disaster response, ultimately enabling these models to work alongside. emergency managers in disasters.
△ Less
Submitted 8 October, 2024;
originally announced October 2024.
-
DiversityMedQA: Assessing Demographic Biases in Medical Diagnosis using Large Language Models
Authors:
Rajat Rawat,
Hudson McBride,
Dhiyaan Nirmal,
Rajarshi Ghosh,
Jong Moon,
Dhruv Alamuri,
Sean O'Brien,
Kevin Zhu
Abstract:
As large language models (LLMs) gain traction in healthcare, concerns about their susceptibility to demographic biases are growing. We introduce {DiversityMedQA}, a novel benchmark designed to assess LLM responses to medical queries across diverse patient demographics, such as gender and ethnicity. By perturbing questions from the MedQA dataset, which comprises medical board exam questions, we cre…
▽ More
As large language models (LLMs) gain traction in healthcare, concerns about their susceptibility to demographic biases are growing. We introduce {DiversityMedQA}, a novel benchmark designed to assess LLM responses to medical queries across diverse patient demographics, such as gender and ethnicity. By perturbing questions from the MedQA dataset, which comprises medical board exam questions, we created a benchmark that captures the nuanced differences in medical diagnosis across varying patient profiles. Our findings reveal notable discrepancies in model performance when tested against these demographic variations. Furthermore, to ensure the perturbations were accurate, we also propose a filtering strategy that validates each perturbation. By releasing DiversityMedQA, we provide a resource for evaluating and mitigating demographic bias in LLM medical diagnoses.
△ Less
Submitted 6 December, 2024; v1 submitted 2 September, 2024;
originally announced September 2024.
-
Intelligent Acoustic Module for Autonomous Vehicles using Fast Gated Recurrent approach
Authors:
Raghav Rawat,
Shreyash Gupta,
Shreyas Mohapatra,
Sujata Priyambada Mishra,
Sreesankar Rajagopal
Abstract:
This paper elucidates a model for acoustic single and multi-tone classification in resource constrained edge devices. The proposed model is of State-of-the-art Fast Accurate Stable Tiny Gated Recurrent Neural Network. This model has resulted in improved performance metrics and lower size compared to previous hypothesized methods by using lesser parameters with higher efficiency and employment of a…
▽ More
This paper elucidates a model for acoustic single and multi-tone classification in resource constrained edge devices. The proposed model is of State-of-the-art Fast Accurate Stable Tiny Gated Recurrent Neural Network. This model has resulted in improved performance metrics and lower size compared to previous hypothesized methods by using lesser parameters with higher efficiency and employment of a noise reduction algorithm. The model is implemented as an acoustic AI module, focused for the application of sound identification, localization, and deployment on AI systems like that of an autonomous car. Further, the inclusion of localization techniques carries the potential of adding a new dimension to the multi-tone classifiers present in autonomous vehicles, as its demand increases in urban cities and developing countries in the future.
△ Less
Submitted 6 December, 2021;
originally announced December 2021.
-
Quantum Machine Learning with HQC Architectures using non-Classically Simulable Feature Maps
Authors:
Syed Farhan Ahmad,
Raghav Rawat,
Minal Moharir
Abstract:
Hybrid Quantum-Classical (HQC) Architectures are used in near-term NISQ Quantum Computers for solving Quantum Machine Learning problems. The quantum advantage comes into picture due to the exponential speedup offered over classical computing. One of the major challenges in implementing such algorithms is the choice of quantum embeddings and the use of a functionally correct quantum variational cir…
▽ More
Hybrid Quantum-Classical (HQC) Architectures are used in near-term NISQ Quantum Computers for solving Quantum Machine Learning problems. The quantum advantage comes into picture due to the exponential speedup offered over classical computing. One of the major challenges in implementing such algorithms is the choice of quantum embeddings and the use of a functionally correct quantum variational circuit. In this paper, we present an application of QSVM (Quantum Support Vector Machines) to predict if a person will require mental health treatment in the tech world in the future using the dataset from OSMI Mental Health Tech Surveys. We achieve this with non-classically simulable feature maps and prove that NISQ HQC Architectures for Quantum Machine Learning can be used alternatively to create good performance models in near-term real-world applications.
△ Less
Submitted 13 April, 2024; v1 submitted 21 March, 2021;
originally announced March 2021.
-
Virtual Proximity Citation (VCP): A Supervised Deep Learning Method to Relate Uncited Papers On Grounds of Citation Proximity
Authors:
Rohit Rawat
Abstract:
Citation based approaches have seen good progress for recommending research papers using citations in the paper. Citation proximity analysis which uses the in-text citation proximity to find relatedness between two research papers is better than co-citation analysis and bibliographic analysis. However, one common problem which exists in each approach is that paper should be well cited. If document…
▽ More
Citation based approaches have seen good progress for recommending research papers using citations in the paper. Citation proximity analysis which uses the in-text citation proximity to find relatedness between two research papers is better than co-citation analysis and bibliographic analysis. However, one common problem which exists in each approach is that paper should be well cited. If documents are not cited properly or not cited at all, then using these approaches will not be helpful. To overcome the problem, this paper discusses the approach Virtual Citation Proximity (VCP) which uses Siamese Neural Network along with the notion of citation proximity analysis and content-based filtering. To train this model, the actual distance between the two citations in a document is used as ground truth, this distance is the word count between the two citations. VCP is trained on Wikipedia articles for which the actual word count is available which is used to calculate the similarity between the documents. This can be used to calculate relatedness between two documents in a way they would have been cited in the proximity even if the documents are uncited. This approach has shown a great improvement in predicting proximity with basic neural networks over the approach which uses the Average Citation Proximity index value as the ground truth. This can be improved by using a complex neural network and proper hyper tuning of parameters.
△ Less
Submitted 25 September, 2020;
originally announced September 2020.
-
Neuro-symbolic EDA-based Optimisation using ILP-enhanced DBNs
Authors:
Sarmimala Saikia,
Lovekesh Vig,
Ashwin Srinivasan,
Gautam Shroff,
Puneet Agarwal,
Richa Rawat
Abstract:
We investigate solving discrete optimisation problems using the estimation of distribution (EDA) approach via a novel combination of deep belief networks(DBN) and inductive logic programming (ILP).While DBNs are used to learn the structure of successively better feasible solutions,ILP enables the incorporation of domain-based background knowledge related to the goodness of solutions.Recent work sh…
▽ More
We investigate solving discrete optimisation problems using the estimation of distribution (EDA) approach via a novel combination of deep belief networks(DBN) and inductive logic programming (ILP).While DBNs are used to learn the structure of successively better feasible solutions,ILP enables the incorporation of domain-based background knowledge related to the goodness of solutions.Recent work showed that ILP could be an effective way to use domain knowledge in an EDA scenario.However,in a purely ILP-based EDA,sampling successive populations is either inefficient or not straightforward.In our Neuro-symbolic EDA,an ILP engine is used to construct a model for good solutions using domain-based background knowledge.These rules are introduced as Boolean features in the last hidden layer of DBNs used for EDA-based optimization.This incorporation of logical ILP features requires some changes while training and sampling from DBNs: (a)our DBNs need to be trained with data for units at the input layer as well as some units in an otherwise hidden layer, and (b)we would like the samples generated to be drawn from instances entailed by the logical model.We demonstrate the viability of our approach on instances of two optimisation problems: predicting optimal depth-of-win for the KRK endgame,and jobshop scheduling.Our results are promising: (i)On each iteration of distribution estimation,samples obtained with an ILP-assisted DBN have a substantially greater proportion of good solutions than samples generated using a DBN without ILP features, and (ii)On termination of distribution estimation,samples obtained using an ILP-assisted DBN contain more near-optimal samples than samples from a DBN without ILP features.These results suggest that the use of ILP-constructed theories could be useful for incorporating complex domain-knowledge into deep models for estimation of distribution based procedures.
△ Less
Submitted 20 December, 2016;
originally announced December 2016.
-
Two Novel Defenses against Motion-Based Keystroke Inference Attacks
Authors:
Yihang Song,
Madhur Kukreti,
Rahul Rawat,
Urs Hengartner
Abstract:
Nowadays smartphones come embedded with multiple motion sensors, such as an accelerometer, a gyroscope and an orientation sensor. With these sensors, apps can gather more information and therefore provide end users with more functionality. However, these sensors also introduce the potential risk of leaking a user's private information because apps can access these sensors without requiring securit…
▽ More
Nowadays smartphones come embedded with multiple motion sensors, such as an accelerometer, a gyroscope and an orientation sensor. With these sensors, apps can gather more information and therefore provide end users with more functionality. However, these sensors also introduce the potential risk of leaking a user's private information because apps can access these sensors without requiring security permissions. By monitoring a device's motion, a malicious app may be able to infer sensitive information about the owner of the device. For example, related work has shown that sensitive information entered by a user on a device's touchscreen, such as numerical PINs or passwords, can be inferred from accelerometer and gyroscope data.
In this paper, we study these motion-based keystroke inference attacks to determine what information they need to succeed. Based on this study, we propose two novel approaches to defend against keystroke inference attacks: 1) Reducing sensor data accuracy; 2) Random keyboard layout generation. We present the design and the implementation of these two defences on the Android platform and show how they significantly reduce the accuracy of keystroke inference attacks. We also conduct multiple user studies to evaluate the usability and feasibility of these two defences. Finally, we determine the impact of the defences on apps that have legitimate reasons to access motion sensors and show that the impact is negligible.
△ Less
Submitted 28 October, 2014;
originally announced October 2014.
