Showing 1–14 of 14 results for author: Turner, L

LLM-as-a-Fuzzy-Judge: Fine-Tuning Large Language Models as a Clinical Evaluation Judge with Fuzzy Logic

Authors: Weibing Zheng, Laurah Turner, Jess Kropczynski, Murat Ozer, Tri Nguyen, Shane Halse

Abstract: Clinical communication skills are critical in medical education, and practicing and assessing clinical communication skills on a scale is challenging. Although LLM-powered clinical scenario simulations have shown promise in enhancing medical students' clinical practice, providing automated and scalable clinical evaluation that follows nuanced physician judgment is difficult. This paper combines fu… ▽ More Clinical communication skills are critical in medical education, and practicing and assessing clinical communication skills on a scale is challenging. Although LLM-powered clinical scenario simulations have shown promise in enhancing medical students' clinical practice, providing automated and scalable clinical evaluation that follows nuanced physician judgment is difficult. This paper combines fuzzy logic and Large Language Model (LLM) and proposes LLM-as-a-Fuzzy-Judge to address the challenge of aligning the automated evaluation of medical students' clinical skills with subjective physicians' preferences. LLM-as-a-Fuzzy-Judge is an approach that LLM is fine-tuned to evaluate medical students' utterances within student-AI patient conversation scripts based on human annotations from four fuzzy sets, including Professionalism, Medical Relevance, Ethical Behavior, and Contextual Distraction. The methodology of this paper started from data collection from the LLM-powered medical education system, data annotation based on multidimensional fuzzy sets, followed by prompt engineering and the supervised fine-tuning (SFT) of the pre-trained LLMs using these human annotations. The results show that the LLM-as-a-Fuzzy-Judge achieves over 80\% accuracy, with major criteria items over 90\%, effectively leveraging fuzzy logic and LLM as a solution to deliver interpretable, human-aligned assessment. This work suggests the viability of leveraging fuzzy logic and LLM to align with human preferences, advances automated evaluation in medical education, and supports more robust assessment and judgment practices. The GitHub repository of this work is available at https://github.com/2sigmaEdTech/LLMAsAJudge △ Less

Submitted 12 June, 2025; originally announced June 2025.

Comments: 12 pages, 1 figure, 2025 IFSA World Congress NAFIPS Annual Meeting

ACM Class: D.2.4; K.3.1; C.3; I.2.6

Dialz: A Python Toolkit for Steering Vectors

Authors: Zara Siddique, Liam D. Turner, Luis Espinosa-Anke

Abstract: We introduce Dialz, a framework for advancing research on steering vectors for open-source LLMs, implemented in Python. Steering vectors allow users to modify activations at inference time to amplify or weaken a 'concept', e.g. honesty or positivity, providing a more powerful alternative to prompting or fine-tuning. Dialz supports a diverse set of tasks, including creating contrastive pair dataset… ▽ More We introduce Dialz, a framework for advancing research on steering vectors for open-source LLMs, implemented in Python. Steering vectors allow users to modify activations at inference time to amplify or weaken a 'concept', e.g. honesty or positivity, providing a more powerful alternative to prompting or fine-tuning. Dialz supports a diverse set of tasks, including creating contrastive pair datasets, computing and applying steering vectors, and visualizations. Unlike existing libraries, Dialz emphasizes modularity and usability, enabling both rapid prototyping and in-depth analysis. We demonstrate how Dialz can be used to reduce harmful outputs such as stereotypes, while also providing insights into model behaviour across different layers. We release Dialz with full documentation, tutorials, and support for popular open-source models to encourage further research in safe and controllable language generation. Dialz enables faster research cycles and facilitates insights into model interpretability, paving the way for safer, more transparent, and more reliable AI systems. △ Less

Submitted 3 June, 2025; v1 submitted 4 May, 2025; originally announced May 2025.

Comments: Accepted to ACL System Demo 2025

Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models

Authors: Tri Nguyen, Lohith Srikanth Pentapalli, Magnus Sieverding, Laurah Turner, Seth Overla, Weibing Zheng, Chris Zhou, David Furniss, Danielle Weber, Michael Gharib, Matt Kelleher, Michael Shukis, Cameron Pawlik, Kelly Cohen

Abstract: Jailbreaking in Large Language Models (LLMs) threatens their safe use in sensitive domains like education by allowing users to bypass ethical safeguards. This study focuses on detecting jailbreaks in 2-Sigma, a clinical education platform that simulates patient interactions using LLMs. We annotated over 2,300 prompts across 158 conversations using four linguistic variables shown to correlate stron… ▽ More Jailbreaking in Large Language Models (LLMs) threatens their safe use in sensitive domains like education by allowing users to bypass ethical safeguards. This study focuses on detecting jailbreaks in 2-Sigma, a clinical education platform that simulates patient interactions using LLMs. We annotated over 2,300 prompts across 158 conversations using four linguistic variables shown to correlate strongly with jailbreak behavior. The extracted features were used to train several predictive models, including Decision Trees, Fuzzy Logic-based classifiers, Boosting methods, and Logistic Regression. Results show that feature-based predictive models consistently outperformed Prompt Engineering, with the Fuzzy Decision Tree achieving the best overall performance. Our findings demonstrate that linguistic-feature-based models are effective and explainable alternatives for jailbreak detection. We suggest future work explore hybrid frameworks that integrate prompt-based flexibility with rule-based robustness for real-time, spectrum-based jailbreak monitoring in educational LLMs. △ Less

Submitted 21 April, 2025; originally announced May 2025.

Adaptive PCA-Based Outlier Detection for Multi-Feature Time Series in Space Missions

Authors: Jonah Ekelund, Savvas Raptis, Vicki Toy-Edens, Wenli Mo, Drew L. Turner, Ian J. Cohen, Stefano Markidis

Abstract: Analyzing multi-featured time series data is critical for space missions making efficient event detection, potentially onboard, essential for automatic analysis. However, limited onboard computational resources and data downlink constraints necessitate robust methods for identifying regions of interest in real time. This work presents an adaptive outlier detection algorithm based on the reconstruc… ▽ More Analyzing multi-featured time series data is critical for space missions making efficient event detection, potentially onboard, essential for automatic analysis. However, limited onboard computational resources and data downlink constraints necessitate robust methods for identifying regions of interest in real time. This work presents an adaptive outlier detection algorithm based on the reconstruction error of Principal Component Analysis (PCA) for feature reduction, designed explicitly for space mission applications. The algorithm adapts dynamically to evolving data distributions by using Incremental PCA, enabling deployment without a predefined model for all possible conditions. A pre-scaling process normalizes each feature's magnitude while preserving relative variance within feature types. We demonstrate the algorithm's effectiveness in detecting space plasma events, such as distinct space environments, dayside and nightside transients phenomena, and transition layers through NASA's MMS mission observations. Additionally, we apply the method to NASA's THEMIS data, successfully identifying a dayside transient using onboard-available measurements. △ Less

Submitted 22 April, 2025; originally announced April 2025.

Comments: Accepted to ICCS 2025

Shifting Perspectives: Steering Vector Ensembles for Robust Bias Mitigation in LLMs

Authors: Zara Siddique, Irtaza Khalid, Liam D. Turner, Luis Espinosa-Anke

Abstract: We present a novel approach to bias mitigation in large language models (LLMs) by applying steering vectors to modify model activations in forward passes. We employ Bayesian optimization to systematically identify effective contrastive pair datasets across nine bias axes. When optimized on the BBQ dataset, our individually tuned steering vectors achieve average improvements of 12.2%, 4.7%, and 3.2… ▽ More We present a novel approach to bias mitigation in large language models (LLMs) by applying steering vectors to modify model activations in forward passes. We employ Bayesian optimization to systematically identify effective contrastive pair datasets across nine bias axes. When optimized on the BBQ dataset, our individually tuned steering vectors achieve average improvements of 12.2%, 4.7%, and 3.2% over the baseline for Mistral, Llama, and Qwen, respectively. Building on these promising results, we introduce Steering Vector Ensembles (SVE), a method that averages multiple individually optimized steering vectors, each targeting a specific bias axis such as age, race, or gender. By leveraging their collective strength, SVE outperforms individual steering vectors in both bias reduction and maintaining model performance. The work presents the first systematic investigation of steering vectors for bias mitigation, and we demonstrate that SVE is a powerful and computationally efficient strategy for reducing bias in LLMs, with broader implications for enhancing AI safety. △ Less

Submitted 7 March, 2025; originally announced March 2025.

Comments: Submitted to ACL 2025

Geometry Aware Passthrough Mitigates Cybersickness

Authors: Trishia El Chemaly, Mohit Goyal, Tinglin Duan, Vrushank Phadnis, Sakar Khattar, Bjorn Vlaskamp, Achin Kulshrestha, Eric Lee Turner, Aveek Purohit, Gregory Neiswander, Konstantine Tsotsos

Abstract: Virtual Reality headsets isolate users from the real-world by restricting their perception to the virtual-world. Video See-Through (VST) headsets address this by utilizing world-facing cameras to create Augmented Reality experiences. However, directly displaying camera feeds causes visual discomfort and cybersickness due to the inaccurate perception of scale and exaggerated motion parallax. This p… ▽ More Virtual Reality headsets isolate users from the real-world by restricting their perception to the virtual-world. Video See-Through (VST) headsets address this by utilizing world-facing cameras to create Augmented Reality experiences. However, directly displaying camera feeds causes visual discomfort and cybersickness due to the inaccurate perception of scale and exaggerated motion parallax. This paper demonstrates the potential of geometry aware passthrough systems in mitigating cybersickness through accurate depth perception. We first present a methodology to benchmark and compare passthrough algorithms. Furthermore, we design a protocol to quantitatively measure cybersickness experienced by users in VST headsets. Using this protocol, we conduct a user study to compare direct passthrough and geometry aware passthrough systems. To the best of our knowledge, our study is the first one to reveal significantly reduced nausea, disorientation, and total scores of cybersickness with geometry aware passthrough. It also uncovers several potential avenues to further mitigate visually-induced discomfort. △ Less

Submitted 24 March, 2025; v1 submitted 17 February, 2025; originally announced February 2025.

Who is better at math, Jenny or Jingzhen? Uncovering Stereotypes in Large Language Models

Authors: Zara Siddique, Liam D. Turner, Luis Espinosa-Anke

Abstract: Large language models (LLMs) have been shown to propagate and amplify harmful stereotypes, particularly those that disproportionately affect marginalised communities. To understand the effect of these stereotypes more comprehensively, we introduce GlobalBias, a dataset of 876k sentences incorporating 40 distinct gender-by-ethnicity groups alongside descriptors typically used in bias literature, wh… ▽ More Large language models (LLMs) have been shown to propagate and amplify harmful stereotypes, particularly those that disproportionately affect marginalised communities. To understand the effect of these stereotypes more comprehensively, we introduce GlobalBias, a dataset of 876k sentences incorporating 40 distinct gender-by-ethnicity groups alongside descriptors typically used in bias literature, which enables us to study a broad set of stereotypes from around the world. We use GlobalBias to directly probe a suite of LMs via perplexity, which we use as a proxy to determine how certain stereotypes are represented in the model's internal representations. Following this, we generate character profiles based on given names and evaluate the prevalence of stereotypes in model outputs. We find that the demographic groups associated with various stereotypes remain consistent across model likelihoods and model outputs. Furthermore, larger models consistently display higher levels of stereotypical outputs, even when explicitly instructed not to. △ Less

Submitted 9 October, 2024; v1 submitted 9 July, 2024; originally announced July 2024.

Comments: Accepted to EMNLP Main 2024

Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?

Authors: Thomas Greatrix, Roger Whitaker, Liam Turner, Walter Colombo

Abstract: The potential for Large Language Models (LLMs) to generate new information offers a potential step change for research and innovation. This is challenging to assert as it can be difficult to determine what an LLM has previously seen during training, making "newness" difficult to substantiate. In this paper we observe that LLMs are able to perform sophisticated reasoning on problems with a spatial… ▽ More The potential for Large Language Models (LLMs) to generate new information offers a potential step change for research and innovation. This is challenging to assert as it can be difficult to determine what an LLM has previously seen during training, making "newness" difficult to substantiate. In this paper we observe that LLMs are able to perform sophisticated reasoning on problems with a spatial dimension, that they are unlikely to have previously directly encountered. While not perfect, this points to a significant level of understanding that state-of-the-art LLMs can now achieve, supporting the proposition that LLMs are able to yield significant emergent properties. In particular, Claude 3 is found to perform well in this regard. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Local Hadwiger's Conjecture

Authors: Benjamin Moore, Luke Postle, Lise Turner

Abstract: We propose local versions of Hadwiger's Conjecture, where only balls of radius $Ω(\log(v(G)))$ around each vertex are required to be $K_{t}$-minor-free. We ask: if a graph is locally-$K_{t}$-minor-free, is it $t$-colourable? We show that the answer is yes when $t \leq 5$, even in the stronger setting of list-colouring, and we complement this result with a $O(\log v(G))$-round distributed colouring… ▽ More We propose local versions of Hadwiger's Conjecture, where only balls of radius $Ω(\log(v(G)))$ around each vertex are required to be $K_{t}$-minor-free. We ask: if a graph is locally-$K_{t}$-minor-free, is it $t$-colourable? We show that the answer is yes when $t \leq 5$, even in the stronger setting of list-colouring, and we complement this result with a $O(\log v(G))$-round distributed colouring algorithm in the LOCAL model. Further, we show that for large enough values of $t$, we can list-colour locally-$K_{t}$-minor-free graphs with $13\cdot \max\left\{h(t),\left\lceil \frac{31}{2}(t-1) \right\rceil \right\})$colours, where $h(t)$ is any value such that all $K_{t}$-minor-free graphs are $h(t)$-list-colourable. We again complement this with a $O(\log v(G))$-round distributed algorithm. △ Less

Submitted 14 September, 2023; v1 submitted 13 March, 2022; originally announced March 2022.

Comments: 25 pages; published in JCTB

MSC Class: 05C15; 05C83; 05C85; 68W15

6-Layer Model for a Structured Description and Categorization of Urban Traffic and Environment

Authors: Maike Scholtes, Lukas Westhofen, Lara Ruth Turner, Katrin Lotto, Michael Schuldes, Hendrik Weber, Nicolas Wagener, Christian Neurohr, Martin Bollmann, Franziska Körtke, Johannes Hiller, Michael Hoss, Julian Bock, Lutz Eckstein

Abstract: Verification and validation of automated driving functions impose large challenges. Currently, scenario-based approaches are investigated in research and industry, aiming at a reduction of testing efforts by specifying safety relevant scenarios. To define those scenarios and operate in a complex real-world design domain, a structured description of the environment is needed. Within the PEGASUS res… ▽ More Verification and validation of automated driving functions impose large challenges. Currently, scenario-based approaches are investigated in research and industry, aiming at a reduction of testing efforts by specifying safety relevant scenarios. To define those scenarios and operate in a complex real-world design domain, a structured description of the environment is needed. Within the PEGASUS research project, the 6-Layer Model (6LM) was introduced for the description of highway scenarios. This paper refines the 6LM and extends it to urban traffic and environment. As defined in PEGASUS, the 6LM provides the possibility to categorize the environment and, therefore, functions as a structured basis for subsequent scenario description. The model enables a structured description and categorization of the general environment, without incorporating any knowledge or anticipating any functions of actors. Beyond that, there is a variety of other applications of the 6LM, which are elaborated in this paper. The 6LM includes a description of the road network and traffic guidance objects, roadside structures, temporary modifications of the former, dynamic objects, environmental conditions and digital information. The work at hand specifies each layer by categorizing its items. Guidelines are formulated and explanatory examples are given to standardize the application of the model for an objective environment description. In contrast to previous publications, the model and its design are described in far more detail. Finally, the holistic description of the 6LM presented includes remarks on possible future work when expanding the concept to machine perception aspects. △ Less

Submitted 2 February, 2021; v1 submitted 9 December, 2020; originally announced December 2020.

Comments: 16 pages, 7 figures, submitted to IEEE Access

Understanding the Signature of Controversial Wikipedia Articles through Motifs in Editor Revision Networks

Authors: James R. Ashford, Liam D. Turner, Roger M. Whitaker, Alun Preece, Diane Felmlee, Don Towsley

Abstract: Wikipedia serves as a good example of how editors collaborate to form and maintain an article. The relationship between editors, derived from their sequence of editing activity, results in a directed network structure called the revision network, that potentially holds valuable insights into editing activity. In this paper we create revision networks to assess differences between controversial and… ▽ More Wikipedia serves as a good example of how editors collaborate to form and maintain an article. The relationship between editors, derived from their sequence of editing activity, results in a directed network structure called the revision network, that potentially holds valuable insights into editing activity. In this paper we create revision networks to assess differences between controversial and non-controversial articles, as labelled by Wikipedia. Originating from complex networks, we apply motif analysis, which determines the under or over-representation of induced sub-structures, in this case triads of editors. We analyse 21,631 Wikipedia articles in this way, and use principal component analysis to consider the relationship between their motif subgraph ratio profiles. Results show that a small number of induced triads play an important role in characterising relationships between editors, with controversial articles having a tendency to cluster. This provides useful insight into editing behaviour and interaction capturing counter-narratives, without recourse to semantic analysis. It also provides a potentially useful feature for future prediction of controversial Wikipedia articles. △ Less

Submitted 17 April, 2019; originally announced April 2019.

Learning Features of Network Structures Using Graphlets

Authors: Kun Tu, Jian Li, Don Towsley, Dave Braines, Liam Turner

Abstract: Networks are fundamental to the study of complex systems, ranging from social contacts, message transactions, to biological regulations and economical networks. In many realistic applications, these networks may vary over time. Modeling and analyzing such temporal properties is of additional interest as it can provide a richer characterization of relations between nodes in networks. In this paper,… ▽ More Networks are fundamental to the study of complex systems, ranging from social contacts, message transactions, to biological regulations and economical networks. In many realistic applications, these networks may vary over time. Modeling and analyzing such temporal properties is of additional interest as it can provide a richer characterization of relations between nodes in networks. In this paper, we explore the role of \emph{graphlets} in network classification for both static and temporal networks. Graphlets are small non-isomorphic induced subgraphs representing connected patterns in a network and their frequency can be used to assess network structures. We show that graphlet features, which are not captured by state-of-the-art methods, play a significant role in enhancing the performance of network classification. To that end, we propose two novel graphlet-based techniques, \emph{gl2vec} for network embedding, and \emph{gl-DCNN} for diffusion-convolutional neural networks. We demonstrate the efficacy and usability of \emph{gl2vec} and \emph{gl-DCNN} through extensive experiments using several real-world static and temporal networks. We find that features learned from graphlets can bring notable performance increases to state-of-the-art methods in network analysis. △ Less

Submitted 5 April, 2020; v1 submitted 13 December, 2018; originally announced December 2018.

Comments: A conference version has been published in IEEE/ACM ASONAM'19 with title "gl2vec: Learning feature representation using graphlets for directed networks" and can be found at https://dl.acm.org/doi/abs/10.1145/3341161.3342908

Network Classification in Temporal Networks Using Motifs

Authors: Kun Tu, Jian Li, Don Towsley, Dave Braines, Liam D. Turner

Abstract: Network classification has a variety of applications, such as detecting communities within networks and finding similarities between those representing different aspects of the real world. However, most existing work in this area focus on examining static undirected networks without considering directed edges or temporality. In this paper, we propose a new methodology that utilizes feature represe… ▽ More Network classification has a variety of applications, such as detecting communities within networks and finding similarities between those representing different aspects of the real world. However, most existing work in this area focus on examining static undirected networks without considering directed edges or temporality. In this paper, we propose a new methodology that utilizes feature representation for network classification based on the temporal motif distribution of the network and a null model for comparing against random graphs. Experimental results show that our method improves accuracy by up $10\%$ compared to the state-of-the-art embedding method in network classification, for tasks such as classifying network type, identifying communities in email exchange network, and identifying users given their app-switching behaviors. △ Less

Submitted 7 August, 2018; v1 submitted 10 July, 2018; originally announced July 2018.

Ligand Pose Optimization with Atomic Grid-Based Convolutional Neural Networks

Authors: Matthew Ragoza, Lillian Turner, David Ryan Koes

Abstract: Docking is an important tool in computational drug discovery that aims to predict the binding pose of a ligand to a target protein through a combination of pose scoring and optimization. A scoring function that is differentiable with respect to atom positions can be used for both scoring and gradient-based optimization of poses for docking. Using a differentiable grid-based atomic representation a… ▽ More Docking is an important tool in computational drug discovery that aims to predict the binding pose of a ligand to a target protein through a combination of pose scoring and optimization. A scoring function that is differentiable with respect to atom positions can be used for both scoring and gradient-based optimization of poses for docking. Using a differentiable grid-based atomic representation as input, we demonstrate that a scoring function learned by training a convolutional neural network (CNN) to identify binding poses can also be applied to pose optimization. We also show that an iteratively-trained CNN that includes poses optimized by the first CNN in its training set performs even better at optimizing randomly initialized poses than either the first CNN scoring function or AutoDock Vina. △ Less

Submitted 19 October, 2017; originally announced October 2017.

Comments: 10 pages