Showing 1–7 of 7 results for author: Wallace, R

AI-Mediated Code Comment Improvement

Authors: Maria Dhakal, Chia-Yi Su, Robert Wallace, Chris Fakhimi, Aakash Bansal, Toby Li, Yu Huang, Collin McMillan

Abstract: This paper describes an approach to improve code comments along different quality axes by rewriting those comments with customized Artificial Intelligence (AI)-based tools. We conduct an empirical study followed by grounded theory qualitative analysis to determine the quality axes to improve. Then we propose a procedure using a Large Language Model (LLM) to rewrite existing code comments along the… ▽ More This paper describes an approach to improve code comments along different quality axes by rewriting those comments with customized Artificial Intelligence (AI)-based tools. We conduct an empirical study followed by grounded theory qualitative analysis to determine the quality axes to improve. Then we propose a procedure using a Large Language Model (LLM) to rewrite existing code comments along the quality axes. We implement our procedure using GPT-4o, then distil the results into a smaller model capable of being run in-house, so users can maintain data custody. We evaluate both our approach using GPT-4o and the distilled model versions. We show in an evaluation how our procedure improves code comments along the quality axes. We release all data and source code in an online repository for reproducibility. △ Less

Submitted 13 May, 2025; originally announced May 2025.

Strain Problems got you in a Twist? Try StrainRelief: A Quantum-Accurate Tool for Ligand Strain Calculations

Authors: Ewan R. S. Wallace, Nathan C. Frey, Joshua A. Rackers

Abstract: Ligand strain energy, the energy difference between the bound and unbound conformations of a ligand, is an important component of structure-based small molecule drug design. A large majority of observed ligands in protein-small molecule co-crystal structures bind in low-strain conformations, making strain energy a useful filter for structure-based drug design. In this work we present a tool for ca… ▽ More Ligand strain energy, the energy difference between the bound and unbound conformations of a ligand, is an important component of structure-based small molecule drug design. A large majority of observed ligands in protein-small molecule co-crystal structures bind in low-strain conformations, making strain energy a useful filter for structure-based drug design. In this work we present a tool for calculating ligand strain with a high accuracy. StrainRelief uses a MACE Neural Network Potential (NNP), trained on a large database of Density Functional Theory (DFT) calculations to estimate ligand strain of neutral molecules with quantum accuracy. We show that this tool estimates strain energy differences relative to DFT to within 1.4 kcal/mol, more accurately than alternative NNPs. These results highlight the utility of NNPs in drug discovery, and provide a useful tool for drug discovery teams. △ Less

Submitted 17 March, 2025; originally announced March 2025.

Moving Beyond LDA: A Comparison of Unsupervised Topic Modelling Techniques for Qualitative Data Analysis of Online Communities

Authors: Amandeep Kaur, James R. Wallace

Abstract: Social media constitutes a rich and influential source of information for qualitative researchers. Although computational techniques like topic modelling assist with managing the volume and diversity of social media content, qualitative researcher's lack of programming expertise creates a significant barrier to their adoption. In this paper we explore how BERTopic, an advanced Large Language Model… ▽ More Social media constitutes a rich and influential source of information for qualitative researchers. Although computational techniques like topic modelling assist with managing the volume and diversity of social media content, qualitative researcher's lack of programming expertise creates a significant barrier to their adoption. In this paper we explore how BERTopic, an advanced Large Language Model (LLM)-based topic modelling technique, can support qualitative data analysis of social media. We conducted interviews and hands-on evaluations in which qualitative researchers compared topics from three modelling techniques: LDA, NMF, and BERTopic. BERTopic was favoured by 8 of 12 participants for its ability to provide detailed, coherent clusters for deeper understanding and actionable insights. Participants also prioritised topic relevance, logical organisation, and the capacity to reveal unexpected relationships within the data. Our findings underscore the potential of LLM-based techniques for supporting qualitative analysis. △ Less

Submitted 18 December, 2024; originally announced December 2024.

The Shape of Agency: Designing for Personal Agency in Qualitative Data Analysis

Authors: Luka Ugaya Mazza, Plinio Morita, James R. Wallace

Abstract: Computational thematic analysis is rapidly emerging as a method of using large text corpora to understand the lived experience of people across the continuum of health care: patients, practitioners, and everyone in between. However, many qualitative researchers do not have the necessary programming skills to write machine learning code on their own, but also seek to maintain ownership, intimacy, a… ▽ More Computational thematic analysis is rapidly emerging as a method of using large text corpora to understand the lived experience of people across the continuum of health care: patients, practitioners, and everyone in between. However, many qualitative researchers do not have the necessary programming skills to write machine learning code on their own, but also seek to maintain ownership, intimacy, and control over their analysis. In this work we explore the use of data visualizations to foster researcher agency and make computational thematic analysis more accessible to domain experts. We used a design science research approach to develop a datavis prototype over four phases: (1) problem comprehension, (2) specifying needs and requirements, (3) prototype development, and (4) feedback on the prototype. We show that qualitative researchers have a wide range of cognitive needs when conducting data analysis and place high importance upon choices and freedom, wanting to feel autonomy over their own research and not be replaced or hindered by AI. △ Less

Submitted 18 December, 2024; originally announced December 2024.

Programmer Visual Attention During Context-Aware Code Summarization

Authors: Robert Wallace, Aakash Bansal, Zachary Karas, Ningzhi Tang, Yu Huang, Toby Jia-Jun Li, Collin McMillan

Abstract: Abridged: Programmer attention represents the visual focus of programmers on parts of the source code in pursuit of programming tasks. We conducted an in-depth human study with 10 Java programmers, where each programmer generated summaries for 40 methods from five large Java projects over five one-hour sessions. We used eye-tracking equipment to map the visual attention of programmers while they w… ▽ More Abridged: Programmer attention represents the visual focus of programmers on parts of the source code in pursuit of programming tasks. We conducted an in-depth human study with 10 Java programmers, where each programmer generated summaries for 40 methods from five large Java projects over five one-hour sessions. We used eye-tracking equipment to map the visual attention of programmers while they wrote the summaries. We also rate the quality of each summary. We found eye-gaze patterns and metrics that define common behaviors between programmer attention during context-aware code summarization. Specifically, we found that programmers need to read significantly (p<0.01) fewer words and make significantly (p<0.03) fewer revisits to words as they summarize more methods during a session, while maintaining the quality of summaries. We also found that the amount of source code a participant looks at correlates with a higher quality summary, but this trend follows a bell-shaped curve, such that after a threshold reading more source code leads to a significant (p<0.01) decrease in the quality of summaries. We also gathered insight into the type of methods in the project that provide the most contextual information for code summarization based on programmer attention. Specifically, we observed that programmers spent a majority of their time looking at methods inside the same class as the target method to be summarized. Surprisingly, we found that programmers spent significantly less time looking at methods in the call graph of the target method. We discuss how our empirical observations may aid future studies towards modeling programmer attention and improving context-aware automatic source code summarization. △ Less

Submitted 25 March, 2025; v1 submitted 28 May, 2024; originally announced May 2024.

Comments: 13 pages, 4 figures, 5 tables. Published in IEEE Transactions on Software Engineering

EyeDoc: Documentation Navigation with Eye Tracking

Authors: Robert Wallace, Collin McMillan

Abstract: We demonstrate EyeDoc, a tool for navigating software documentation with the use of the eyes. When programming, developers often have many windows open such as an IDE, consoles and GUIs for third-party utilities, the application under development, and a web browser for navigating documentation. Several studies have shown that the navigation among these different tasks imposes a small mental load w… ▽ More We demonstrate EyeDoc, a tool for navigating software documentation with the use of the eyes. When programming, developers often have many windows open such as an IDE, consoles and GUIs for third-party utilities, the application under development, and a web browser for navigating documentation. Several studies have shown that the navigation among these different tasks imposes a small mental load which, over time, adds to a significant decrease in productivity. One solution to this problem is to increase "screen real estate" with larger monitors and higher resolutions, so that more information can be presented to the programmer at one time. But this solution has limits: the complexity and size of software is also increasing rapidly. In this paper, we use eye-tracking technology to build a tool for navigating documentation. All a programmer needs to do to use EyeDoc is look at the monitor containing the documentation. The tool detects when the eyes glance at different components of the documentation, and allows the programmer to navigate to those components by gazes and blinking. The programmer does not need to move his or her hands, or risk losing the context of other tasks on the desktop. We built EyeDoc as a research prototype and proof-of-concept using low-cost consumer eye-tracking hardware and our own software built as a JavaScript addition to JavaDocs. This paper describes the tool's design, use, and strategy for evaluation and future development. △ Less

Submitted 28 February, 2019; originally announced March 2019.

Using English as Pivot to Extract Persian-Italian Parallel Sentences from Non-Parallel Corpora

Authors: Ebrahim Ansari, M. H. Sadreddini, Mostafa Sheikhalishahi, Richard Wallace, Fatemeh Alimardani

Abstract: The effectiveness of a statistical machine translation system (SMT) is very dependent upon the amount of parallel corpus used in the training phase. For low-resource language pairs there are not enough parallel corpora to build an accurate SMT. In this paper, a novel approach is presented to extract bilingual Persian-Italian parallel sentences from a non-parallel (comparable) corpus. In this study… ▽ More The effectiveness of a statistical machine translation system (SMT) is very dependent upon the amount of parallel corpus used in the training phase. For low-resource language pairs there are not enough parallel corpora to build an accurate SMT. In this paper, a novel approach is presented to extract bilingual Persian-Italian parallel sentences from a non-parallel (comparable) corpus. In this study, English is used as the pivot language to compute the matching scores between source and target sentences and candidate selection phase. Additionally, a new monolingual sentence similarity metric, Normalized Google Distance (NGD) is proposed to improve the matching process. Moreover, some extensions of the baseline system are applied to improve the quality of extracted sentences measured with BLEU. Experimental results show that using the new pivot based extraction can increase the quality of bilingual corpus significantly and consequently improves the performance of the Persian-Italian SMT system. △ Less

Submitted 28 January, 2017; originally announced January 2017.

Comments: 30 pages, Accepted to be published in "Applications of Comparable Corpora", Berlin: Language Science Press