-
Exploring the use of Generative AI to Support Automated Just-in-Time Programming for Visual Scene Displays
Authors:
Cynthia Zastudil,
Christine Holyfield,
Christine Kapp,
Xandria Crosland,
Elizabeth Lorah,
Tara Zimmerman,
Stephen MacNeil
Abstract:
Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options within contextualized images. However, existing VSDs often present default images that may lack relevance or require manual configuration, placing a significant burden on communication…
▽ More
Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options within contextualized images. However, existing VSDs often present default images that may lack relevance or require manual configuration, placing a significant burden on communication partners. In this study, we assess the feasibility of leveraging large multimodal models (LMM), such as GPT-4V, to automatically create communication options for VSDs. Communication options were sourced from a LMM and speech-language pathologists (SLPs) and AAC researchers (N=13) for evaluation through an expert assessment conducted by the SLPs and AAC researchers. We present the study's findings, supplemented by insights from semi-structured interviews (N=5) about SLP's and AAC researchers' opinions on the use of generative AI in augmentative and alternative communication devices. Our results indicate that the communication options generated by the LMM were contextually relevant and often resembled those created by humans. However, vital questions remain that must be addressed before LMMs can be confidently implemented in AAC devices.
△ Less
Submitted 20 August, 2024;
originally announced August 2024.
-
Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study
Authors:
Lena Schmidt,
Kaitlyn Hair,
Sergio Graziosi,
Fiona Campbell,
Claudia Kapp,
Alireza Khanteymoori,
Dawn Craig,
Mark Engelbert,
James Thomas
Abstract:
This paper describes a rapid feasibility study of using GPT-4, a large language model (LLM), to (semi)automate data extraction in systematic reviews. Despite the recent surge of interest in LLMs there is still a lack of understanding of how to design LLM-based automation tools and how to robustly evaluate their performance. During the 2023 Evidence Synthesis Hackathon we conducted two feasibility…
▽ More
This paper describes a rapid feasibility study of using GPT-4, a large language model (LLM), to (semi)automate data extraction in systematic reviews. Despite the recent surge of interest in LLMs there is still a lack of understanding of how to design LLM-based automation tools and how to robustly evaluate their performance. During the 2023 Evidence Synthesis Hackathon we conducted two feasibility studies. Firstly, to automatically extract study characteristics from human clinical, animal, and social science domain studies. We used two studies from each category for prompt-development; and ten for evaluation. Secondly, we used the LLM to predict Participants, Interventions, Controls and Outcomes (PICOs) labelled within 100 abstracts in the EBM-NLP dataset. Overall, results indicated an accuracy of around 80%, with some variability between domains (82% for human clinical, 80% for animal, and 72% for studies of human social sciences). Causal inference methods and study design were the data extraction items with the most errors. In the PICO study, participants and intervention/control showed high accuracy (>80%), outcomes were more challenging. Evaluation was done manually; scoring methods such as BLEU and ROUGE showed limited value. We observed variability in the LLMs predictions and changes in response quality. This paper presents a template for future evaluations of LLMs in the context of data extraction for systematic review automation. Our results show that there might be value in using LLMs, for example as second or third reviewers. However, caution is advised when integrating models such as GPT-4 into tools. Further research on stability and reliability in practical settings is warranted for each type of data that is processed by the LLM.
△ Less
Submitted 13 February, 2025; v1 submitted 23 May, 2024;
originally announced May 2024.
-
Generative AI in Computing Education: Perspectives of Students and Instructors
Authors:
Cynthia Zastudil,
Magdalena Rogalska,
Christine Kapp,
Jennifer Vaughn,
Stephen MacNeil
Abstract:
Generative models are now capable of producing natural language text that is, in some cases, comparable in quality to the text produced by people. In the computing education context, these models are being used to generate code, code explanations, and programming exercises. The rapid adoption of these models has prompted multiple position papers and workshops which discuss the implications of thes…
▽ More
Generative models are now capable of producing natural language text that is, in some cases, comparable in quality to the text produced by people. In the computing education context, these models are being used to generate code, code explanations, and programming exercises. The rapid adoption of these models has prompted multiple position papers and workshops which discuss the implications of these models for computing education, both positive and negative. This paper presents results from a series of semi-structured interviews with 12 students and 6 instructors about their awareness, experiences, and preferences regarding the use of tools powered by generative AI in computing classrooms. The results suggest that Generative AI (GAI) tools will play an increasingly significant role in computing education. However, students and instructors also raised numerous concerns about how these models should be integrated to best support the needs and learning goals of students. We also identified interesting tensions and alignments that emerged between how instructors and students prefer to engage with these models. We discuss these results and provide recommendations related to curriculum development, assessment methods, and pedagogical practice. As GAI tools become increasingly prevalent, it's important to understand educational stakeholders' preferences and values to ensure that these tools can be used for good and that potential harms can be mitigated.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
