Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study
Authors:
Lena Schmidt,
Kaitlyn Hair,
Sergio Graziosi,
Fiona Campbell,
Claudia Kapp,
Alireza Khanteymoori,
Dawn Craig,
Mark Engelbert,
James Thomas
Abstract:
This paper describes a rapid feasibility study of using GPT-4, a large language model (LLM), to (semi)automate data extraction in systematic reviews. Despite the recent surge of interest in LLMs there is still a lack of understanding of how to design LLM-based automation tools and how to robustly evaluate their performance. During the 2023 Evidence Synthesis Hackathon we conducted two feasibility…
▽ More
This paper describes a rapid feasibility study of using GPT-4, a large language model (LLM), to (semi)automate data extraction in systematic reviews. Despite the recent surge of interest in LLMs there is still a lack of understanding of how to design LLM-based automation tools and how to robustly evaluate their performance. During the 2023 Evidence Synthesis Hackathon we conducted two feasibility studies. Firstly, to automatically extract study characteristics from human clinical, animal, and social science domain studies. We used two studies from each category for prompt-development; and ten for evaluation. Secondly, we used the LLM to predict Participants, Interventions, Controls and Outcomes (PICOs) labelled within 100 abstracts in the EBM-NLP dataset. Overall, results indicated an accuracy of around 80%, with some variability between domains (82% for human clinical, 80% for animal, and 72% for studies of human social sciences). Causal inference methods and study design were the data extraction items with the most errors. In the PICO study, participants and intervention/control showed high accuracy (>80%), outcomes were more challenging. Evaluation was done manually; scoring methods such as BLEU and ROUGE showed limited value. We observed variability in the LLMs predictions and changes in response quality. This paper presents a template for future evaluations of LLMs in the context of data extraction for systematic review automation. Our results show that there might be value in using LLMs, for example as second or third reviewers. However, caution is advised when integrating models such as GPT-4 into tools. Further research on stability and reliability in practical settings is warranted for each type of data that is processed by the LLM.
△ Less
Submitted 13 February, 2025; v1 submitted 23 May, 2024;
originally announced May 2024.
A Review and Outlook for the Removal of Radon-Generated Po-210 Surface Contamination
Authors:
V. E. Guiseppe,
C. D. Christofferson,
K. R. Hair,
F. M. Adams
Abstract:
The next generation low-background detectors operating deep underground aim for unprecedented low levels of radioactive backgrounds. The deposition and presence of radon progeny on detector surfaces is an added source of energetic background events. In addition to limiting the detector material's radon exposure in order to reduce potential surface backgrounds, it is just as important to clean surf…
▽ More
The next generation low-background detectors operating deep underground aim for unprecedented low levels of radioactive backgrounds. The deposition and presence of radon progeny on detector surfaces is an added source of energetic background events. In addition to limiting the detector material's radon exposure in order to reduce potential surface backgrounds, it is just as important to clean surfaces to remove inevitable contamination. Such studies of radon progeny removal have generally found that a form of etching is effective at removing some of the progeny (Bi and Pb), however more aggressive techniques, including electropolishing, have been shown to effectively remove the Po atoms. In the absence of an aggressive etch, a significant fraction of the Po atoms are believed to either remain behind within the surface or redeposit from the etching solution back onto the surface. We explore the chemical nature of the aqueous Po ions and the effect of the oxidation state of Po to maximize the Po ions remaining in the etching solution of contaminated Cu surfaces. We present a review of the previous studies of surface radon progeny removal and our findings on the role of oxidizing agents and a cell potential in the preparation of a clean etching technique.
△ Less
Submitted 21 December, 2017;
originally announced December 2017.
