Assessing the Quality of AI-Generated Clinical Notes: A Validated Evaluation of a Large Language Model Scribe
Authors:
Erin Palm,
Astrit Manikantan,
Mark E. Pepin,
Herprit Mahal,
Srikanth Subramanya Belwadi
Abstract:
In medical practices across the United States, physicians have begun implementing generative artificial intelligence (AI) tools to perform the function of scribes in order to reduce the burden of documenting clinical encounters. Despite their widespread use, no established methods exist to gauge the quality of AI scribes. To address this gap, we developed a blinded study comparing the relative per…
▽ More
In medical practices across the United States, physicians have begun implementing generative artificial intelligence (AI) tools to perform the function of scribes in order to reduce the burden of documenting clinical encounters. Despite their widespread use, no established methods exist to gauge the quality of AI scribes. To address this gap, we developed a blinded study comparing the relative performance of large language model (LLM) generated clinical notes with those from field experts based on audio-recorded clinical encounters. Quantitative metrics from the Physician Documentation Quality Instrument (PDQI9) provided a framework to measure note quality, which we adapted to assess relative performance of AI generated notes. Clinical experts spanning 5 medical specialties used the PDQI9 tool to evaluate specialist-drafted Gold notes and LLM authored Ambient notes. Two evaluators from each specialty scored notes drafted from a total of 97 patient visits. We found uniformly high inter rater agreement (RWG greater than 0.7) between evaluators in general medicine, orthopedics, and obstetrics and gynecology, and moderate (RWG 0.5 to 0.7) to high inter rater agreement in pediatrics and cardiology. We found a modest yet significant difference in the overall note quality, wherein Gold notes achieved a score of 4.25 out of 5 and Ambient notes scored 4.20 out of 5 (p = 0.04). Our findings support the use of the PDQI9 instrument as a practical method to gauge the quality of LLM authored notes, as compared to human-authored notes.
△ Less
Submitted 15 May, 2025;
originally announced May 2025.
A Practical System Architecture for Contract Automation: Design and Uses
Authors:
Emanuel Palm,
Ulf Bodin,
Olov Schelén
Abstract:
While the blockchain-based smart contract has become a hot topic of research over the last decade, not the least in the context of Industry 4.0, it now has well-known legal and technical shortcomings that currently prohibit its real-world application. These shortcomings come from (1) that a smart contract is a computer program, not a document describing legal obligations, and (2) that blockchain-b…
▽ More
While the blockchain-based smart contract has become a hot topic of research over the last decade, not the least in the context of Industry 4.0, it now has well-known legal and technical shortcomings that currently prohibit its real-world application. These shortcomings come from (1) that a smart contract is a computer program, not a document describing legal obligations, and (2) that blockchain-based systems are complicated to use and operate. In this paper, we present a refined and extended summary of our work taking key technologies from the blockchain sphere and applying them to the ricardian contract, which is a traditional contract in digital form with machine-readable parameters. By putting the ricardian contract in the context of our contract network architecture, we facilitate the infrastructure required for contracts to be offered, negotiated, performed, renegotiated and terminated in a completely digital and automatable fashion. Our architecture circumvents the legal issues of blockchains by facilitating an artifact very much alike a traditional contract, as well as its operational complexity by requiring consensus only between nodes representing directly involved parties. To demonstrate its utility, we also present how it could be used for (1) private data purchasing, (2) treasury management, (3) order-driven manufacturing and (4) automated device on-boarding.
△ Less
Submitted 12 August, 2024;
originally announced August 2024.
