-
Ensuring Reliability of Curated EHR-Derived Data: The Validation of Accuracy for LLM/ML-Extracted Information and Data (VALID) Framework
Authors:
Melissa Estevez,
Nisha Singh,
Lauren Dyson,
Blythe Adamson,
Qianyu Yuan,
Megan W. Hildner,
Erin Fidyk,
Olive Mbah,
Farhad Khan,
Kathi Seidl-Rathkopf,
Aaron B. Cohen
Abstract:
Large language models (LLMs) are increasingly used to extract clinical data from electronic health records (EHRs), offering significant improvements in scalability and efficiency for real-world data (RWD) curation in oncology. However, the adoption of LLMs introduces new challenges in ensuring the reliability, accuracy, and fairness of extracted data, which are essential for research, regulatory,…
▽ More
Large language models (LLMs) are increasingly used to extract clinical data from electronic health records (EHRs), offering significant improvements in scalability and efficiency for real-world data (RWD) curation in oncology. However, the adoption of LLMs introduces new challenges in ensuring the reliability, accuracy, and fairness of extracted data, which are essential for research, regulatory, and clinical applications. Existing quality assurance frameworks for RWD and artificial intelligence do not fully address the unique error modes and complexities associated with LLM-extracted data. In this paper, we propose a comprehensive framework for evaluating the quality of clinical data extracted by LLMs. The framework integrates variable-level performance benchmarking against expert human abstraction, automated verification checks for internal consistency and plausibility, and replication analyses comparing LLM-extracted data to human-abstracted datasets or external standards. This multidimensional approach enables the identification of variables most in need of improvement, systematic detection of latent errors, and confirmation of dataset fitness-for-purpose in real-world research. Additionally, the framework supports bias assessment by stratifying metrics across demographic subgroups. By providing a rigorous and transparent method for assessing LLM-extracted RWD, this framework advances industry standards and supports the trustworthy use of AI-powered evidence generation in oncology research and practice.
△ Less
Submitted 9 June, 2025;
originally announced June 2025.
-
Enabling Communication and Control Co-Design in 6G Networks
Authors:
Onur Ayan,
Nikolaos Pappas,
Miguel Angel Gutierrez Estevez,
Xueli An,
Wolfgang Kellerer
Abstract:
Networked control systems (NCSs), which are feedback control loops closed over a communication network, have been a popular research topic over the past decades. Numerous works in the literature propose novel algorithms and protocols with joint consideration of communication and control. However, the vast majority of the recent research results, which have shown remarkable performance improvements…
▽ More
Networked control systems (NCSs), which are feedback control loops closed over a communication network, have been a popular research topic over the past decades. Numerous works in the literature propose novel algorithms and protocols with joint consideration of communication and control. However, the vast majority of the recent research results, which have shown remarkable performance improvements if a cross-layer methodology is followed, have not been widely adopted by the industry. In this work, we review the shortcomings of today's mobile networks that render cross-layer solutions, such as semantic and goal-oriented communications, very challenging in practice. To tackle this, we propose a new framework for 6G user plane design that simplifies the adoption of recent research results in networked control, thereby facilitating the joint communication and control design in next-generation mobile networks.
△ Less
Submitted 26 February, 2024;
originally announced February 2024.
-
Study on the Fairness of Speaker Verification Systems on Underrepresented Accents in English
Authors:
Mariel Estevez,
Luciana Ferrer
Abstract:
Speaker verification (SV) systems are currently being used to make sensitive decisions like giving access to bank accounts or deciding whether the voice of a suspect coincides with that of the perpetrator of a crime. Ensuring that these systems are fair and do not disfavor any particular group is crucial. In this work, we analyze the performance of several state-of-the-art SV systems across groups…
▽ More
Speaker verification (SV) systems are currently being used to make sensitive decisions like giving access to bank accounts or deciding whether the voice of a suspect coincides with that of the perpetrator of a crime. Ensuring that these systems are fair and do not disfavor any particular group is crucial. In this work, we analyze the performance of several state-of-the-art SV systems across groups defined by the accent of the speakers when speaking English. To this end, we curated a new dataset based on the VoxCeleb corpus where we carefully selected samples from speakers with accents from different countries. We use this dataset to evaluate system performance for several SV systems trained with VoxCeleb data. We show that, while discrimination performance is reasonably robust across accent groups, calibration performance degrades dramatically on some accents that are not well represented in the training data. Finally, we show that a simple data balancing approach mitigates this undesirable bias, being particularly effective when applied to our recently-proposed discriminative condition-aware backend.
△ Less
Submitted 26 April, 2022;
originally announced April 2022.
-
Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research
Authors:
Benjamin Birnbaum,
Nathan Nussbaum,
Katharina Seidl-Rathkopf,
Monica Agrawal,
Melissa Estevez,
Evan Estola,
Joshua Haimson,
Lucy He,
Peter Larson,
Paul Richardson
Abstract:
Objective Electronic health records (EHRs) are a promising source of data for health outcomes research in oncology. A challenge in using EHR data is that selecting cohorts of patients often requires information in unstructured parts of the record. Machine learning has been used to address this, but even high-performing algorithms may select patients in a non-random manner and bias the resulting co…
▽ More
Objective Electronic health records (EHRs) are a promising source of data for health outcomes research in oncology. A challenge in using EHR data is that selecting cohorts of patients often requires information in unstructured parts of the record. Machine learning has been used to address this, but even high-performing algorithms may select patients in a non-random manner and bias the resulting cohort. To improve the efficiency of cohort selection while measuring potential bias, we introduce a technique called Model-Assisted Cohort Selection (MACS) with Bias Analysis and apply it to the selection of metastatic breast cancer (mBC) patients. Materials and Methods We trained a model on 17,263 patients using term-frequency inverse-document-frequency (TF-IDF) and logistic regression. We used a test set of 17,292 patients to measure algorithm performance and perform Bias Analysis. We compared the cohort generated by MACS to the cohort that would have been generated without MACS as reference standard, first by comparing distributions of an extensive set of clinical and demographic variables and then by comparing the results of two analyses addressing existing example research questions. Results Our algorithm had an area under the curve (AUC) of 0.976, a sensitivity of 96.0%, and an abstraction efficiency gain of 77.9%. During Bias Analysis, we found no large differences in baseline characteristics and no differences in the example analyses. Conclusion MACS with bias analysis can significantly improve the efficiency of cohort selection on EHR data while instilling confidence that outcomes research performed on the resulting cohort will not be biased.
△ Less
Submitted 13 January, 2020;
originally announced January 2020.
