Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset
Authors:
Shruti Hegde,
Mabon Manoj Ninan,
Jonathan R. Dillman,
Shireen Hayatghaibi,
Lynn Babcock,
Elanchezhian Somasundaram
Abstract:
General-purpose clinical natural language processing (NLP) tools are increasingly used for the automatic labeling of clinical reports. However, independent evaluations for specific tasks, such as pediatric chest radiograph (CXR) report labeling, are limited. This study compares four commercial clinical NLP systems - Amazon Comprehend Medical (AWS), Google Healthcare NLP (GC), Azure Clinical NLP (A…
▽ More
General-purpose clinical natural language processing (NLP) tools are increasingly used for the automatic labeling of clinical reports. However, independent evaluations for specific tasks, such as pediatric chest radiograph (CXR) report labeling, are limited. This study compares four commercial clinical NLP systems - Amazon Comprehend Medical (AWS), Google Healthcare NLP (GC), Azure Clinical NLP (AZ), and SparkNLP (SP) - for entity extraction and assertion detection in pediatric CXR reports. Additionally, CheXpert and CheXbert, two dedicated chest radiograph report labelers, were evaluated on the same task using CheXpert-defined labels. We analyzed 95,008 pediatric CXR reports from a large academic pediatric hospital. Entities and assertion statuses (positive, negative, uncertain) from the findings and impression sections were extracted by the NLP systems, with impression section entities mapped to 12 disease categories and a No Findings category. CheXpert and CheXbert extracted the same 13 categories. Outputs were compared using Fleiss Kappa and accuracy against a consensus pseudo-ground truth. Significant differences were found in the number of extracted entities and assertion distributions across NLP systems. SP extracted 49,688 unique entities, GC 16,477, AZ 31,543, and AWS 27,216. Assertion accuracy across models averaged around 62%, with SP highest (76%) and AWS lowest (50%). CheXpert and CheXbert achieved 56% accuracy. Considerable variability in performance highlights the need for careful validation and review before deploying NLP tools for clinical report labeling.
△ Less
Submitted 28 May, 2025;
originally announced May 2025.
Robotized polarization characterization platform for free-space quantum communication optics
Authors:
Youn Seok Lee,
Kimia Mohammadi,
Lindsay Babcock,
Brendon L. Higgins,
Hugh Podmore,
Thomas Jennewein
Abstract:
We develop a polarization characterization platform for optical devices in free-space quantum communications. We demonstrate an imaging polarimeter, which analyzes both incident polarization states and the angle of incidence, attached to a six-axis collaborative robot arm, enabling polarization characterization at any position and direction with consistent precision. We present a detailed descript…
▽ More
We develop a polarization characterization platform for optical devices in free-space quantum communications. We demonstrate an imaging polarimeter, which analyzes both incident polarization states and the angle of incidence, attached to a six-axis collaborative robot arm, enabling polarization characterization at any position and direction with consistent precision. We present a detailed description of each subsystem including the calibration and polarization-test procedure, and analyze polarization-measurement errors caused by imperfect orientations of the robot arm using a Mueller-matrix model of polarimeters at tilt incidence. We perform a proof-of-principle experiment for an angle-dependent polarization test for a commercial silver-coated mirror for which the polarization states of the reflected light can be accurately calculated. Quantitative agreement between the theory and experiment validates our methodology. We demonstrate the polarization test for a 20.3 cm lens designed for a quantum optical transmitter in Canada's Quantum Encryption and Science Satellite (QEYSSat) mission.
△ Less
Submitted 12 April, 2022; v1 submitted 4 September, 2021;
originally announced September 2021.
