Showing 1–10 of 10 results for author: Pellegrino, N

Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models

Authors: Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen K. Wong, Graham Wills, Elliot First, Miranda Schnier, Kyle Burton, Cris G. Ebby, Jillian Gorskic, Matthew Kalscheur, Samy Khalil, Marie Pisani, Tyler Rubeor, Peter Stetson, Frank Liao, Cherodeep Goswami, Brian Patterson, Majid Afshar

Abstract: As Large Language Models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrume… ▽ More As Large Language Models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. Multi-document summaries were generated from real-world EHR data across multiple specialties using several LLMs (GPT-4o, Mixtral 8x7b, and Llama 3-8b). Validation included Pearson correlation for substantive validity, factor analysis and Cronbach's alpha for structural validity, inter-rater reliability (ICC and Krippendorff's alpha) for generalizability, a semi-Delphi process for content validity, and comparisons of high-versus low-quality summaries for discriminant validity. Seven physician raters evaluated 779 summaries and answered 8,329 questions, achieving over 80% power for inter-rater reliability. The PDSQI-9 demonstrated strong internal consistency (Cronbach's alpha = 0.879; 95% CI: 0.867-0.891) and high inter-rater reliability (ICC = 0.867; 95% CI: 0.867-0.868), supporting structural validity and generalizability. Factor analysis identified a 4-factor model explaining 58% of the variance, representing organization, clarity, accuracy, and utility. Substantive validity was supported by correlations between note length and scores for Succinct (rho = -0.200, p = 0.029) and Organized ($ρ= -0.190$, $p = 0.037$). Discriminant validity distinguished high- from low-quality summaries ($p < 0.001$). The PDSQI-9 demonstrates robust construct validity, supporting its use in clinical practice to evaluate LLM-generated summaries and facilitate safer integration of LLMs into healthcare workflows. △ Less

Submitted 17 January, 2025; v1 submitted 15 January, 2025; originally announced January 2025.

Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review

Authors: Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen K. Wong, Graham Wills, Elliot First, Frank J. Liao, Cherodeep Goswami, Brian Patterson, Majid Afshar

Abstract: Large Language Models have advanced clinical Natural Language Generation, creating opportunities to manage the volume of medical text. However, the high-stakes nature of medicine requires reliable evaluation, which remains a challenge. In this narrative review, we assess the current evaluation state for clinical summarization tasks and propose future directions to address the resource constraints… ▽ More Large Language Models have advanced clinical Natural Language Generation, creating opportunities to manage the volume of medical text. However, the high-stakes nature of medicine requires reliable evaluation, which remains a challenge. In this narrative review, we assess the current evaluation state for clinical summarization tasks and propose future directions to address the resource constraints of expert human evaluation. △ Less

Submitted 26 September, 2024; originally announced September 2024.

Particle-Filtering-based Latent Diffusion for Inverse Problems

Authors: Amir Nazemi, Mohammad Hadi Sepanj, Nicholas Pellegrino, Chris Czarnecki, Paul Fieguth

Abstract: Current strategies for solving image-based inverse problems apply latent diffusion models to perform posterior sampling.However, almost all approaches make no explicit attempt to explore the solution space, instead drawing only a single sample from a Gaussian distribution from which to generate their solution. In this paper, we introduce a particle-filtering-based framework for a nonlinear explora… ▽ More Current strategies for solving image-based inverse problems apply latent diffusion models to perform posterior sampling.However, almost all approaches make no explicit attempt to explore the solution space, instead drawing only a single sample from a Gaussian distribution from which to generate their solution. In this paper, we introduce a particle-filtering-based framework for a nonlinear exploration of the solution space in the initial stages of reverse SDE methods. Our proposed particle-filtering-based latent diffusion (PFLD) method and proposed problem formulation and framework can be applied to any diffusion-based solution for linear or nonlinear inverse problems. Our experimental results show that PFLD outperforms the SoTA solver PSLD on the FFHQ-1K and ImageNet-1K datasets on inverse problem tasks of super resolution, Gaussian debluring and inpainting. △ Less

Submitted 25 August, 2024; originally announced August 2024.

Comments: Mohammad Hadi Sepanj, Nicholas Pellegrino, and Chris Czarnecki contributed equally

BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity

Authors: Zahra Gharaee, Scott C. Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Zarubiieva, Lila Kari, Dirk Steinke, Graham W. Taylor, Paul Fieguth, Angel X. Chang

Abstract: As part of an ongoing worldwide effort to comprehend and monitor insect biodiversity, this paper presents the BIOSCAN-5M Insect dataset to the machine learning community and establish several benchmark tasks. BIOSCAN-5M is a comprehensive dataset containing multi-modal information for over 5 million insect specimens, and it significantly expands existing image-based biological datasets by includin… ▽ More As part of an ongoing worldwide effort to comprehend and monitor insect biodiversity, this paper presents the BIOSCAN-5M Insect dataset to the machine learning community and establish several benchmark tasks. BIOSCAN-5M is a comprehensive dataset containing multi-modal information for over 5 million insect specimens, and it significantly expands existing image-based biological datasets by including taxonomic labels, raw nucleotide barcode sequences, assigned barcode index numbers, geographical, and size information. We propose three benchmark experiments to demonstrate the impact of the multi-modal data types on the classification and clustering accuracy. First, we pretrain a masked language model on the DNA barcode sequences of the BIOSCAN-5M dataset, and demonstrate the impact of using this large reference library on species- and genus-level classification performance. Second, we propose a zero-shot transfer learning task applied to images and DNA barcodes to cluster feature embeddings obtained from self-supervised learning, to investigate whether meaningful clusters can be derived from these representation embeddings. Third, we benchmark multi-modality by performing contrastive learning on DNA barcodes, image data, and taxonomic information. This yields a general shared embedding space enabling taxonomic classification using multiple types of information and modalities. The code repository of the BIOSCAN-5M Insect dataset is available at https://github.com/bioscan-ml/BIOSCAN-5M. △ Less

Submitted 28 February, 2025; v1 submitted 18 June, 2024; originally announced June 2024.

Journal ref: NeurIPS 2024

A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Dataset

Authors: Zahra Gharaee, ZeMing Gong, Nicholas Pellegrino, Iuliia Zarubiieva, Joakim Bruslund Haurum, Scott C. Lowe, Jaclyn T. A. McKeown, Chris C. Y. Ho, Joschka McLeod, Yi-Yun C Wei, Jireh Agda, Sujeevan Ratnasingham, Dirk Steinke, Angel X. Chang, Graham W. Taylor, Paul Fieguth

Abstract: In an effort to catalog insect biodiversity, we propose a new large dataset of hand-labelled insect images, the BIOSCAN-Insect Dataset. Each record is taxonomically classified by an expert, and also has associated genetic information including raw nucleotide barcode sequences and assigned barcode index numbers, which are genetically-based proxies for species classification. This paper presents a c… ▽ More In an effort to catalog insect biodiversity, we propose a new large dataset of hand-labelled insect images, the BIOSCAN-Insect Dataset. Each record is taxonomically classified by an expert, and also has associated genetic information including raw nucleotide barcode sequences and assigned barcode index numbers, which are genetically-based proxies for species classification. This paper presents a curated million-image dataset, primarily to train computer-vision models capable of providing image-based taxonomic assessment, however, the dataset also presents compelling characteristics, the study of which would be of interest to the broader machine learning community. Driven by the biological nature inherent to the dataset, a characteristic long-tailed class-imbalance distribution is exhibited. Furthermore, taxonomic labelling is a hierarchical classification scheme, presenting a highly fine-grained classification problem at lower levels. Beyond spurring interest in biodiversity research within the machine learning community, progress on creating an image-based taxonomic classifier will also further the ultimate goal of all BIOSCAN research: to lay the foundation for a comprehensive survey of global biodiversity. This paper introduces the dataset and explores the classification task through the implementation and analysis of a baseline classifier. △ Less

Submitted 13 November, 2023; v1 submitted 19 July, 2023; originally announced July 2023.

Machine Learning Challenges of Biological Factors in Insect Image Data

Authors: Nicholas Pellegrino, Zahra Gharaee, Paul Fieguth

Abstract: The BIOSCAN project, led by the International Barcode of Life Consortium, seeks to study changes in biodiversity on a global scale. One component of the project is focused on studying the species interaction and dynamics of all insects. In addition to genetically barcoding insects, over 1.5 million images per year will be collected, each needing taxonomic classification. With the immense volume of… ▽ More The BIOSCAN project, led by the International Barcode of Life Consortium, seeks to study changes in biodiversity on a global scale. One component of the project is focused on studying the species interaction and dynamics of all insects. In addition to genetically barcoding insects, over 1.5 million images per year will be collected, each needing taxonomic classification. With the immense volume of incoming images, relying solely on expert taxonomists to label the images would be impossible; however, artificial intelligence and computer vision technology may offer a viable high-throughput solution. Additional tasks including manually weighing individual insects to determine biomass, remain tedious and costly. Here again, computer vision may offer an efficient and compelling alternative. While the use of computer vision methods is appealing for addressing these problems, significant challenges resulting from biological factors present themselves. These challenges are formulated in the context of machine learning in this paper. △ Less

Submitted 4 November, 2022; originally announced November 2022.

Comments: 4 pages, 3 figures. Submitted to the Journal of Computational Vision and Imaging Systems

ACM Class: I.4.0; E.0; J.3

Time-domain feature extraction for target-specificity in Photoacoustic Remote Sensing Microscopy

Authors: Nicholas Pellegrino, Benjamin R. Ecclestone, Paul Fieguth, Parsin Haji Reza

Abstract: Photoacoustic Remote Sensing (PARS) microscopy is an emerging label-free optical absorption imaging modality. PARS operates by capturing nanosecond-scale optical perturbations generated by photoacoustic pressures. These time-domain (TD) modulations are usually projected by amplitude to determine absorption magnitude. However, significant information on the target's material properties is contained… ▽ More Photoacoustic Remote Sensing (PARS) microscopy is an emerging label-free optical absorption imaging modality. PARS operates by capturing nanosecond-scale optical perturbations generated by photoacoustic pressures. These time-domain (TD) modulations are usually projected by amplitude to determine absorption magnitude. However, significant information on the target's material properties is contained within the TD signals. This work proposes a novel clustering method to learn TD features which relate to underlying biomolecule characteristics. This technique identifies features related to constituent biomolecules, enabling single-acquisition virtual tissue labelling. Colorized visualizations of tissue are produced, highlighting specific tissue components. This is demonstrated on freshly resected murine brain tissue, clearly discerning structures including myelinated and unmyelinated neurons (white and gray matter) and nuclear structures. △ Less

Submitted 13 March, 2022; v1 submitted 8 March, 2022; originally announced March 2022.

Comments: 4 pages, 4 figures

K-Means for Noise-Insensitive Multi-Dimensional Feature Learning

Authors: Nicholas Pellegrino, Paul Fieguth, Parsin Haji Reza

Abstract: Many measurement modalities which perform imaging by probing an object pixel-by-pixel, such as via Photoacoustic Microscopy, produce a multi-dimensional feature (typically a time-domain signal) at each pixel. In principle, the many degrees of freedom in the time-domain signal would admit the possibility of significant multi-modal information being implicitly present, much more than a single scalar… ▽ More Many measurement modalities which perform imaging by probing an object pixel-by-pixel, such as via Photoacoustic Microscopy, produce a multi-dimensional feature (typically a time-domain signal) at each pixel. In principle, the many degrees of freedom in the time-domain signal would admit the possibility of significant multi-modal information being implicitly present, much more than a single scalar "brightness", regarding the underlying targets being observed. However, the measured signal is neither a weighted-sum of basis functions (such as principal components) nor one of a set of prototypes (K-means), which has motivated the novel clustering method proposed here. Signals are clustered based on their shape, but not amplitude, via angular distance and centroids are calculated as the direction of maximal intra-cluster variance, resulting in a clustering algorithm capable of learning centroids (signal shapes) that are related to the underlying, albeit unknown, target characteristics in a scalable and noise-robust manner. △ Less

Submitted 8 August, 2022; v1 submitted 15 February, 2022; originally announced February 2022.

Comments: Under consideration at Pattern Recognition Letters. 6 pages (excluding references), 5 figures

MSC Class: 68T10 ACM Class: I.5.3

In vivo functional and structural retina imaging using multimodal photoacoustic remote sensing microscopy and optical coherence tomography

Authors: Zohreh Hosseinaee, Nicholas Pellegrino, Nima Abbasi, Tara Amiri, James A. Tummon Simmons, Paul Fieguth, Parsin Haji Reza

Abstract: We have developed a multimodal photoacoustic remote sensing (PARS) microscope combined with swept source optical coherence tomography for in vivo, non-contact retinal imaging. Building on the proven strength of multiwavelength PARS imaging, the system is applied for estimating retinal oxygen saturation in the rat retina. The capability of the technology is demonstrated by imaging both microanatomy… ▽ More We have developed a multimodal photoacoustic remote sensing (PARS) microscope combined with swept source optical coherence tomography for in vivo, non-contact retinal imaging. Building on the proven strength of multiwavelength PARS imaging, the system is applied for estimating retinal oxygen saturation in the rat retina. The capability of the technology is demonstrated by imaging both microanatomy and the microvasculature of the retina in vivo. To our knowledge this is the first time a non-contact photoacoustic imaging technique is employed for in vivo oxygen saturation measurement in the retina. △ Less

Submitted 25 August, 2021; originally announced August 2021.

Non-contact, in-vivo, functional, and structural ophthalmic imaging using multimodal photoacoustic remote sensing (PARS) microscopy and optical coherence tomography (OCT)

Authors: Zohreh Hosseinaee, Nima Abbasi, Layla Khali, Lyazzat Mukhangaliyeva, Nicholas Pellegrino, Parsin Haji Reza

Abstract: Early diagnosis of ocular diseases improves the understanding of pathophysiology and helps with accurate monitoring and effective treatment. Advanced multimodal ocular imaging platforms play a crucial role in the visualization of the ocular components and provide clinicians with a valuable tool for evaluating different eye diseases. Here, for the first time, we present a non-contact, multimodal ph… ▽ More Early diagnosis of ocular diseases improves the understanding of pathophysiology and helps with accurate monitoring and effective treatment. Advanced multimodal ocular imaging platforms play a crucial role in the visualization of the ocular components and provide clinicians with a valuable tool for evaluating different eye diseases. Here, for the first time, we present a non-contact, multimodal photoacoustic remote sensing (PARS) microscopy and swept-source optical coherence tomography (SS-OCT) for in-vivo functional and structural imaging of the eye. The system provides complementary imaging contrasts of optical absorption and optical scattering and is used for non-contact, in-vivo imaging of the murine eye. Results of vasculature and structural imaging as well as melanin content in the retinal pigment epithelium (RPE) layer are presented. Multiwavelength PARS microscopy using Stimulated Raman Scattering (SRS) is applied for the first time, to provide non-contact oxygen saturation estimation in the ocular tissue. The reported work may be a major step toward clinical translation of ophthalmic technologies and has the potential to advance the diagnosis and treatment of ocular diseases. △ Less

Submitted 26 April, 2021; v1 submitted 17 February, 2021; originally announced February 2021.