-
How Much Data are Enough? Investigating Dataset Requirements for Patch-Based Brain MRI Segmentation Tasks
Authors:
Dongang Wang,
Peilin Liu,
Hengrui Wang,
Heidi Beadnall,
Kain Kyle,
Linda Ly,
Mariano Cabezas,
Geng Zhan,
Ryan Sullivan,
Weidong Cai,
Wanli Ouyang,
Fernando Calamante,
Michael Barnett,
Chenyu Wang
Abstract:
Training deep neural networks reliably requires access to large-scale datasets. However, obtaining such datasets can be challenging, especially in the context of neuroimaging analysis tasks, where the cost associated with image acquisition and annotation can be prohibitive. To mitigate both the time and financial costs associated with model development, a clear understanding of the amount of data…
▽ More
Training deep neural networks reliably requires access to large-scale datasets. However, obtaining such datasets can be challenging, especially in the context of neuroimaging analysis tasks, where the cost associated with image acquisition and annotation can be prohibitive. To mitigate both the time and financial costs associated with model development, a clear understanding of the amount of data required to train a satisfactory model is crucial. This paper focuses on an early stage phase of deep learning research, prior to model development, and proposes a strategic framework for estimating the amount of annotated data required to train patch-based segmentation networks. This framework includes the establishment of performance expectations using a novel Minor Boundary Adjustment for Threshold (MinBAT) method, and standardizing patch selection through the ROI-based Expanded Patch Selection (REPS) method. Our experiments demonstrate that tasks involving regions of interest (ROIs) with different sizes or shapes may yield variably acceptable Dice Similarity Coefficient (DSC) scores. By setting an acceptable DSC as the target, the required amount of training data can be estimated and even predicted as data accumulates. This approach could assist researchers and engineers in estimating the cost associated with data collection and annotation when defining a new segmentation task based on deep neural networks, ultimately contributing to their efficient translation to real-world applications.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
Improving Multiple Sclerosis Lesion Segmentation Across Clinical Sites: A Federated Learning Approach with Noise-Resilient Training
Authors:
Lei Bai,
Dongang Wang,
Michael Barnett,
Mariano Cabezas,
Weidong Cai,
Fernando Calamante,
Kain Kyle,
Dongnan Liu,
Linda Ly,
Aria Nguyen,
Chun-Chien Shieh,
Ryan Sullivan,
Hengrui Wang,
Geng Zhan,
Wanli Ouyang,
Chenyu Wang
Abstract:
Accurately measuring the evolution of Multiple Sclerosis (MS) with magnetic resonance imaging (MRI) critically informs understanding of disease progression and helps to direct therapeutic strategy. Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. Obtaining sufficient data from a single clin…
▽ More
Accurately measuring the evolution of Multiple Sclerosis (MS) with magnetic resonance imaging (MRI) critically informs understanding of disease progression and helps to direct therapeutic strategy. Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. Obtaining sufficient data from a single clinical site is challenging and does not address the heterogeneous need for model robustness. Conversely, the collection of data from multiple sites introduces data privacy concerns and potential label noise due to varying annotation standards. To address this dilemma, we explore the use of the federated learning framework while considering label noise. Our approach enables collaboration among multiple clinical sites without compromising data privacy under a federated learning paradigm that incorporates a noise-robust training strategy based on label correction. Specifically, we introduce a Decoupled Hard Label Correction (DHLC) strategy that considers the imbalanced distribution and fuzzy boundaries of MS lesions, enabling the correction of false annotations based on prediction confidence. We also introduce a Centrally Enhanced Label Correction (CELC) strategy, which leverages the aggregated central model as a correction teacher for all sites, enhancing the reliability of the correction process. Extensive experiments conducted on two multi-site datasets demonstrate the effectiveness and robustness of our proposed methods, indicating their potential for clinical applications in multi-site collaborations.
△ Less
Submitted 30 August, 2023;
originally announced August 2023.
-
Span Identification of Epistemic Stance-Taking in Academic Written English
Authors:
Masaki Eguchi,
Kristopher Kyle
Abstract:
Responding to the increasing need for automated writing evaluation (AWE) systems to assess language use beyond lexis and grammar (Burstein et al., 2016), we introduce a new approach to identify rhetorical features of stance in academic English writing. Drawing on the discourse-analytic framework of engagement in the Appraisal analysis (Martin & White, 2005), we manually annotated 4,688 sentences (…
▽ More
Responding to the increasing need for automated writing evaluation (AWE) systems to assess language use beyond lexis and grammar (Burstein et al., 2016), we introduce a new approach to identify rhetorical features of stance in academic English writing. Drawing on the discourse-analytic framework of engagement in the Appraisal analysis (Martin & White, 2005), we manually annotated 4,688 sentences (126,411 tokens) for eight rhetorical stance categories (e.g., PROCLAIM, ATTRIBUTION) and additional discourse elements. We then report an experiment to train machine learning models to identify and categorize the spans of these stance expressions. The best-performing model (RoBERTa + LSTM) achieved macro-averaged F1 of .7208 in the span identification of stance-taking expressions, slightly outperforming the intercoder reliability estimates before adjudication (F1 = .6629).
△ Less
Submitted 3 June, 2023;
originally announced June 2023.
-
Learning from pseudo-labels: deep networks improve consistency in longitudinal brain volume estimation
Authors:
Geng Zhan,
Dongang Wang,
Mariano Cabezas,
Lei Bai,
Kain Kyle,
Wanli Ouyang,
Michael Barnett,
Chenyu Wang
Abstract:
Brain atrophy is an important biomarker for monitoring neurodegeneration and disease progression in conditions such as multiple sclerosis (MS). An accurate and robust quantitative measurement of brain volume change is paramount for translational research and clinical applications. This paper presents a deep learning based method, DeepBVC, for longitudinal brain volume change measurement using 3D T…
▽ More
Brain atrophy is an important biomarker for monitoring neurodegeneration and disease progression in conditions such as multiple sclerosis (MS). An accurate and robust quantitative measurement of brain volume change is paramount for translational research and clinical applications. This paper presents a deep learning based method, DeepBVC, for longitudinal brain volume change measurement using 3D T1-weighted MRI scans. Trained with the intermediate outputs from SIENA, DeepBVC is designed to take into account the variance caused by different scanners and acquisition protocols. Compared with SIENA, DeepBVC demonstrates higher consistency in terms of volume change estimation across multiple time points in MS subjects; and greater stability and superior performance in scan-rescan experiments. Moreover, the results also show that DeepBVC is insensitive to acquisition variance in terms of imaging contrast, voxel resolution, random bias field and signal-to-noise ratio. Measurement robustness, automation and processing speed suggest a broad potential of DeepBVC in both research and clinical quantitative neuroimaging applications.
△ Less
Submitted 8 February, 2023;
originally announced February 2023.
-
MS Lesion Segmentation: Revisiting Weighting Mechanisms for Federated Learning
Authors:
Dongnan Liu,
Mariano Cabezas,
Dongang Wang,
Zihao Tang,
Lei Bai,
Geng Zhan,
Yuling Luo,
Kain Kyle,
Linda Ly,
James Yu,
Chun-Chien Shieh,
Aria Nguyen,
Ettikan Kandasamy Karuppiah,
Ryan Sullivan,
Fernando Calamante,
Michael Barnett,
Wanli Ouyang,
Weidong Cai,
Chenyu Wang
Abstract:
Federated learning (FL) has been widely employed for medical image analysis to facilitate multi-client collaborative learning without sharing raw data. Despite great success, FL's performance is limited for multiple sclerosis (MS) lesion segmentation tasks, due to variance in lesion characteristics imparted by different scanners and acquisition parameters. In this work, we propose the first FL MS…
▽ More
Federated learning (FL) has been widely employed for medical image analysis to facilitate multi-client collaborative learning without sharing raw data. Despite great success, FL's performance is limited for multiple sclerosis (MS) lesion segmentation tasks, due to variance in lesion characteristics imparted by different scanners and acquisition parameters. In this work, we propose the first FL MS lesion segmentation framework via two effective re-weighting mechanisms. Specifically, a learnable weight is assigned to each local node during the aggregation process, based on its segmentation performance. In addition, the segmentation loss function in each client is also re-weighted according to the lesion volume for the data during training. Comparison experiments on two FL MS segmentation scenarios using public and clinical datasets have demonstrated the effectiveness of the proposed method by outperforming other FL methods significantly. Furthermore, the segmentation performance of FL incorporating our proposed aggregation mechanism can exceed centralised training with all the raw data. The extensive evaluation also indicated the superiority of our method when estimating brain volume differences estimation after lesion inpainting.
△ Less
Submitted 3 May, 2022;
originally announced May 2022.
