-
SurvSurf: a partially monotonic neural network for first-hitting time prediction of intermittently observed discrete and continuous sequential events
Authors:
Yichen Kelly Chen,
Sören Dittmer,
Kinga Bernatowicz,
Josep Arús-Pous,
Kamen Bliznashki,
John Aston,
James H. F. Rudd,
Carola-Bibiane Schönlieb,
James Jones,
Michael Roberts
Abstract:
We propose a neural-network based survival model (SurvSurf) specifically designed for direct and simultaneous probabilistic prediction of the first hitting time of sequential events from baseline. Unlike existing models, SurvSurf is theoretically guaranteed to never violate the monotonic relationship between the cumulative incidence functions of sequential events, while allowing nonlinear influenc…
▽ More
We propose a neural-network based survival model (SurvSurf) specifically designed for direct and simultaneous probabilistic prediction of the first hitting time of sequential events from baseline. Unlike existing models, SurvSurf is theoretically guaranteed to never violate the monotonic relationship between the cumulative incidence functions of sequential events, while allowing nonlinear influence from predictors. It also incorporates implicit truths for unobserved intermediate events in model fitting, and supports both discrete and continuous time and events. We also identified a variant of the Integrated Brier Score (IBS) that showed robust correlation with the mean squared error (MSE) between the true and predicted probabilities by accounting for implied truths about the missing intermediate events. We demonstrated the superiority of SurvSurf compared to modern and traditional predictive survival models in two simulated datasets and two real-world datasets, using MSE, the more robust IBS and by measuring the extent of monotonicity violation.
△ Less
Submitted 7 April, 2025;
originally announced April 2025.
-
Parameter choices in HaarPSI for IQA with medical images
Authors:
Clemens Karner,
Janek Gröhl,
Ian Selby,
Judith Babar,
Jake Beckford,
Thomas R Else,
Timothy J Sadler,
Shahab Shahipasand,
Arthikkaa Thavakumar,
Michael Roberts,
James H. F. Rudd,
Carola-Bibiane Schönlieb,
Jonathan R Weir-McCall,
Anna Breger
Abstract:
When developing machine learning models, image quality assessment (IQA) measures are a crucial component for the evaluation of obtained output images. However, commonly used full-reference IQA (FR-IQA) measures have been primarily developed and optimized for natural images. In many specialized settings, such as medical images, this poses an often overlooked problem regarding suitability. In previo…
▽ More
When developing machine learning models, image quality assessment (IQA) measures are a crucial component for the evaluation of obtained output images. However, commonly used full-reference IQA (FR-IQA) measures have been primarily developed and optimized for natural images. In many specialized settings, such as medical images, this poses an often overlooked problem regarding suitability. In previous studies, the FR-IQA measure HaarPSI showed promising behavior regarding generalizability. The measure is based on Haar wavelet representations and the framework allows optimization of two parameters. So far, these parameters have been aligned for natural images. Here, we optimize these parameters for two medical image data sets, a photoacoustic and a chest X-ray data set, with IQA expert ratings. We observe that they lead to similar parameter values, different to the natural image data, and are more sensitive to parameter changes. We denote the novel optimized setting as HaarPSI$_{MED}$, which improves the performance of the employed medical images significantly (p<0.05). Additionally, we include an independent CT test data set that illustrates the generalizability of HaarPSI$_{MED}$, as well as visual examples that qualitatively demonstrate the improvement. The results suggest that adapting common IQA measures within their frameworks for medical images can provide a valuable, generalizable addition to employment of more specific task-based measures.
△ Less
Submitted 8 April, 2025; v1 submitted 31 October, 2024;
originally announced October 2024.
-
The curious case of the test set AUROC
Authors:
Michael Roberts,
Alon Hazan,
Sören Dittmer,
James H. F. Rudd,
Carola-Bibiane Schönlieb
Abstract:
Whilst the size and complexity of ML models have rapidly and significantly increased over the past decade, the methods for assessing their performance have not kept pace. In particular, among the many potential performance metrics, the ML community stubbornly continues to use (a) the area under the receiver operating characteristic curve (AUROC) for a validation and test cohort (distinct from trai…
▽ More
Whilst the size and complexity of ML models have rapidly and significantly increased over the past decade, the methods for assessing their performance have not kept pace. In particular, among the many potential performance metrics, the ML community stubbornly continues to use (a) the area under the receiver operating characteristic curve (AUROC) for a validation and test cohort (distinct from training data) or (b) the sensitivity and specificity for the test data at an optimal threshold determined from the validation ROC. However, we argue that considering scores derived from the test ROC curve alone gives only a narrow insight into how a model performs and its ability to generalise.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
Recent Methodological Advances in Federated Learning for Healthcare
Authors:
Fan Zhang,
Daniel Kreuter,
Yichen Chen,
Sören Dittmer,
Samuel Tull,
Tolou Shadbahr,
BloodCounts! Collaboration,
Jacobus Preller,
James H. F. Rudd,
John A. D. Aston,
Carola-Bibiane Schönlieb,
Nicholas Gleadall,
Michael Roberts
Abstract:
For healthcare datasets, it is often not possible to combine data samples from multiple sites due to ethical, privacy or logistical concerns. Federated learning allows for the utilisation of powerful machine learning algorithms without requiring the pooling of data. Healthcare data has many simultaneous challenges which require new methodologies to address, such as highly-siloed data, class imbala…
▽ More
For healthcare datasets, it is often not possible to combine data samples from multiple sites due to ethical, privacy or logistical concerns. Federated learning allows for the utilisation of powerful machine learning algorithms without requiring the pooling of data. Healthcare data has many simultaneous challenges which require new methodologies to address, such as highly-siloed data, class imbalance, missing data, distribution shifts and non-standardised variables. Federated learning adds significant methodological complexity to conventional centralised machine learning, requiring distributed optimisation, communication between nodes, aggregation of models and redistribution of models. In this systematic review, we consider all papers on Scopus that were published between January 2015 and February 2023 and which describe new federated learning methodologies for addressing challenges with healthcare data. We performed a detailed review of the 89 papers which fulfilled these criteria. Significant systemic issues were identified throughout the literature which compromise the methodologies in many of the papers reviewed. We give detailed recommendations to help improve the quality of the methodology development for federated learning in healthcare.
△ Less
Submitted 4 October, 2023;
originally announced October 2023.
-
Reinterpreting survival analysis in the universal approximator age
Authors:
Sören Dittmer,
Michael Roberts,
Jacobus Preller,
AIX COVNET,
James H. F. Rudd,
John A. D. Aston,
Carola-Bibiane Schönlieb
Abstract:
Survival analysis is an integral part of the statistical toolbox. However, while most domains of classical statistics have embraced deep learning, survival analysis only recently gained some minor attention from the deep learning community. This recent development is likely in part motivated by the COVID-19 pandemic. We aim to provide the tools needed to fully harness the potential of survival ana…
▽ More
Survival analysis is an integral part of the statistical toolbox. However, while most domains of classical statistics have embraced deep learning, survival analysis only recently gained some minor attention from the deep learning community. This recent development is likely in part motivated by the COVID-19 pandemic. We aim to provide the tools needed to fully harness the potential of survival analysis in deep learning. On the one hand, we discuss how survival analysis connects to classification and regression. On the other hand, we provide technical tools. We provide a new loss function, evaluation metrics, and the first universal approximating network that provably produces survival curves without numeric integration. We show that the loss function and model outperform other approaches using a large numerical study.
△ Less
Submitted 25 July, 2023;
originally announced July 2023.
-
Dis-AE: Multi-domain & Multi-task Generalisation on Real-World Clinical Data
Authors:
Daniel Kreuter,
Samuel Tull,
Julian Gilbey,
Jacobus Preller,
BloodCounts! Consortium,
John A. D. Aston,
James H. F. Rudd,
Suthesh Sivapalaratnam,
Carola-Bibiane Schönlieb,
Nicholas Gleadall,
Michael Roberts
Abstract:
Clinical data is often affected by clinically irrelevant factors such as discrepancies between measurement devices or differing processing methods between sites. In the field of machine learning (ML), these factors are known as domains and the distribution differences they cause in the data are known as domain shifts. ML models trained using data from one domain often perform poorly when applied t…
▽ More
Clinical data is often affected by clinically irrelevant factors such as discrepancies between measurement devices or differing processing methods between sites. In the field of machine learning (ML), these factors are known as domains and the distribution differences they cause in the data are known as domain shifts. ML models trained using data from one domain often perform poorly when applied to data from another domain, potentially leading to wrong predictions. As such, developing machine learning models that can generalise well across multiple domains is a challenging yet essential task in the successful application of ML in clinical practice. In this paper, we propose a novel disentangled autoencoder (Dis-AE) neural network architecture that can learn domain-invariant data representations for multi-label classification of medical measurements even when the data is influenced by multiple interacting domain shifts at once. The model utilises adversarial training to produce data representations from which the domain can no longer be determined. We evaluate the model's domain generalisation capabilities on synthetic datasets and full blood count (FBC) data from blood donors as well as primary and secondary care patients, showing that Dis-AE improves model generalisation on multiple domains simultaneously while preserving clinically relevant information.
△ Less
Submitted 15 June, 2023;
originally announced June 2023.
-
Navigating the challenges in creating complex data systems: a development philosophy
Authors:
Sören Dittmer,
Michael Roberts,
Julian Gilbey,
Ander Biguri,
AIX-COVNET Collaboration,
Jacobus Preller,
James H. F. Rudd,
John A. D. Aston,
Carola-Bibiane Schönlieb
Abstract:
In this perspective, we argue that despite the democratization of powerful tools for data science and machine learning over the last decade, developing the code for a trustworthy and effective data science system (DSS) is getting harder. Perverse incentives and a lack of widespread software engineering (SE) skills are among many root causes we identify that naturally give rise to the current syste…
▽ More
In this perspective, we argue that despite the democratization of powerful tools for data science and machine learning over the last decade, developing the code for a trustworthy and effective data science system (DSS) is getting harder. Perverse incentives and a lack of widespread software engineering (SE) skills are among many root causes we identify that naturally give rise to the current systemic crisis in reproducibility of DSSs. We analyze why SE and building large complex systems is, in general, hard. Based on these insights, we identify how SE addresses those difficulties and how we can apply and generalize SE methods to construct DSSs that are fit for purpose. We advocate two key development philosophies, namely that one should incrementally grow -- not biphasically plan and build -- DSSs, and one should always employ two types of feedback loops during development: one which tests the code's correctness and another that evaluates the code's efficacy.
△ Less
Submitted 21 October, 2022;
originally announced October 2022.
-
Classification of datasets with imputed missing values: does imputation quality matter?
Authors:
Tolou Shadbahr,
Michael Roberts,
Jan Stanczuk,
Julian Gilbey,
Philip Teare,
Sören Dittmer,
Matthew Thorpe,
Ramon Vinas Torne,
Evis Sala,
Pietro Lio,
Mishal Patel,
AIX-COVNET Collaboration,
James H. F. Rudd,
Tuomas Mirtti,
Antti Rannikko,
John A. D. Aston,
Jing Tang,
Carola-Bibiane Schönlieb
Abstract:
Classifying samples in incomplete datasets is a common aim for machine learning practitioners, but is non-trivial. Missing data is found in most real-world datasets and these missing values are typically imputed using established methods, followed by classification of the now complete, imputed, samples. The focus of the machine learning researcher is then to optimise the downstream classification…
▽ More
Classifying samples in incomplete datasets is a common aim for machine learning practitioners, but is non-trivial. Missing data is found in most real-world datasets and these missing values are typically imputed using established methods, followed by classification of the now complete, imputed, samples. The focus of the machine learning researcher is then to optimise the downstream classification performance. In this study, we highlight that it is imperative to consider the quality of the imputation. We demonstrate how the commonly used measures for assessing quality are flawed and propose a new class of discrepancy scores which focus on how well the method recreates the overall distribution of the data. To conclude, we highlight the compromised interpretability of classifier models trained using poorly imputed data.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Authors:
Michael Roberts,
Derek Driggs,
Matthew Thorpe,
Julian Gilbey,
Michael Yeung,
Stephan Ursprung,
Angelica I. Aviles-Rivero,
Christian Etmann,
Cathal McCague,
Lucian Beer,
Jonathan R. Weir-McCall,
Zhongzhao Teng,
Effrossyni Gkrania-Klotsas,
James H. F. Rudd,
Evis Sala,
Carola-Bibiane Schönlieb
Abstract:
Machine learning methods offer great promise for fast and accurate detection and prognostication of COVID-19 from standard-of-care chest radiographs (CXR) and computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we search…
▽ More
Machine learning methods offer great promise for fast and accurate detection and prognostication of COVID-19 from standard-of-care chest radiographs (CXR) and computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we search EMBASE via OVID, MEDLINE via PubMed, bioRxiv, medRxiv and arXiv for published papers and preprints uploaded from January 1, 2020 to October 3, 2020 which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 61 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher quality model development and well documented manuscripts.
△ Less
Submitted 5 January, 2021; v1 submitted 14 August, 2020;
originally announced August 2020.
