-
Improving the repeatability of deep learning models with Monte Carlo dropout
Authors:
Andreanne Lemay,
Katharina Hoebel,
Christopher P. Bridge,
Brian Befano,
Silvia De Sanjosé,
Diden Egemen,
Ana Cecilia Rodriguez,
Mark Schiffman,
John Peter Campbell,
Jayashree Kalpathy-Cramer
Abstract:
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Repeatable models output predictions with low variation during independent tests carried out under similar conditions. During model development and evaluation, much attention is given to classification performance while model repeatability is…
▽ More
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Repeatable models output predictions with low variation during independent tests carried out under similar conditions. During model development and evaluation, much attention is given to classification performance while model repeatability is rarely assessed, leading to the development of models that are unusable in clinical practice. In this work, we evaluate the repeatability of four model types (binary classification, multi-class classification, ordinal classification, and regression) on images that were acquired from the same patient during the same visit. We study the performance of binary, multi-class, ordinal, and regression models on four medical image classification tasks from public and private datasets: knee osteoarthritis, cervical cancer screening, breast density estimation, and retinopathy of prematurity. Repeatability is measured and compared on ResNet and DenseNet architectures. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increased repeatability for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95\% limits of agreement by 16% points and of the disagreement rate by 7% points. The classification accuracy improved in most settings along with the repeatability. Our results suggest that beyond about 20 Monte Carlo iterations, there is no further gain in repeatability. In addition to the higher test-retest agreement, Monte Carlo predictions were better calibrated which leads to output probabilities reflecting more accurately the true likelihood of being correctly classified.
△ Less
Submitted 15 February, 2022;
originally announced February 2022.
-
Monte Carlo dropout increases model repeatability
Authors:
Andreanne Lemay,
Katharina Hoebel,
Christopher P. Bridge,
Didem Egemen,
Ana Cecilia Rodriguez,
Mark Schiffman,
John Peter Campbell,
Jayashree Kalpathy-Cramer
Abstract:
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Among the main features of robustness is repeatability. Much attention is given to classification performance without assessing the model repeatability, leading to the development of models that turn out to be unusable in practice. In this work, we evaluate the repeatability of four model types…
▽ More
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Among the main features of robustness is repeatability. Much attention is given to classification performance without assessing the model repeatability, leading to the development of models that turn out to be unusable in practice. In this work, we evaluate the repeatability of four model types on images from the same patient that were acquired during the same visit. We study the performance of binary, multi-class, ordinal, and regression models on three medical image analysis tasks: cervical cancer screening, breast density estimation, and retinopathy of prematurity classification. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increased repeatability for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95% limits of agreement by 17% points.
△ Less
Submitted 12 November, 2021;
originally announced November 2021.
-
Highly accurate closed-form approximation for the probability of detection of Weibull fluctuating targets in non-coherent detectors
Authors:
Fernando Darío Almeida García,
Andrea Carolina Flores Rodriguez,
Gustavo Fraidenraich
Abstract:
In this paper, we derive a highly accurate approximation for the probability of detection (PD) of a non-coherent detector operating with Weibull fluctuation targets. To do so, we assume a pulse-to-pulse decorrelation during the coherent processing interval (CPI). Specifically, the proposed approximation is given in terms of: i) a closed-form expression derived in terms of the Fox's H-function, for…
▽ More
In this paper, we derive a highly accurate approximation for the probability of detection (PD) of a non-coherent detector operating with Weibull fluctuation targets. To do so, we assume a pulse-to-pulse decorrelation during the coherent processing interval (CPI). Specifically, the proposed approximation is given in terms of: i) a closed-form expression derived in terms of the Fox's H-function, for which we also provide a portable and efficient MATHEMATICA routine; and ii) a fast converging series obtained through a comprehensive calculus of residues. Both solutions are fast and provide very accurate results. In particular, our series representation, besides being a more tractable solution, also exhibits impressive savings in computational load and computation time compared to previous studies. Numerical results and Monte-Carlo simulations corroborated the validity of our expressions.
△ Less
Submitted 12 December, 2020;
originally announced December 2020.
-
The p-AAA algorithm for data driven modeling of parametric dynamical systems
Authors:
Andrea Carracedo Rodriguez,
Linus Balicki,
Serkan Gugercin
Abstract:
The AAA algorithm has become a popular tool for data-driven rational approximation of single variable functions, such as transfer functions of a linear dynamical system. In the setting of parametric dynamical systems appearing in many prominent applications, the underlying (transfer) function to be modeled is a multivariate function. With this in mind, we develop the AAA framework for approximatin…
▽ More
The AAA algorithm has become a popular tool for data-driven rational approximation of single variable functions, such as transfer functions of a linear dynamical system. In the setting of parametric dynamical systems appearing in many prominent applications, the underlying (transfer) function to be modeled is a multivariate function. With this in mind, we develop the AAA framework for approximating multivariate functions where the approximant is constructed in the multivariate barycentric form. The method is data-driven, in the sense that it does not require access to full state-space model and requires only function evaluations. We discuss an extension to the case of matrix-valued functions, i.e., multi-input/multi-output dynamical systems, and provide a connection to the tangential interpolation theory. Several numerical examples illustrate the effectiveness of the proposed approach.
△ Less
Submitted 10 July, 2022; v1 submitted 13 March, 2020;
originally announced March 2020.
-
Interpolatory Model Reduction of Parameterized Bilinear Dynamical Systems
Authors:
Andrea Carracedo Rodriguez,
Serkan Gugercin,
Jeff Borggaard
Abstract:
Interpolatory projection methods for model reduction of nonparametric linear dynamical systems have been successfully extended to nonparametric bilinear dynamical systems. However, this is not the case for parametric bilinear systems. In this work, we aim to close this gap by providing a natural extension of interpolatory projections to model reduction of parametric bilinear dynamical systems. We…
▽ More
Interpolatory projection methods for model reduction of nonparametric linear dynamical systems have been successfully extended to nonparametric bilinear dynamical systems. However, this is not the case for parametric bilinear systems. In this work, we aim to close this gap by providing a natural extension of interpolatory projections to model reduction of parametric bilinear dynamical systems. We introduce necessary conditions that the projection subspaces must satisfy to obtain parametric tangential interpolation of each subsystem transfer function. These conditions also guarantee that the parameter sensitivities (Jacobian) of each subsystem transfer function is matched tangentially by those of the corresponding reduced order model transfer function. Similarly, we obtain conditions for interpolating the parameter Hessian of the transfer function by including extra vectors in the projection subspaces. As in the parametric linear case, the basis construction for two-sided projections does not require computing the Jacobian or the Hessian.
△ Less
Submitted 19 December, 2017;
originally announced December 2017.
