-
A Graphical Comparison of Screening Designs using Support Recovery Probabilities
Authors:
Kade Young,
Maria L. Weese,
Jonathan W. Stallrich,
Byran J. Smucker,
David J. Edwards
Abstract:
A screening experiment attempts to identify a subset of important effects using a relatively small number of experimental runs. Given the limited run size and a large number of possible effects, penalized regression is a popular tool used to analyze screening designs. In particular, an automated implementation of the Gauss-Dantzig selector has been widely recommended to compare screening design co…
▽ More
A screening experiment attempts to identify a subset of important effects using a relatively small number of experimental runs. Given the limited run size and a large number of possible effects, penalized regression is a popular tool used to analyze screening designs. In particular, an automated implementation of the Gauss-Dantzig selector has been widely recommended to compare screening design construction methods. Here, we illustrate potential reproducibility issues that arise when comparing screening designs via simulation, and recommend a graphical method, based on screening probabilities, which compares designs by evaluating them along the penalized regression solution path. This method can be implemented using simulation, or, in the case of lasso, by using exact local lasso sign recovery probabilities. Our approach circumvents the need to specify tuning parameters associated with regularization methods, leading to more reliable design comparisons. This article contains supplementary materials including code to implement the proposed methods.
△ Less
Submitted 21 November, 2023;
originally announced November 2023.
-
Wavelet Image Restoration Using Multifractal Priors
Authors:
Karl Young,
John Kornak,
Eric Friedman
Abstract:
Bayesian image restoration has had a long history of successful application but one of the limitations that has prevented more widespread use is that the methods are generally computationally intensive. The authors recently addressed this issue by developing a method that performs the image enhancement in an orthogonal space (Fourier space in that case) which effectively transforms the problem fro…
▽ More
Bayesian image restoration has had a long history of successful application but one of the limitations that has prevented more widespread use is that the methods are generally computationally intensive. The authors recently addressed this issue by developing a method that performs the image enhancement in an orthogonal space (Fourier space in that case) which effectively transforms the problem from a large multivariate optimization problem to a set of smaller independent univariate optimization problems. The current paper extends these methods to analysis in another orthogonal basis, wavelets. While still providing the computational efficiency obtained with the original method in Fourier space, this extension allows more flexibility in adapting to local properties of the images, as well as capitalizing on the long history of developments for wavelet shrinkage methods. In addition, wavelet methods, including empirical Bayes specific methods, have recently been developed to effectively capture multifractal properties of images. An extension of these methods is utilized to enhance the recovery of textural characteristics of the underlying image. These enhancements should be beneficial in characterizing textural differences such as those occurring in medical images of diseased and healthy tissues. The Bayesian framework defined in the space of wavelets provides a flexible model that is easily extended to a variety of imaging contexts.
△ Less
Submitted 31 May, 2023;
originally announced June 2023.
-
Bayesian Image Analysis in Fourier Space
Authors:
John Kornak,
Karl Young,
Eric Friedman
Abstract:
Bayesian image analysis has played a large role over the last 40+ years in solving problems in image noise-reduction, de-blurring, feature enhancement, and object detection. However, these problems can be complex and lead to computational difficulties, due to the modeled interdependence between spatial locations. The Bayesian image analysis in Fourier space (BIFS) approach proposed here reformulat…
▽ More
Bayesian image analysis has played a large role over the last 40+ years in solving problems in image noise-reduction, de-blurring, feature enhancement, and object detection. However, these problems can be complex and lead to computational difficulties, due to the modeled interdependence between spatial locations. The Bayesian image analysis in Fourier space (BIFS) approach proposed here reformulates the conventional Bayesian image analysis paradigm for continuous valued images as a large set of independent (but heterogeneous) processes over Fourier space. The original high-dimensional estimation problem in image space is thereby broken down into (trivially parallelizable) independent one-dimensional problems in Fourier space. The BIFS approach leads to easy model specification with fast and direct computation, a wide range of possible prior characteristics, easy modeling of isotropy into the prior, and models that are effectively invariant to changes in image resolution.
△ Less
Submitted 11 January, 2024; v1 submitted 30 May, 2023;
originally announced May 2023.
-
Deep neural network or dermatologist?
Authors:
Kyle Young,
Gareth Booth,
Becks Simpson,
Reuben Dutton,
Sally Shrapnel
Abstract:
Deep learning techniques have proven high accuracy for identifying melanoma in digitised dermoscopic images. A strength is that these methods are not constrained by features that are pre-defined by human semantics. A down-side is that it is difficult to understand the rationale of the model predictions and to identify potential failure modes. This is a major barrier to adoption of deep learning in…
▽ More
Deep learning techniques have proven high accuracy for identifying melanoma in digitised dermoscopic images. A strength is that these methods are not constrained by features that are pre-defined by human semantics. A down-side is that it is difficult to understand the rationale of the model predictions and to identify potential failure modes. This is a major barrier to adoption of deep learning in clinical practice. In this paper we ask if two existing local interpretability methods, Grad-CAM and Kernel SHAP, can shed light on convolutional neural networks trained in the context of melanoma detection. Our contributions are (i) we first explore the domain space via a reproducible, end-to-end learning framework that creates a suite of 30 models, all trained on a publicly available data set (HAM10000), (ii) we next explore the reliability of GradCAM and Kernel SHAP in this context via some basic sanity check experiments (iii) finally, we investigate a random selection of models from our suite using GradCAM and Kernel SHAP. We show that despite high accuracy, the models will occasionally assign importance to features that are not relevant to the diagnostic task. We also show that models of similar accuracy will produce different explanations as measured by these methods. This work represents first steps in bridging the gap between model accuracy and interpretability in the domain of skin cancer classification.
△ Less
Submitted 19 August, 2019;
originally announced August 2019.
-
Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling
Authors:
Kenny J. Young,
Richard S. Sutton,
Shuo Yang
Abstract:
Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep l…
▽ More
Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.
△ Less
Submitted 1 June, 2018;
originally announced June 2018.
-
Metatrace Actor-Critic: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Control
Authors:
Kenny Young,
Baoxiang Wang,
Matthew E. Taylor
Abstract:
Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability. A variety of techniques exist to combat this --- most notably large experie…
▽ More
Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability. A variety of techniques exist to combat this --- most notably large experience replay buffers or the use of multiple parallel actors. These techniques come at the cost of moving away from the online RL problem as it is traditionally formulated (i.e., a single agent learning online without maintaining a large database of training examples). Meta-learning can potentially help with both these issues by tuning hyperparameters online and allowing the algorithm to more robustly adjust to non-stationarity in a problem. This paper applies meta-gradient descent to derive a set of step-size tuning algorithms specifically for online RL control with eligibility traces. Our novel technique, Metatrace, makes use of an eligibility trace analogous to methods like $TD(λ)$. We explore tuning both a single scalar step-size and a separate step-size for each learned parameter. We evaluate Metatrace first for control with linear function approximation in the classic mountain car problem and then in a noisy, non-stationary version. Finally, we apply Metatrace for control with nonlinear function approximation in 5 games in the Arcade Learning Environment where we explore how it impacts learning speed and robustness to initial step-size choice. Results show that the meta-step-size parameter of Metatrace is easy to set, Metatrace can speed learning, and Metatrace can allow an RL algorithm to deal with non-stationarity in the learning task.
△ Less
Submitted 24 May, 2019; v1 submitted 10 May, 2018;
originally announced May 2018.
