Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning
Authors:
Nils Lehmann,
Jakob Gawlikowski,
Adam J. Stewart,
Vytautas Jancauskas,
Stefan Depeweg,
Eric Nalisnick,
Nina Maria Gottschling
Abstract:
Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox tha…
▽ More
Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modelling workflow, without significant overhead. We introduce \texttt{Lightning UQ Box}: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. By highlighting the differences between methods our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking UQ methods. The toolbox, example implementations, and further information are available at: https://github.com/lightning-uq-box/lightning-uq-box
△ Less
Submitted 4 October, 2024;
originally announced October 2024.
VECMAtk: A Scalable Verification, Validation and Uncertainty Quantification Toolkit for Scientific Simulations
Authors:
D. Groen,
H. Arabnejad,
V. Jancauskas,
W. N. Edeling,
F. Jansson,
R. A. Richardson,
J. Lakhlili,
L. Veen,
B. Bosak,
P. Kopta,
D. W. Wright,
N. Monnier,
P. Karlshoefer,
D. Suleimenova,
R. Sinclair,
M. Vassaux,
A. Nikishova,
M. Bieniek,
O. O. Luk,
M. Kulczewski,
E. Raffin,
D. Crommelin,
O. Hoenen,
D. P. Coster,
T. Piontek
, et al. (1 additional authors not shown)
Abstract:
We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantification (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against ob…
▽ More
We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantification (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year, we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail.
△ Less
Submitted 11 October, 2020; v1 submitted 8 October, 2020;
originally announced October 2020.
