-
A Case Study in Model Failure? COVID-19 Daily Deaths and ICU Bed Utilisation Predictions in New York State
Authors:
Vincent Chin,
Noelle I. Samia,
Roman Marchant,
Ori Rosen,
John P. A. Ioannidis,
Martin A. Tanner,
Sally Cripps
Abstract:
Forecasting models have been influential in shaping decision-making in the COVID-19 pandemic. However, there is concern that their predictions may have been misleading. Here, we dissect the predictions made by four models for the daily COVID-19 death counts between March 25 and June 5 in New York state, as well as the predictions of ICU bed utilisation made by the influential IHME model. We evalua…
▽ More
Forecasting models have been influential in shaping decision-making in the COVID-19 pandemic. However, there is concern that their predictions may have been misleading. Here, we dissect the predictions made by four models for the daily COVID-19 death counts between March 25 and June 5 in New York state, as well as the predictions of ICU bed utilisation made by the influential IHME model. We evaluated the accuracy of the point estimates and the accuracy of the uncertainty estimates of the model predictions. First, we compared the "ground truth" data sources on daily deaths against which these models were trained. Three different data sources were used by these models, and these had substantial differences in recorded daily death counts. Two additional data sources that we examined also provided different death counts per day. For accuracy of prediction, all models fared very poorly. Only 10.2% of the predictions fell within 10% of their training ground truth, irrespective of distance into the future. For accurate assessment of uncertainty, only one model matched relatively well the nominal 95% coverage, but that model did not start predictions until April 16, thus had no impact on early, major decisions. For ICU bed utilisation, the IHME model was highly inaccurate; the point estimates only started to match ground truth after the pandemic wave had started to wane. We conclude that trustworthy models require trustworthy input data to be trained upon. Moreover, models need to be subjected to prespecified real time performance tests, before their results are provided to policy makers and public health officials.
△ Less
Submitted 25 June, 2020;
originally announced June 2020.
-
Multi-core parallel tempering Bayeslands for basin and landscape evolution
Authors:
Rohitash Chandra,
R. Dietmar Müller,
Danial Azam,
Ratneel Deo,
Nathaniel Butterworth,
Tristan Salles,
Sally Cripps
Abstract:
The Bayesian paradigm is becoming an increasingly popular framework for estimation and uncertainty quantification of unknown parameters in geo-physical inversion problems. Badlands is a basin and landscape evolution forward model for simulating topography evolution at a large range of spatial and time scales. Our previous work presented Bayeslands that used the Bayesian paradigm to make inference…
▽ More
The Bayesian paradigm is becoming an increasingly popular framework for estimation and uncertainty quantification of unknown parameters in geo-physical inversion problems. Badlands is a basin and landscape evolution forward model for simulating topography evolution at a large range of spatial and time scales. Our previous work presented Bayeslands that used the Bayesian paradigm to make inference for unknown parameters in the Badlands model using Markov chain Monte Carlo (MCMC) sampling. Bayeslands faced challenges in convergence due to multi-modal posterior distributions in the selected parameters of Badlands. Parallel tempering is an advanced MCMC method suited for irregular and multi-modal posterior distributions. In this paper, we extend Bayeslands using parallel tempering (PT-Bayeslands) with high performance computing to address previous limitations in parameter space exploration in the context of the computationally expensive Badlands model. Our results show that PT-Bayeslands not only reduces the computation time, but also provides an improvement of the sampling for multi-modal posterior distributions. This provides an improvement over Bayeslands which used single chain MCMC that face difficulties in convergence and can lead to misleading inference. This motivates its usage in large-scale basin and landscape evolution models.
△ Less
Submitted 19 July, 2019; v1 submitted 22 June, 2018;
originally announced June 2018.
-
Bayeslands: A Bayesian inference approach for parameter uncertainty quantification in Badlands
Authors:
Rohitash Chandra,
Danial Azam,
R. Dietmar Müller,
Tristan Salles,
Sally Cripps
Abstract:
Bayesian inference provides a rigorous methodology for estimation and uncertainty quantification of parameters in geophysical forward models. Badlands (basin and landscape dynamics model) is a landscape evolution model that simulates topography development at various space and time scales. Badlands consists of a number of geophysical parameters that needs estimation with appropriate uncertainty qu…
▽ More
Bayesian inference provides a rigorous methodology for estimation and uncertainty quantification of parameters in geophysical forward models. Badlands (basin and landscape dynamics model) is a landscape evolution model that simulates topography development at various space and time scales. Badlands consists of a number of geophysical parameters that needs estimation with appropriate uncertainty quantification; given the observed present-day ground truth such as surface topography and the stratigraphy of sediment deposition through time. The inference of unknown parameters is challenging due to the scarcity of data, sensitivity of the parameter setting and complexity of the Badlands model. In this paper, we take a Bayesian approach to provide inference using Markov chain Monte Carlo sampling (MCMC). We present \textit{Bayeslands}; a Bayesian framework for Badlands that fuses information obtained from complex forward models with observational data and prior knowledge. As a proof-of-concept, we consider a synthetic and real-world topography with two parameters for Bayeslands inference, namely precipitation and erodibility. The results of the experiments show that Bayeslands yields a promising distribution of the parameters. Moreover, we demonstrate the challenge in sampling irregular and multi-modal posterior distributions using a likelihood surface that has a range of sub-optimal modes.
△ Less
Submitted 5 July, 2021; v1 submitted 2 May, 2018;
originally announced May 2018.
