-
Real-time COVID-19 hospital admissions forecasting with leading indicators and ensemble methods in England
Authors:
Jonathon Mellor,
Rachel Christie,
Robert S Paton,
Rhianna Leslie,
Maria Tang,
Martyn Fyles,
Sarah Deeny,
Thomas Ward,
Christopher E Overton
Abstract:
Hospitalisations from COVID-19 with Omicron sub-lineages have put a sustained pressure on the English healthcare system. Understanding the expected healthcare demand enables more effective and timely planning from public health. We collect syndromic surveillance sources, which include online search data, NHS 111 telephonic and online triages. Incorporating this data we explore generalised additive…
▽ More
Hospitalisations from COVID-19 with Omicron sub-lineages have put a sustained pressure on the English healthcare system. Understanding the expected healthcare demand enables more effective and timely planning from public health. We collect syndromic surveillance sources, which include online search data, NHS 111 telephonic and online triages. Incorporating this data we explore generalised additive models, generalised linear mixed-models, penalised generalised linear models and model ensemble methods to forecast over a two-week forecast horizon at an NHS Trust level. Furthermore, we showcase how model combinations improve forecast scoring through a mean ensemble, weighted ensemble, and ensemble by regression. Validated over multiple Omicron waves, at different spatial scales, we show that leading indicators can improve performance of forecasting models, particularly at epidemic changepoints. Using a variety of scoring rules, we show that ensemble approaches outperformed all individual models, providing higher performance at a 21-day window than the corresponding individual models at 14-days. We introduce a modelling structure used by public health officials in England in 2022 to inform NHS healthcare strategy and policy decision making. This paper explores the significance of ensemble methods to improve forecasting performance and how novel syndromic surveillance can be practically applied in epidemic forecasting.
△ Less
Submitted 16 August, 2023; v1 submitted 9 June, 2023;
originally announced June 2023.
-
Understanding the leading indicators of hospital admissions from COVID-19 across successive waves in the UK
Authors:
Jonathon Mellor,
Christopher E Overton,
Martyn Fyles,
Liam Chawner,
James Baxter,
Tarrion Baird,
Thomas Ward
Abstract:
Following the UK Government's Living with COVID-19 Strategy and the end of universal testing, hospital admissions are an increasingly important measure of COVID-19 pandemic pressure. Understanding leading indicators of admissions at National Health Service (NHS) Trust, regional and national geographies help health services plan capacity needs and prepare for ongoing pressures. We explored the spat…
▽ More
Following the UK Government's Living with COVID-19 Strategy and the end of universal testing, hospital admissions are an increasingly important measure of COVID-19 pandemic pressure. Understanding leading indicators of admissions at National Health Service (NHS) Trust, regional and national geographies help health services plan capacity needs and prepare for ongoing pressures. We explored the spatio-temporal relationships of leading indicators of hospital pressure across successive waves of SARS-CoV-2 incidence in England. This includes an analysis of internet search volume values from Google Trends, NHS triage calls and online queries, the NHS COVID-19 App, lateral flow devices and the ZOE App. Data sources were analysed for their feasibility as leading indicators using linear and non-linear methods; granger causality, cross correlations and dynamic time warping at fine spatial scales. Consistent temporal and spatial relationships were found for some of the leading indicators assessed across resurgent waves of COVID-19. Google Trends and NHS queries consistently led admissions in over 70% of Trusts, with lead times ranging from 5-20 days, whereas an inconsistent relationship was found for the ZOE app, NHS COVID-19 App, and rapid testing, that diminished with granularity, showing limited autocorrelation of leads between -7 to 7 days. This work shows that novel syndromic surveillance data has utility for understanding the expected hospital burden at fine spatial scales. The analysis shows at low level geographies that some surveillance sources can predict hospital admissions, though care must be taken in relying on the lead times and consistency between waves.
△ Less
Submitted 16 August, 2023; v1 submitted 21 March, 2023;
originally announced March 2023.
-
Forecasting influenza hospital admissions within English sub-regions using hierarchical generalised additive models
Authors:
Jonathon Mellor,
Rachel Christie,
Christopher E Overton,
Robert S Paton,
Rhianna Leslie,
Maria Tang,
Sarah Deeny,
Thomas Ward
Abstract:
Background: Seasonal influenza causes a substantial burden on healthcare services over the winter period when these systems are already under pressure. Policies during the COVID-19 pandemic supressed the transmission of season influenza, making the timing and magnitude of a potential resurgence difficult to predict.
Methods: We developed a hierarchical generalised additive model (GAM) for the sh…
▽ More
Background: Seasonal influenza causes a substantial burden on healthcare services over the winter period when these systems are already under pressure. Policies during the COVID-19 pandemic supressed the transmission of season influenza, making the timing and magnitude of a potential resurgence difficult to predict.
Methods: We developed a hierarchical generalised additive model (GAM) for the short-term forecasting of hospital admissions with a positive test for the influenza virus sub-regionally across England. The model incorporates a multi-level structure of spatio-temporal splines, weekly seasonality, and spatial correlation. Using multiple performance metrics including interval score, coverage, bias, and median absolute error, the predictive performance is evaluated for the 2022/23 seasonal wave. Performance is measured against an autoregressive integrated moving average (ARIMA) time series model.
Results: The GAM method outperformed the ARIMA model across scoring rules at both high and low-level geographies, and across the different phases of the epidemic wave including the turning point. The performance of the GAM with a 14-day forecast horizon was comparable in error to the ARIMA at 7 days. The performance of the GAM is found to be most sensitive to the flexibility of the smoothing function that measures the national epidemic trend.
Interpretation: This study introduces a novel approach to short-term forecasting of hospital admissions with influenza using hierarchical, spatial, and temporal components. The model is data-driven and practical to deploy using information realistically available at time of prediction, addressing key limitations of epidemic forecasting approaches. This model was used across the winter for healthcare operational planning by the UK Health Security Agency and the National Health Service in England.
△ Less
Submitted 23 February, 2023;
originally announced February 2023.
-
Statistical Design and Analysis for Robust Machine Learning: A Case Study from COVID-19
Authors:
Davide Pigoli,
Kieran Baker,
Jobie Budd,
Lorraine Butler,
Harry Coppock,
Sabrina Egglestone,
Steven G. Gilmour,
Chris Holmes,
David Hurley,
Radka Jersakova,
Ivan Kiskin,
Vasiliki Koutra,
Jonathon Mellor,
George Nicholson,
Joe Packham,
Selina Patel,
Richard Payne,
Stephen J. Roberts,
Björn W. Schuller,
Ana Tendero-Cañadas,
Tracey Thornley,
Alexander Titcomb
Abstract:
Since early in the coronavirus disease 2019 (COVID-19) pandemic, there has been interest in using artificial intelligence methods to predict COVID-19 infection status based on vocal audio signals, for example cough recordings. However, existing studies have limitations in terms of data collection and of the assessment of the performances of the proposed predictive models. This paper rigorously ass…
▽ More
Since early in the coronavirus disease 2019 (COVID-19) pandemic, there has been interest in using artificial intelligence methods to predict COVID-19 infection status based on vocal audio signals, for example cough recordings. However, existing studies have limitations in terms of data collection and of the assessment of the performances of the proposed predictive models. This paper rigorously assesses state-of-the-art machine learning techniques used to predict COVID-19 infection status based on vocal audio signals, using a dataset collected by the UK Health Security Agency. This dataset includes acoustic recordings and extensive study participant meta-data. We provide guidelines on testing the performance of methods to classify COVID-19 infection status based on acoustic features and we discuss how these can be extended more generally to the development and assessment of predictive methods based on public health datasets.
△ Less
Submitted 27 February, 2023; v1 submitted 15 December, 2022;
originally announced December 2022.
-
Neural Architecture Search without Training
Authors:
Joseph Mellor,
Jack Turner,
Amos Storkey,
Elliot J. Crowley
Abstract:
The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be alleviated if we could partially predict a network's trained…
▽ More
The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be alleviated if we could partially predict a network's trained accuracy from its initial state. In this work, we examine the overlap of activations between datapoints in untrained networks and motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Bench-101, NAS-Bench-201, NATS-Bench, and Network Design Spaces. Our approach can be readily combined with more expensive search methods; we examine a simple adaptation of regularised evolutionary search. Code for reproducing our experiments is available at https://github.com/BayesWatch/nas-without-training.
△ Less
Submitted 11 June, 2021; v1 submitted 8 June, 2020;
originally announced June 2020.
-
Better Boosting with Bandits for Online Learning
Authors:
Nikolaos Nikolaou,
Joseph Mellor,
Nikunj C. Oza,
Gavin Brown
Abstract:
Probability estimates generated by boosting ensembles are poorly calibrated because of the margin maximization nature of the algorithm. The outputs of the ensemble need to be properly calibrated before they can be used as probability estimates. In this work, we demonstrate that online boosting is also prone to producing distorted probability estimates. In batch learning, calibration is achieved by…
▽ More
Probability estimates generated by boosting ensembles are poorly calibrated because of the margin maximization nature of the algorithm. The outputs of the ensemble need to be properly calibrated before they can be used as probability estimates. In this work, we demonstrate that online boosting is also prone to producing distorted probability estimates. In batch learning, calibration is achieved by reserving part of the training data for training the calibrator function. In the online setting, a decision needs to be made on each round: shall the new example(s) be used to update the parameters of the ensemble or those of the calibrator. We proceed to resolve this decision with the aid of bandit optimization algorithms. We demonstrate superior performance to uncalibrated and naively-calibrated on-line boosting ensembles in terms of probability estimation. Our proposed mechanism can be easily adapted to other tasks(e.g. cost-sensitive classification) and is robust to the choice of hyperparameters of both the calibrator and the ensemble.
△ Less
Submitted 16 January, 2020;
originally announced January 2020.
-
Unsupervised Doodling and Painting with Improved SPIRAL
Authors:
John F. J. Mellor,
Eunbyung Park,
Yaroslav Ganin,
Igor Babuschkin,
Tejas Kulkarni,
Dan Rosenbaum,
Andy Ballard,
Theophane Weber,
Oriol Vinyals,
S. M. Ali Eslami
Abstract:
We investigate using reinforcement learning agents as generative models of images (extending arXiv:1804.01118). A generative agent controls a simulated painting environment, and is trained with rewards provided by a discriminator network simultaneously trained to assess the realism of the agent's samples, either unconditional or reconstructions. Compared to prior work, we make a number of improvem…
▽ More
We investigate using reinforcement learning agents as generative models of images (extending arXiv:1804.01118). A generative agent controls a simulated painting environment, and is trained with rewards provided by a discriminator network simultaneously trained to assess the realism of the agent's samples, either unconditional or reconstructions. Compared to prior work, we make a number of improvements to the architectures of the agents and discriminators that lead to intriguing and at times surprising results. We find that when sufficiently constrained, generative agents can learn to produce images with a degree of visual abstraction, despite having only ever seen real photographs (no human brush strokes). And given enough time with the painting environment, they can produce images with considerable realism. These results show that, under the right circumstances, some aspects of human drawing can emerge from simulated embodiment, without the need for external supervision, imitation or social cues. Finally, we note the framework's potential for use in creative applications.
△ Less
Submitted 2 October, 2019;
originally announced October 2019.
-
The K-Nearest Neighbour UCB algorithm for multi-armed bandits with covariates
Authors:
Henry WJ Reeve,
Joe Mellor,
Gavin Brown
Abstract:
In this paper we propose and explore the k-Nearest Neighbour UCB algorithm for multi-armed bandits with covariates. We focus on a setting where the covariates are supported on a metric space of low intrinsic dimension, such as a manifold embedded within a high dimensional ambient feature space. The algorithm is conceptually simple and straightforward to implement. The k-Nearest Neighbour UCB algor…
▽ More
In this paper we propose and explore the k-Nearest Neighbour UCB algorithm for multi-armed bandits with covariates. We focus on a setting where the covariates are supported on a metric space of low intrinsic dimension, such as a manifold embedded within a high dimensional ambient feature space. The algorithm is conceptually simple and straightforward to implement. The k-Nearest Neighbour UCB algorithm does not require prior knowledge of the either the intrinsic dimension of the marginal distribution or the time horizon. We prove a regret bound for the k-Nearest Neighbour UCB algorithm which is minimax optimal up to logarithmic factors. In particular, the algorithm automatically takes advantage of both low intrinsic dimensionality of the marginal distribution over the covariates and low noise in the data, expressed as a margin condition. In addition, focusing on the case of bounded rewards, we give corresponding regret bounds for the k-Nearest Neighbour KL-UCB algorithm, which is an analogue of the KL-UCB algorithm adapted to the setting of multi-armed bandits with covariates. Finally, we present empirical results which demonstrate the ability of both the k-Nearest Neighbour UCB and k-Nearest Neighbour KL-UCB to take advantage of situations where the data is supported on an unknown sub-manifold of a high-dimensional feature space.
△ Less
Submitted 1 March, 2018;
originally announced March 2018.
