-
Improved Online Conformal Prediction via Strongly Adaptive Online Learning
Authors:
Aadyot Bhatnagar,
Huan Wang,
Caiming Xiong,
Yu Bai
Abstract:
We study the problem of uncertainty quantification via prediction sets, in an online setting where the data distribution may vary arbitrarily over time. Recent work develops online conformal prediction techniques that leverage regret minimization algorithms from the online learning literature to learn prediction sets with approximately valid coverage and small regret. However, standard regret mini…
▽ More
We study the problem of uncertainty quantification via prediction sets, in an online setting where the data distribution may vary arbitrarily over time. Recent work develops online conformal prediction techniques that leverage regret minimization algorithms from the online learning literature to learn prediction sets with approximately valid coverage and small regret. However, standard regret minimization could be insufficient for handling changing environments, where performance guarantees may be desired not only over the full time horizon but also in all (sub-)intervals of time. We develop new online conformal prediction methods that minimize the strongly adaptive regret, which measures the worst-case regret over all intervals of a fixed length. We prove that our methods achieve near-optimal strongly adaptive regret for all interval lengths simultaneously, and approximately valid coverage. Experiments show that our methods consistently obtain better coverage and smaller prediction sets than existing methods on real-world tasks, such as time series forecasting and image classification under distribution shift.
△ Less
Submitted 15 February, 2023;
originally announced February 2023.
-
Merlion: A Machine Learning Library for Time Series
Authors:
Aadyot Bhatnagar,
Paul Kassianik,
Chenghao Liu,
Tian Lan,
Wenzhuo Yang,
Rowan Cassius,
Doyen Sahoo,
Devansh Arpit,
Sri Subramanian,
Gerald Woo,
Amrita Saha,
Arun Kumar Jagota,
Gokulakrishnan Gopalakrishnan,
Manpreet Singh,
K C Krithika,
Sukumar Maddineni,
Daeki Cho,
Bo Zong,
Yingbo Zhou,
Caiming Xiong,
Silvio Savarese,
Steven Hoi,
Huan Wang
Abstract:
We introduce Merlion, an open-source machine learning library for time series. It features a unified interface for many commonly used models and datasets for anomaly detection and forecasting on both univariate and multivariate time series, along with standard pre/post-processing layers. It has several modules to improve ease-of-use, including visualization, anomaly score calibration to improve in…
▽ More
We introduce Merlion, an open-source machine learning library for time series. It features a unified interface for many commonly used models and datasets for anomaly detection and forecasting on both univariate and multivariate time series, along with standard pre/post-processing layers. It has several modules to improve ease-of-use, including visualization, anomaly score calibration to improve interpetability, AutoML for hyperparameter tuning and model selection, and model ensembling. Merlion also provides a unique evaluation framework that simulates the live deployment and re-training of a model in production. This library aims to provide engineers and researchers a one-stop solution to rapidly develop models for their specific time series needs and benchmark them across multiple time series datasets. In this technical report, we highlight Merlion's architecture and major functionalities, and we report benchmark numbers across different baseline models and ensembles.
△ Less
Submitted 19 September, 2021;
originally announced September 2021.
-
Learning to Search via Retrospective Imitation
Authors:
Jialin Song,
Ravi Lanka,
Albert Zhao,
Aadyot Bhatnagar,
Yisong Yue,
Masahiro Ono
Abstract:
We study the problem of learning a good search policy for combinatorial search spaces. We propose retrospective imitation learning, which, after initial training by an expert, improves itself by learning from \textit{retrospective inspections} of its own roll-outs. That is, when the policy eventually reaches a feasible solution in a combinatorial search tree after making mistakes and backtracks, i…
▽ More
We study the problem of learning a good search policy for combinatorial search spaces. We propose retrospective imitation learning, which, after initial training by an expert, improves itself by learning from \textit{retrospective inspections} of its own roll-outs. That is, when the policy eventually reaches a feasible solution in a combinatorial search tree after making mistakes and backtracks, it retrospectively constructs an improved search trace to the solution by removing backtracks, which is then used to further train the policy. A key feature of our approach is that it can iteratively scale up, or transfer, to larger problem sizes than those solved by the initial expert demonstrations, thus dramatically expanding its applicability beyond that of conventional imitation learning. We showcase the effectiveness of our approach on a range of tasks, including synthetic maze solving and combinatorial problems expressed as integer programs.
△ Less
Submitted 23 June, 2019; v1 submitted 3 April, 2018;
originally announced April 2018.
-
Risk Factors Associated with Mortality in Game of Thrones: A Longitudinal Cohort Study
Authors:
Suveen Angraal,
Ambika Bhatnagar,
Suraj Verma,
Sukhman Shergill,
Aakriti Gupta,
Rohan Khera
Abstract:
Objective: To assess mortality, and identify the risk factors associated with mortality in Game of Thrones (GoT). Design and Setting: A longitudinal cohort study in the fictional kingdom of Westeros and Essos. Participants: All the characters appearing in the GoT since airing of its first episode with screen time of greater than or equal to 5 minutes. Main Outcome Measures: All-cause mortality. Mu…
▽ More
Objective: To assess mortality, and identify the risk factors associated with mortality in Game of Thrones (GoT). Design and Setting: A longitudinal cohort study in the fictional kingdom of Westeros and Essos. Participants: All the characters appearing in the GoT since airing of its first episode with screen time of greater than or equal to 5 minutes. Main Outcome Measures: All-cause mortality. Multivariate Cox proportional hazard model was used to assess the risk factors associated with mortality, represented by hazard ratios, with episodes as the unit of time. Results: Of the 132 characters, followed up for a median time of 32 episodes, a total 89 (67.4%) characters died; with external invasive injury as the most common cause of death, attributing to 42.4% of the total deaths. Age (in decades) was a significant risk factor for death [HR, 1.24 (95% CI, 1.08-1.43), P=0.0001]. Although statistically non-significant, allegiance to house Targaryen [HR, 1.10 (95% CI, 0.32-3.77)] was associated with a higher risk for mortality per episode than house Stark. Characters residing in South were less likely to die than characters residing in North [HR, 0.58 (95% CI, 0.29-1.16), P=0.12]. Advisors showed a lower risk of mortality than the members of houses, with some statistical significance [HR, 0.39 (95% CI, 0.14-1.08), P=0.07]. Conclusions: There is a high mortality rate among the characters in GoT. Residing in the North and being a member of a house is very dangerous in GoT. Allegiance to house Stark trended to be safer than house Targaryen.
△ Less
Submitted 1 August, 2018; v1 submitted 12 February, 2018;
originally announced February 2018.
