-
The Value of Context: Human versus Black Box Evaluators
Authors:
Andrei Iakovlev,
Annie Liang
Abstract:
Machine learning algorithms are now capable of performing evaluations previously conducted by human experts (e.g., medical diagnoses). How should we conceptualize the difference between evaluation by humans and by algorithms, and when should an individual prefer one over the other? We propose a framework to examine one key distinction between the two forms of evaluation: Machine learning algorithm…
▽ More
Machine learning algorithms are now capable of performing evaluations previously conducted by human experts (e.g., medical diagnoses). How should we conceptualize the difference between evaluation by humans and by algorithms, and when should an individual prefer one over the other? We propose a framework to examine one key distinction between the two forms of evaluation: Machine learning algorithms are standardized, fixing a common set of covariates by which to assess all individuals, while human evaluators customize which covariates are acquired to each individual. Our framework defines and analyzes the advantage of this customization -- the value of context -- in environments with high-dimensional data. We show that unless the agent has precise knowledge about the joint distribution of covariates, the benefit of additional covariates generally outweighs the value of context.
△ Less
Submitted 29 June, 2024; v1 submitted 16 February, 2024;
originally announced February 2024.
-
Deep Double Descent for Time Series Forecasting: Avoiding Undertrained Models
Authors:
Valentino Assandri,
Sam Heshmati,
Burhaneddin Yaman,
Anton Iakovlev,
Ariel Emiliano Repetur
Abstract:
Deep learning models, particularly Transformers, have achieved impressive results in various domains, including time series forecasting. While existing time series literature primarily focuses on model architecture modifications and data augmentation techniques, this paper explores the training schema of deep learning models for time series; how models are trained regardless of their architecture.…
▽ More
Deep learning models, particularly Transformers, have achieved impressive results in various domains, including time series forecasting. While existing time series literature primarily focuses on model architecture modifications and data augmentation techniques, this paper explores the training schema of deep learning models for time series; how models are trained regardless of their architecture. We perform extensive experiments to investigate the occurrence of deep double descent in several Transformer models trained on public time series data sets. We demonstrate epoch-wise deep double descent and that overfitting can be reverted using more epochs. Leveraging these findings, we achieve state-of-the-art results for long sequence time series forecasting in nearly 70% of the 72 benchmarks tested. This suggests that many models in the literature may possess untapped potential. Additionally, we introduce a taxonomy for classifying training schema modifications, covering data augmentation, model inputs, model targets, time series per model, and computational budget.
△ Less
Submitted 30 November, 2023; v1 submitted 2 November, 2023;
originally announced November 2023.
-
The ATLAS EventIndex: a BigData catalogue for all ATLAS experiment events
Authors:
Dario Barberis,
Igor Aleksandrov,
Evgeny Alexandrov,
Zbigniew Baranowski,
Luca Canali,
Elizaveta Cherepanova,
Gancho Dimitrov,
Andrea Favareto,
Alvaro Fernandez Casani,
Elizabeth J. Gallas,
Carlos Garcia Montoro,
Santiago Gonzalez de la Hoz,
Julius Hrivnac,
Alexander Iakovlev,
Andrei Kazymov,
Mikhail Mineev,
Fedor Prokoshin,
Grigori Rybkin,
Jose Salt,
Javier Sanchez,
Roman Sorokoletov,
Rainer Toebbicke,
Petya Vasileva,
Miguel Villaplana Perez,
Ruijun Yuan
Abstract:
The ATLAS EventIndex system comprises the catalogue of all events collected, processed or generated by the ATLAS experiment at the CERN LHC accelerator, and all associated software tools to collect, store and query this information. ATLAS records several billion particle interactions every year of operation, processes them for analysis and generates even larger simulated data samples; a global cat…
▽ More
The ATLAS EventIndex system comprises the catalogue of all events collected, processed or generated by the ATLAS experiment at the CERN LHC accelerator, and all associated software tools to collect, store and query this information. ATLAS records several billion particle interactions every year of operation, processes them for analysis and generates even larger simulated data samples; a global catalogue is needed to keep track of the location of each event record and be able to search and retrieve specific events for in-depth investigations. Each EventIndex record includes summary information on the event itself and the pointers to the files containing the full event. Most components of the EventIndex system are implemented using BigData open-source tools. This paper describes the architectural choices and their evolution in time, as well as the past, current and foreseen future implementations of all EventIndex components.
△ Less
Submitted 12 March, 2023; v1 submitted 15 November, 2022;
originally announced November 2022.
