-
CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation
Authors:
Claudius Krause,
Michele Faucci Giannelli,
Gregor Kasieczka,
Benjamin Nachman,
Dalila Salamani,
David Shih,
Anna Zaborowska,
Oz Amram,
Kerstin Borras,
Matthew R. Buckley,
Erik Buhmann,
Thorsten Buss,
Renato Paulo Da Costa Cardoso,
Anthony L. Caterini,
Nadezda Chernyavskaya,
Federico A. G. Corchia,
Jesse C. Cresswell,
Sascha Diefenbacher,
Etienne Dreyer,
Vijay Ekambaram,
Engin Eren,
Florian Ernst,
Luigi Favaro,
Matteo Franchini,
Frank Gaede
, et al. (44 additional authors not shown)
Abstract:
We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including Variational AutoEncoder…
▽ More
We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including Variational AutoEncoders (VAEs), Generative Adversarial Networks (GANs), Normalizing Flows, Diffusion models, and models based on Conditional Flow Matching. We compare all submissions in terms of quality of generated calorimeter showers, as well as shower generation time and model size. To assess the quality we use a broad range of different metrics including differences in 1-dimensional histograms of observables, KPD/FPD scores, AUCs of binary classifiers, and the log-posterior of a multiclass classifier. The results of the CaloChallenge provide the most complete and comprehensive survey of cutting-edge approaches to calorimeter fast simulation to date. In addition, our work provides a uniquely detailed perspective on the important problem of how to evaluate generative models. As such, the results presented here should be applicable for other domains that use generative AI and require fast and faithful generation of samples in a large phase space.
△ Less
Submitted 28 October, 2024;
originally announced October 2024.
-
Refereeing the Referees: Evaluating Two-Sample Tests for Validating Generators in Precision Sciences
Authors:
Samuele Grossi,
Marco Letizia,
Riccardo Torre
Abstract:
We propose a robust methodology to evaluate the performance and computational efficiency of non-parametric two-sample tests, specifically designed for high-dimensional generative models in scientific applications such as in particle physics. The study focuses on tests built from univariate integral probability measures: the sliced Wasserstein distance and the mean of the Kolmogorov-Smirnov statist…
▽ More
We propose a robust methodology to evaluate the performance and computational efficiency of non-parametric two-sample tests, specifically designed for high-dimensional generative models in scientific applications such as in particle physics. The study focuses on tests built from univariate integral probability measures: the sliced Wasserstein distance and the mean of the Kolmogorov-Smirnov statistics, already discussed in the literature, and the novel sliced Kolmogorov-Smirnov statistic. These metrics can be evaluated in parallel, allowing for fast and reliable estimates of their distribution under the null hypothesis. We also compare these metrics with the recently proposed unbiased Fréchet Gaussian Distance and the unbiased quadratic Maximum Mean Discrepancy, computed with a quartic polynomial kernel. We evaluate the proposed tests on various distributions, focusing on their sensitivity to deformations parameterized by a single parameter $ε$. Our experiments include correlated Gaussians and mixtures of Gaussians in 5, 20, and 100 dimensions, and a particle physics dataset of gluon jets from the JetNet dataset, considering both jet- and particle-level features. Our results demonstrate that one-dimensional-based tests provide a level of sensitivity comparable to other multivariate metrics, but with significantly lower computational cost, making them ideal for evaluating generative models in high-dimensional settings. This methodology offers an efficient, standardized tool for model comparison and can serve as a benchmark for more advanced tests, including machine-learning-based approaches.
△ Less
Submitted 24 September, 2024;
originally announced September 2024.
-
Multiple testing for signal-agnostic searches of new physics with machine learning
Authors:
Gaia Grosso,
Marco Letizia
Abstract:
In this work, we address the question of how to enhance signal-agnostic searches by leveraging multiple testing strategies. Specifically, we consider hypothesis tests relying on machine learning, where model selection can introduce a bias towards specific families of new physics signals. We show that it is beneficial to combine different tests, characterised by distinct choices of hyperparameters,…
▽ More
In this work, we address the question of how to enhance signal-agnostic searches by leveraging multiple testing strategies. Specifically, we consider hypothesis tests relying on machine learning, where model selection can introduce a bias towards specific families of new physics signals. We show that it is beneficial to combine different tests, characterised by distinct choices of hyperparameters, and that performances comparable to the best available test are generally achieved while providing a more uniform response to various types of anomalies. Focusing on the New Physics Learning Machine, a methodology to perform a signal-agnostic likelihood-ratio test, we explore a number of approaches to multiple testing, such as combining p-values and aggregating test statistics.
△ Less
Submitted 22 August, 2024;
originally announced August 2024.
-
Goodness of fit by Neyman-Pearson testing
Authors:
Gaia Grosso,
Marco Letizia,
Maurizio Pierini,
Andrea Wulzer
Abstract:
The Neyman-Pearson strategy for hypothesis testing can be employed for goodness of fit if the alternative hypothesis is selected from data by exploring a rich parametrised family of models, while controlling the impact of statistical fluctuations. The New Physics Learning Machine (NPLM) methodology has been developed as a concrete implementation of this idea, to target the detection of new physica…
▽ More
The Neyman-Pearson strategy for hypothesis testing can be employed for goodness of fit if the alternative hypothesis is selected from data by exploring a rich parametrised family of models, while controlling the impact of statistical fluctuations. The New Physics Learning Machine (NPLM) methodology has been developed as a concrete implementation of this idea, to target the detection of new physical effects in the context of high energy physics collider experiments. In this paper we conduct a comparison of this approach to goodness of fit with others, in particular with classifier-based strategies that share strong similarities with NPLM. From our comparison, NPLM emerges as the more sensitive test to small departures of the data from the expected distribution and not biased towards detecting specific types of anomalies. These features make it suited for agnostic searches for new physics at collider experiments. Its deployment in other scientific and industrial scenarios should be investigated.
△ Less
Submitted 15 March, 2024; v1 submitted 23 May, 2023;
originally announced May 2023.
-
Comparative Study of Coupling and Autoregressive Flows through Robust Statistical Tests
Authors:
Andrea Coccaro,
Marco Letizia,
Humberto Reyes-Gonzalez,
Riccardo Torre
Abstract:
Normalizing Flows have emerged as a powerful brand of generative models, as they not only allow for efficient sampling of complicated target distributions, but also deliver density estimation by construction. We propose here an in-depth comparison of coupling and autoregressive flows, both of the affine and rational quadratic spline type, considering four different architectures: Real-valued Non-V…
▽ More
Normalizing Flows have emerged as a powerful brand of generative models, as they not only allow for efficient sampling of complicated target distributions, but also deliver density estimation by construction. We propose here an in-depth comparison of coupling and autoregressive flows, both of the affine and rational quadratic spline type, considering four different architectures: Real-valued Non-Volume Preserving (RealNVP), Masked Autoregressive Flow (MAF), Coupling Rational Quadratic Spline (C-RQS), and Autoregressive Rational Quadratic Spline (A-RQS). We focus on a set of multimodal target distributions of increasing dimensionality ranging from 4 to 400. The performances are compared by means of different test-statistics for two-sample tests, built from known distance measures: the sliced Wasserstein distance, the dimension-averaged one-dimensional Kolmogorov-Smirnov test, and the Frobenius norm of the difference between correlation matrices. Furthermore, we include estimations of the variance of both the metrics and the trained models. Our results indicate that the A-RQS algorithm stands out both in terms of accuracy and training speed. Nonetheless, all the algorithms are generally able, without too much fine-tuning, to learn complicated distributions with limited training data and in a reasonable time, of the order of hours on a Tesla A40 GPU. The only exception is the C-RQS, which takes significantly longer to train, does not always provide good accuracy, and becomes unstable for large dimensionalities. All algorithms have been implemented using TensorFlow2 and TensorFlow Probability and made available on \href{https://github.com/NF4HEP/NormalizingFlowsHD}{GitHub}.
△ Less
Submitted 16 January, 2024; v1 submitted 23 February, 2023;
originally announced February 2023.
-
CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds
Authors:
Jesse C. Cresswell,
Brendan Leigh Ross,
Gabriel Loaiza-Ganem,
Humberto Reyes-Gonzalez,
Marco Letizia,
Anthony L. Caterini
Abstract:
Precision measurements and new physics searches at the Large Hadron Collider require efficient simulations of particle propagation and interactions within the detectors. The most computationally expensive simulations involve calorimeter showers. Advances in deep generative modelling - particularly in the realm of high-dimensional data - have opened the possibility of generating realistic calorimet…
▽ More
Precision measurements and new physics searches at the Large Hadron Collider require efficient simulations of particle propagation and interactions within the detectors. The most computationally expensive simulations involve calorimeter showers. Advances in deep generative modelling - particularly in the realm of high-dimensional data - have opened the possibility of generating realistic calorimeter showers orders of magnitude more quickly than physics-based simulation. However, the high-dimensional representation of showers belies the relative simplicity and structure of the underlying physical laws. This phenomenon is yet another example of the manifold hypothesis from machine learning, which states that high-dimensional data is supported on low-dimensional manifolds. We thus propose modelling calorimeter showers first by learning their manifold structure, and then estimating the density of data across this manifold. Learning manifold structure reduces the dimensionality of the data, which enables fast training and generation when compared with competing methods.
△ Less
Submitted 23 November, 2022;
originally announced November 2022.
-
Learning new physics efficiently with nonparametric methods
Authors:
Marco Letizia,
Gianvito Losapio,
Marco Rando,
Gaia Grosso,
Andrea Wulzer,
Maurizio Pierini,
Marco Zanetti,
Lorenzo Rosasco
Abstract:
We present a machine learning approach for model-independent new physics searches. The corresponding algorithm is powered by recent large-scale implementations of kernel methods, nonparametric learning algorithms that can approximate any continuous function given enough data. Based on the original proposal by D'Agnolo and Wulzer (arXiv:1806.02350), the model evaluates the compatibility between exp…
▽ More
We present a machine learning approach for model-independent new physics searches. The corresponding algorithm is powered by recent large-scale implementations of kernel methods, nonparametric learning algorithms that can approximate any continuous function given enough data. Based on the original proposal by D'Agnolo and Wulzer (arXiv:1806.02350), the model evaluates the compatibility between experimental data and a reference model, by implementing a hypothesis testing procedure based on the likelihood ratio. Model-independence is enforced by avoiding any prior assumption about the presence or shape of new physics components in the measurements. We show that our approach has dramatic advantages compared to neural network implementations in terms of training times and computational resources, while maintaining comparable performances. In particular, we conduct our tests on higher dimensional datasets, a step forward with respect to previous studies.
△ Less
Submitted 5 April, 2022;
originally announced April 2022.
-
Phenomenology of effective geometries from quantum gravity
Authors:
Ricardo Gallego Torromé,
Marco Letizia,
Stefano Liberati
Abstract:
In a recent paper (arXiv:1412.6000) a general mechanism for emergence of cosmological space-time geometry from a quantum gravity setting was devised and departure from standard dispersion relations for elementary particle were predicted. We elaborate here on this approach extending the results obtained in that paper and showing that generically such a framework will not lead to higher order modifi…
▽ More
In a recent paper (arXiv:1412.6000) a general mechanism for emergence of cosmological space-time geometry from a quantum gravity setting was devised and departure from standard dispersion relations for elementary particle were predicted. We elaborate here on this approach extending the results obtained in that paper and showing that generically such a framework will not lead to higher order modified dispersion relations in the matter sector. Furthermore, we shall discuss possible phenomenological constraints to this scenarios showing that space-time will have to be by nowadays classical to a very high degree in order to be consistent with current observations.
△ Less
Submitted 12 July, 2015;
originally announced July 2015.
