-
Meta-learning Loss Functions of Parametric Partial Differential Equations Using Physics-Informed Neural Networks
Authors:
Michail Koumpanakis,
Ricardo Vilalta
Abstract:
This paper proposes a new way to learn Physics-Informed Neural Network loss functions using Generalized Additive Models. We apply our method by meta-learning parametric partial differential equations, PDEs, on Burger's and 2D Heat Equations. The goal is to learn a new loss function for each parametric PDE using meta-learning. The derived loss function replaces the traditional data loss, allowing u…
▽ More
This paper proposes a new way to learn Physics-Informed Neural Network loss functions using Generalized Additive Models. We apply our method by meta-learning parametric partial differential equations, PDEs, on Burger's and 2D Heat Equations. The goal is to learn a new loss function for each parametric PDE using meta-learning. The derived loss function replaces the traditional data loss, allowing us to learn each parametric PDE more efficiently, improving the meta-learner's performance and convergence.
△ Less
Submitted 29 November, 2024;
originally announced December 2024.
-
Robust Errant Beam Prognostics with Conditional Modeling for Particle Accelerators
Authors:
Kishansingh Rajput,
Malachi Schram,
Willem Blokland,
Yasir Alanazi,
Pradeep Ramuhalli,
Alexander Zhukov,
Charles Peters,
Ricardo Vilalta
Abstract:
Particle accelerators are complex and comprise thousands of components, with many pieces of equipment running at their peak power. Consequently, particle accelerators can fault and abort operations for numerous reasons. These faults impact the availability of particle accelerators during scheduled run-time and hamper the efficiency and the overall science output. To avoid these faults, we apply an…
▽ More
Particle accelerators are complex and comprise thousands of components, with many pieces of equipment running at their peak power. Consequently, particle accelerators can fault and abort operations for numerous reasons. These faults impact the availability of particle accelerators during scheduled run-time and hamper the efficiency and the overall science output. To avoid these faults, we apply anomaly detection techniques to predict any unusual behavior and perform preemptive actions to improve the total availability of particle accelerators. Semi-supervised Machine Learning (ML) based anomaly detection approaches such as autoencoders and variational autoencoders are often used for such tasks. However, supervised ML techniques such as Siamese Neural Network (SNN) models can outperform unsupervised or semi-supervised approaches for anomaly detection by leveraging the label information. One of the challenges specific to anomaly detection for particle accelerators is the data's variability due to system configuration changes. To address this challenge, we employ Conditional Siamese Neural Network (CSNN) models and Conditional Variational Auto Encoder (CVAE) models to predict errant beam pulses at the Spallation Neutron Source (SNS) under different system configuration conditions and compare their performance. We demonstrate that CSNN outperforms CVAE in our application.
△ Less
Submitted 19 February, 2024; v1 submitted 22 November, 2023;
originally announced December 2023.
-
Physics-informed neural networks in the recreation of hydrodynamic simulations from dark matter
Authors:
Zhenyu Dai,
Ben Moews,
Ricardo Vilalta,
Romeel Dave
Abstract:
Physics-informed neural networks have emerged as a coherent framework for building predictive models that combine statistical patterns with domain knowledge. The underlying notion is to enrich the optimization loss function with known relationships to constrain the space of possible solutions. Hydrodynamic simulations are a core constituent of modern cosmology, while the required computations are…
▽ More
Physics-informed neural networks have emerged as a coherent framework for building predictive models that combine statistical patterns with domain knowledge. The underlying notion is to enrich the optimization loss function with known relationships to constrain the space of possible solutions. Hydrodynamic simulations are a core constituent of modern cosmology, while the required computations are both expensive and time-consuming. At the same time, the comparatively fast simulation of dark matter requires fewer resources, which has led to the emergence of machine learning algorithms for baryon inpainting as an active area of research; here, recreating the scatter found in hydrodynamic simulations is an ongoing challenge. This paper presents the first application of physics-informed neural networks to baryon inpainting by combining advances in neural network architectures with physical constraints, injecting theory on baryon conversion efficiency into the model loss function. We also introduce a punitive prediction comparison based on the Kullback-Leibler divergence, which enforces scatter reproduction. By simultaneously extracting the complete set of baryonic properties for the Simba suite of cosmological simulations, our results demonstrate improved accuracy of baryonic predictions based on dark matter halo properties, successful recovery of the fundamental metallicity relation, and retrieve scatter that traces the target simulation's distribution.
△ Less
Submitted 19 October, 2023; v1 submitted 24 March, 2023;
originally announced March 2023.
-
Applications and Techniques for Fast Machine Learning in Science
Authors:
Allison McCarn Deiana,
Nhan Tran,
Joshua Agar,
Michaela Blott,
Giuseppe Di Guglielmo,
Javier Duarte,
Philip Harris,
Scott Hauck,
Mia Liu,
Mark S. Neubauer,
Jennifer Ngadiuba,
Seda Ogrenci-Memik,
Maurizio Pierini,
Thea Aarrestad,
Steffen Bahr,
Jurgen Becker,
Anne-Sophie Berthold,
Richard J. Bonventre,
Tomas E. Muller Bravo,
Markus Diefenthaler,
Zhen Dong,
Nick Fritzsche,
Amir Gholami,
Ekaterina Govorkova,
Kyle J Hazelwood
, et al. (62 additional authors not shown)
Abstract:
In this community review report, we discuss applications and techniques for fast machine learning (ML) in science -- the concept of integrating power ML methods into the real-time experimental data processing loop to accelerate scientific discovery. The material for the report builds on two workshops held by the Fast ML for Science community and covers three main areas: applications for fast ML ac…
▽ More
In this community review report, we discuss applications and techniques for fast machine learning (ML) in science -- the concept of integrating power ML methods into the real-time experimental data processing loop to accelerate scientific discovery. The material for the report builds on two workshops held by the Fast ML for Science community and covers three main areas: applications for fast ML across a number of scientific domains; techniques for training and implementing performant and resource-efficient ML algorithms; and computing architectures, platforms, and technologies for deploying these algorithms. We also present overlapping challenges across the multiple scientific domains where common solutions can be found. This community report is intended to give plenty of examples and inspiration for scientific discovery through integrated and accelerated ML solutions. This is followed by a high-level overview and organization of technical advances, including an abundance of pointers to source material, which can enable these breakthroughs.
△ Less
Submitted 25 October, 2021;
originally announced October 2021.
-
Learning Abstract Task Representations
Authors:
Mikhail M. Meskhi,
Adriano Rivolli,
Rafael G. Mantovani,
Ricardo Vilalta
Abstract:
A proper form of data characterization can guide the process of learning-algorithm selection and model-performance estimation. The field of meta-learning has provided a rich body of work describing effective forms of data characterization using different families of meta-features (statistical, model-based, information-theoretic, topological, etc.). In this paper, we start with the abundant set of…
▽ More
A proper form of data characterization can guide the process of learning-algorithm selection and model-performance estimation. The field of meta-learning has provided a rich body of work describing effective forms of data characterization using different families of meta-features (statistical, model-based, information-theoretic, topological, etc.). In this paper, we start with the abundant set of existing meta-features and propose a method to induce new abstract meta-features as latent variables in a deep neural network. We discuss the pitfalls of using traditional meta-features directly and argue for the importance of learning high-level task properties. We demonstrate our methodology using a deep neural network as a feature extractor. We demonstrate that 1) induced meta-models mapping abstract meta-features to generalization performance outperform other methods by ~18% on average, and 2) abstract meta-features attain high feature-relevance scores.
△ Less
Submitted 28 January, 2021; v1 submitted 19 January, 2021;
originally announced January 2021.
-
Active learning with RESSPECT: Resource allocation for extragalactic astronomical transients
Authors:
Noble Kennamer,
Emille E. O. Ishida,
Santiago Gonzalez-Gaitan,
Rafael S. de Souza,
Alexander Ihler,
Kara Ponder,
Ricardo Vilalta,
Anais Moller,
David O. Jones,
Mi Dai,
Alberto Krone-Martins,
Bruno Quint,
Sreevarsha Sreejith,
Alex I. Malz,
Lluis Galbany
Abstract:
The recent increase in volume and complexity of available astronomical data has led to a wide use of supervised machine learning techniques. Active learning strategies have been proposed as an alternative to optimize the distribution of scarce labeling resources. However, due to the specific conditions in which labels can be acquired, fundamental assumptions, such as sample representativeness and…
▽ More
The recent increase in volume and complexity of available astronomical data has led to a wide use of supervised machine learning techniques. Active learning strategies have been proposed as an alternative to optimize the distribution of scarce labeling resources. However, due to the specific conditions in which labels can be acquired, fundamental assumptions, such as sample representativeness and labeling cost stability cannot be fulfilled. The Recommendation System for Spectroscopic follow-up (RESSPECT) project aims to enable the construction of optimized training samples for the Rubin Observatory Legacy Survey of Space and Time (LSST), taking into account a realistic description of the astronomical data environment. In this work, we test the robustness of active learning techniques in a realistic simulated astronomical data scenario. Our experiment takes into account the evolution of training and pool samples, different costs per object, and two different sources of budget. Results show that traditional active learning strategies significantly outperform random sampling. Nevertheless, more complex batch strategies are not able to significantly overcome simple uncertainty sampling techniques. Our findings illustrate three important points: 1) active learning strategies are a powerful tool to optimize the label-acquisition task in astronomy, 2) for upcoming large surveys like LSST, such techniques allow us to tailor the construction of the training sample for the first day of the survey, and 3) the peculiar data environment related to the detection of astronomical transients is a fertile ground that calls for the development of tailored machine learning algorithms.
△ Less
Submitted 26 October, 2020; v1 submitted 12 October, 2020;
originally announced October 2020.
-
Algorithms and Statistical Models for Scientific Discovery in the Petabyte Era
Authors:
Brian Nord,
Andrew J. Connolly,
Jamie Kinney,
Jeremy Kubica,
Gautaum Narayan,
Joshua E. G. Peek,
Chad Schafer,
Erik J. Tollerud,
Camille Avestruz,
G. Jogesh Babu,
Simon Birrer,
Douglas Burke,
João Caldeira,
Douglas A. Caldwell,
Joleen K. Carlberg,
Yen-Chi Chen,
Chuanfei Dong,
Eric D. Feigelson,
V. Zach Golkhou,
Vinay Kashyap,
T. S. Li,
Thomas Loredo,
Luisa Lucie-Smith,
Kaisey S. Mandel,
J. R. Martínez-Galarza
, et al. (13 additional authors not shown)
Abstract:
The field of astronomy has arrived at a turning point in terms of size and complexity of both datasets and scientific collaboration. Commensurately, algorithms and statistical models have begun to adapt --- e.g., via the onset of artificial intelligence --- which itself presents new challenges and opportunities for growth. This white paper aims to offer guidance and ideas for how we can evolve our…
▽ More
The field of astronomy has arrived at a turning point in terms of size and complexity of both datasets and scientific collaboration. Commensurately, algorithms and statistical models have begun to adapt --- e.g., via the onset of artificial intelligence --- which itself presents new challenges and opportunities for growth. This white paper aims to offer guidance and ideas for how we can evolve our technical and collaborative frameworks to promote efficient algorithmic development and take advantage of opportunities for scientific discovery in the petabyte era. We discuss challenges for discovery in large and complex data sets; challenges and requirements for the next stage of development of statistical methodologies and algorithmic tool sets; how we might change our paradigms of collaboration and education; and the ethical implications of scientists' contributions to widely applicable algorithms and computational modeling. We start with six distinct recommendations that are supported by the commentary following them. This white paper is related to a larger corpus of effort that has taken place within and around the Petabytes to Science Workshops (https://petabytestoscience.github.io/).
△ Less
Submitted 4 November, 2019;
originally announced November 2019.
-
Transfer Learning in Astronomy: A New Machine-Learning Paradigm
Authors:
Ricardo Vilalta
Abstract:
The widespread dissemination of machine learning tools in science, particularly in astronomy, has revealed the limitation of working with simple single-task scenarios in which any task in need of a predictive model is looked in isolation, and ignores the existence of other similar tasks. In contrast, a new generation of techniques is emerging where predictive models can take advantage of previous…
▽ More
The widespread dissemination of machine learning tools in science, particularly in astronomy, has revealed the limitation of working with simple single-task scenarios in which any task in need of a predictive model is looked in isolation, and ignores the existence of other similar tasks. In contrast, a new generation of techniques is emerging where predictive models can take advantage of previous experience to leverage information from similar tasks. The new emerging area is referred to as transfer learning. In this paper, I briefly describe the motivation behind the use of transfer learning techniques, and explain how such techniques can be used to solve popular problems in astronomy. As an example, a prevalent problem in astronomy is to estimate the class of an object (e.g., Supernova Ia) using a generation of photometric light-curve datasets where data abounds, but class labels are scarce; such analysis can benefit from spectroscopic data where class labels are known with high confidence, but the data sample is small. Transfer learning provides a robust and practical solution to leverage information from one domain to improve the accuracy of a model built on a different domain. In the example above, transfer learning would look to overcome the difficulty in the compatibility of models between spectroscopic data and photometric data, since data properties such as size, class priors, and underlying distributions, are all expected to be significantly different.
△ Less
Submitted 20 December, 2018;
originally announced December 2018.
-
A General Approach to Domain Adaptation with Applications in Astronomy
Authors:
Ricardo Vilalta,
Kinjal Dhar Gupta,
Dainis Boumber,
Mikhail M. Meskhi
Abstract:
The ability to build a model on a source task and subsequently adapt such model on a new target task is a pervasive need in many astronomical applications. The problem is generally known as transfer learning in machine learning, where domain adaptation is a popular scenario. An example is to build a predictive model on spectroscopic data to identify Supernovae IA, while subsequently trying to adap…
▽ More
The ability to build a model on a source task and subsequently adapt such model on a new target task is a pervasive need in many astronomical applications. The problem is generally known as transfer learning in machine learning, where domain adaptation is a popular scenario. An example is to build a predictive model on spectroscopic data to identify Supernovae IA, while subsequently trying to adapt such model on photometric data. In this paper we propose a new general approach to domain adaptation that does not rely on the proximity of source and target distributions. Instead we simply assume a strong similarity in model complexity across domains, and use active learning to mitigate the dependency on source examples. Our work leads to a new formulation for the likelihood as a function of empirical error using a theoretical learning bound; the result is a novel mapping from generalization error to a likelihood estimation. Results using two real astronomical problems, Supernova Ia classification and identification of Mars landforms, show two main advantages with our approach: increased accuracy performance and substantial savings in computational cost.
△ Less
Submitted 20 December, 2018;
originally announced December 2018.
-
Conceptual Domain Adaptation Using Deep Learning
Authors:
Behrang Mehrparvar,
Ricardo Vilalta
Abstract:
Deep learning has recently been shown to be instrumental in the problem of domain adaptation, where the goal is to learn a model on a target domain using a similar --but not identical-- source domain. The rationale for coupling both techniques is the possibility of extracting common concepts across domains. Considering (strictly) local representations, traditional deep learning assumes common conc…
▽ More
Deep learning has recently been shown to be instrumental in the problem of domain adaptation, where the goal is to learn a model on a target domain using a similar --but not identical-- source domain. The rationale for coupling both techniques is the possibility of extracting common concepts across domains. Considering (strictly) local representations, traditional deep learning assumes common concepts must be captured in the same hidden units. We contend that jointly training a model with source and target data using a single deep network is prone to failure when there is inherently lower-level representational discrepancy between the two domains; such discrepancy leads to a misalignment of corresponding concepts in separate hidden units. We introduce a search framework to correctly align high-level representations when training deep networks; such framework leads to the notion of conceptual --as opposed to representational-- domain adaptation.
△ Less
Submitted 16 August, 2018;
originally announced August 2018.
-
End-to-end 5G services via an SDN/NFV-based multi-tenant network and cloud testbed
Authors:
Raul Muñoz,
Josep Mangues-Bafalluy,
Nikolaos Bartzoudis,
Ricard Vilalta,
Ricardo Martínez,
Ramon Casellas,
Nicola Baldo,
José Núñez-Martínez,
Manuel Requena-Esteso,
Oriol Font-Bach,
Marco Miozzo,
Pol Henarejos,
Ana Pérez-Neira,
Miquel Payaró
Abstract:
5G has a main requirement of highly flexible, ultralow latency and ultra-high bandwidth virtualized infrastructure in order to deliver end-to-end services. This requirement can be met by efficiently integrating all network segments (radio access, aggregation and core) with heterogeneous wireless and optical technologies (5G, mmWave, LTE/LTE-A, Wi-Fi, Ethernet, MPLS, WDM, software-defined optical t…
▽ More
5G has a main requirement of highly flexible, ultralow latency and ultra-high bandwidth virtualized infrastructure in order to deliver end-to-end services. This requirement can be met by efficiently integrating all network segments (radio access, aggregation and core) with heterogeneous wireless and optical technologies (5G, mmWave, LTE/LTE-A, Wi-Fi, Ethernet, MPLS, WDM, software-defined optical transmission, etc.), and massive computing and storage cloud services (offered in edge/core data centers). This paper introduces the preliminary architecture aiming at integrating three consolidated and standalone experimental infrastructures at CTTC, in order to deploy the required end-to-end top-to-bottom converged infrastructure pointed out above for testing and developing advanced 5G services.
△ Less
Submitted 20 March, 2018;
originally announced March 2018.
-
The CTTC 5G end-to-end experimental platform: Integrating heterogeneous wireless/optical networks, distributed cloud, and IoT devices
Authors:
Raul Muñóz,
Josep Mangues,
Ricard Vilalta,
Christos Verikoukis,
Jesús Alonso-Zarate,
Nikolaos Bartzoudis,
Apostolos Georgiadis,
Miquel Payaró,
Ana Pérez-Neira,
Ramon Casellas,
Ricardo Martínez,
José Núñez-Martínez,
Manuel Requena-Esteso,
David Pubill,
Oriol Font-Bach,
Pol Henarejos,
Jordi Serra,
Francisco Vazquez-Gallego
Abstract:
The Internet of Things (IoT) will facilitate a wide variety of applications in different domains, such as smart cities, smart grids, industrial automation (Industry 4.0), smart driving, assistance of the elderly, and home automation. Billions of heterogeneous smart devices with different application requirements will be connected to the networks and will generate huge aggregated volumes of data th…
▽ More
The Internet of Things (IoT) will facilitate a wide variety of applications in different domains, such as smart cities, smart grids, industrial automation (Industry 4.0), smart driving, assistance of the elderly, and home automation. Billions of heterogeneous smart devices with different application requirements will be connected to the networks and will generate huge aggregated volumes of data that will be processed in distributed cloud infrastructures. On the other hand, there is also a general trend to deploy functions as software (SW) instances in cloud infrastructures [e.g., network function virtualization (NFV) or mobile edge computing (MEC)]. Thus, the next generation of mobile networks, the fifth-generation (5G), will need not only to develop new radio interfaces or waveforms to cope with the expected traffic growth but also to integrate heterogeneous networks from end to end (E2E) with distributed cloud resources to deliver E2E IoT and mobile services. This article presents the E2E 5G platform that is being developed by the Centre Tecnològic de Telecomunicacions de Catalunya (CTTC), the first known platform capable of reproducing such an ambitious scenario.
△ Less
Submitted 20 March, 2018;
originally announced March 2018.