-
Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems
Authors:
Hunter S. Heidenreich,
Pantelis R. Vlachas,
Petros Koumoutsakos
Abstract:
Machine learning architectures, including transformers and recurrent neural networks (RNNs) have revolutionized forecasting in applications ranging from text processing to extreme weather. Notably, advanced network architectures, tuned for applications such as natural language processing, are transferable to other tasks such as spatiotemporal forecasting tasks. However, there is a scarcity of abla…
▽ More
Machine learning architectures, including transformers and recurrent neural networks (RNNs) have revolutionized forecasting in applications ranging from text processing to extreme weather. Notably, advanced network architectures, tuned for applications such as natural language processing, are transferable to other tasks such as spatiotemporal forecasting tasks. However, there is a scarcity of ablation studies to illustrate the key components that enable this forecasting accuracy. The absence of such studies, although explainable due to the associated computational cost, intensifies the belief that these models ought to be considered as black boxes. In this work, we decompose the key architectural components of the most powerful neural architectures, namely gating and recurrence in RNNs, and attention mechanisms in transformers. Then, we synthesize and build novel hybrid architectures from the standard blocks, performing ablation studies to identify which mechanisms are effective for each task. The importance of considering these components as hyper-parameters that can augment the standard architectures is exhibited on various forecasting datasets, from the spatiotemporal chaotic dynamics of the multiscale Lorenz 96 system, the Kuramoto-Sivashinsky equation, as well as standard real world time-series benchmarks. A key finding is that neural gating and attention improves the performance of all standard RNNs in most tasks, while the addition of a notion of recurrence in transformers is detrimental. Furthermore, our study reveals that a novel, sparsely used, architecture which integrates Recurrent Highway Networks with neural gating and attention mechanisms, emerges as the best performing architecture in high-dimensional spatiotemporal forecasting of dynamical systems.
△ Less
Submitted 3 October, 2024;
originally announced October 2024.
-
Mixed Noise and Posterior Estimation with Conditional DeepGEM
Authors:
Paul Hagemann,
Johannes Hertrich,
Maren Casfor,
Sebastian Heidenreich,
Gabriele Steidl
Abstract:
Motivated by indirect measurements and applications from nanometrology with a mixed noise model, we develop a novel algorithm for jointly estimating the posterior and the noise parameters in Bayesian inverse problems. We propose to solve the problem by an expectation maximization (EM) algorithm. Based on the current noise parameters, we learn in the E-step a conditional normalizing flow that appro…
▽ More
Motivated by indirect measurements and applications from nanometrology with a mixed noise model, we develop a novel algorithm for jointly estimating the posterior and the noise parameters in Bayesian inverse problems. We propose to solve the problem by an expectation maximization (EM) algorithm. Based on the current noise parameters, we learn in the E-step a conditional normalizing flow that approximates the posterior. In the M-step, we propose to find the noise parameter updates again by an EM algorithm, which has analytical formulas. We compare the training of the conditional normalizing flow with the forward and reverse KL, and show that our model is able to incorporate information from many measurements, unlike previous approaches.
△ Less
Submitted 5 July, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.
-
EigenNoise: A Contrastive Prior to Warm-Start Representations
Authors:
Hunter Scott Heidenreich,
Jake Ryland Williams
Abstract:
In this work, we present a naive initialization scheme for word vectors based on a dense, independent co-occurrence model and provide preliminary results that suggest it is competitive and warrants further investigation. Specifically, we demonstrate through information-theoretic minimum description length (MDL) probing that our model, EigenNoise, can approach the performance of empirically trained…
▽ More
In this work, we present a naive initialization scheme for word vectors based on a dense, independent co-occurrence model and provide preliminary results that suggest it is competitive and warrants further investigation. Specifically, we demonstrate through information-theoretic minimum description length (MDL) probing that our model, EigenNoise, can approach the performance of empirically trained GloVe despite the lack of any pre-training data (in the case of EigenNoise). We present these preliminary results with interest to set the stage for further investigations into how this competitive initialization works without pre-training data, as well as to invite the exploration of more intelligent initialization schemes informed by the theory of harmonic linguistic structure. Our application of this theory likewise contributes a novel (and effective) interpretation of recent discoveries which have elucidated the underlying distributional information that linguistic representations capture from data and contrast distributions.
△ Less
Submitted 9 May, 2022;
originally announced May 2022.
-
To Know by the Company Words Keep and What Else Lies in the Vicinity
Authors:
Jake Ryland Williams,
Hunter Scott Heidenreich
Abstract:
The development of state-of-the-art (SOTA) Natural Language Processing (NLP) systems has steadily been establishing new techniques to absorb the statistics of linguistic data. These techniques often trace well-known constructs from traditional theories, and we study these connections to close gaps around key NLP methods as a means to orient future work. For this, we introduce an analytic model of…
▽ More
The development of state-of-the-art (SOTA) Natural Language Processing (NLP) systems has steadily been establishing new techniques to absorb the statistics of linguistic data. These techniques often trace well-known constructs from traditional theories, and we study these connections to close gaps around key NLP methods as a means to orient future work. For this, we introduce an analytic model of the statistics learned by seminal algorithms (including GloVe and Word2Vec), and derive insights for systems that use these algorithms and the statistics of co-occurrence, in general. In this work, we derive -- to the best of our knowledge -- the first known solution to Word2Vec's softmax-optimized, skip-gram algorithm. This result presents exciting potential for future development as a direct solution to a deep learning (DL) language model's (LM's) matrix factorization. However, we use the solution to demonstrate a seemingly-universal existence of a property that word vectors exhibit and which allows for the prophylactic discernment of biases in data -- prior to their absorption by DL models. To qualify our work, we conduct an analysis of independence, i.e., on the density of statistical dependencies in co-occurrence models, which in turn renders insights on the distributional hypothesis' partial fulfillment by co-occurrence statistics.
△ Less
Submitted 29 April, 2022;
originally announced May 2022.
-
Invertible Neural Networks versus MCMC for Posterior Reconstruction in Grazing Incidence X-Ray Fluorescence
Authors:
Anna Andrle,
Nando Farchmin,
Paul Hagemann,
Sebastian Heidenreich,
Victor Soltwisch,
Gabriele Steidl
Abstract:
Grazing incidence X-ray fluorescence is a non-destructive technique for analyzing the geometry and compositional parameters of nanostructures appearing e.g. in computer chips. In this paper, we propose to reconstruct the posterior parameter distribution given a noisy measurement generated by the forward model by an appropriately learned invertible neural network. This network resembles the transpo…
▽ More
Grazing incidence X-ray fluorescence is a non-destructive technique for analyzing the geometry and compositional parameters of nanostructures appearing e.g. in computer chips. In this paper, we propose to reconstruct the posterior parameter distribution given a noisy measurement generated by the forward model by an appropriately learned invertible neural network. This network resembles the transport map from a reference distribution to the posterior. We demonstrate by numerical comparisons that our method can compete with established Markov Chain Monte Carlo approaches, while being more efficient and flexible in applications.
△ Less
Submitted 5 February, 2021;
originally announced February 2021.
-
Investigating Coordinated 'Social' Targeting of High-Profile Twitter Accounts
Authors:
Hunter Scott Heidenreich,
Munif Ishad Mujib,
Jake Ryland Williams
Abstract:
Following the 2016 US presidential election, there has been an increased focus on politically-motivated manipulation of mass-user behavior on social media platforms. Since a large volume of political discussion occurs on these platforms, identifying malicious activity and coordinated campaigns is essential to ensuring a robust democratic environment. Twitter has become a critical communication cha…
▽ More
Following the 2016 US presidential election, there has been an increased focus on politically-motivated manipulation of mass-user behavior on social media platforms. Since a large volume of political discussion occurs on these platforms, identifying malicious activity and coordinated campaigns is essential to ensuring a robust democratic environment. Twitter has become a critical communication channel for politicians and other public figures, enabling them to maintain a direct relationship with supporters. However, the platform has been fertile ground for large-scale malicious activity. As the 2020 U.S. presidential election approaches, we have developed tools to monitor follower dynamics of some of the most prominent Twitter users, including U.S. presidential candidates. We investigate numerous, strange phenomena, such as dramatic spike and saw-tooth waveforms on follower-count charts; cohorts of user accounts which 'circulate', i.e., re-follow high profile accounts numerous times; and other 'resurrected' accounts, which have recently re-engaged on Twitter after years of non-activity. So through various analyses in these contexts, we reveal multiple, coordinated 'social' targeting campaigns aimed at affecting the outcomes of socially critical events through the use of networks of social automations (bots), often optimizing their social capital through 'compromised' accounts, which have--unbeknownst to the greater world--been hijacked.
△ Less
Submitted 6 August, 2020;
originally announced August 2020.
-
NewsTweet: A Dataset of Social Media Embedding in Online Journalism
Authors:
Munif Ishad Mujib,
Hunter Scott Heidenreich,
Colin J. Murphy,
Giovanni C. Santia,
Asta Zelenkauskaite,
Jake Ryland Williams
Abstract:
The inclusion of social media posts---tweets, in particular---in digital news stories, both as commentary and increasingly as news sources, has become commonplace in recent years. In order to study this phenomenon with sufficient depth, robust large-scale data collection from both news publishers and social media platforms is necessary. This work describes the construction of such a data pipeline.…
▽ More
The inclusion of social media posts---tweets, in particular---in digital news stories, both as commentary and increasingly as news sources, has become commonplace in recent years. In order to study this phenomenon with sufficient depth, robust large-scale data collection from both news publishers and social media platforms is necessary. This work describes the construction of such a data pipeline. In the data collected from Google News, 13% of all stories were found to include embedded tweets, with sports and entertainment news containing the largest volumes of them. Public figures and celebrities are found to dominate these stories; however, relatively unknown users have also been found to achieve newsworthiness. The collected data set, NewsTweet, and the associated pipeline for acquisition stand to engender a wave of new inquiries into social content embedding from multiple research communities.
△ Less
Submitted 6 August, 2020;
originally announced August 2020.
