-
The Importance of the Current Input in Sequence Modeling
Authors:
Christian Oliva,
Luis F. Lago-Fernández
Abstract:
The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this a…
▽ More
The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.
△ Less
Submitted 22 December, 2021;
originally announced December 2021.
-
Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages
Authors:
Christian Oliva,
Luis F. Lago-Fernández
Abstract:
We provide an empirical study of the stability of recurrent neural networks trained to recognize regular languages. When a small amount of noise is introduced into the activation function, the neurons in the recurrent layer tend to saturate in order to compensate the variability. In this saturated regime, analysis of the network activation shows a set of clusters that resemble discrete states in a…
▽ More
We provide an empirical study of the stability of recurrent neural networks trained to recognize regular languages. When a small amount of noise is introduced into the activation function, the neurons in the recurrent layer tend to saturate in order to compensate the variability. In this saturated regime, analysis of the network activation shows a set of clusters that resemble discrete states in a finite state machine. We show that transitions between these states in response to input symbols are deterministic and stable. The networks display a stable behavior for arbitrarily long strings, and when random perturbations are applied to any of the states, they are able to recover and their evolution converges to the original clusters. This observation reinforces the interpretation of the networks as finite automata, with neurons or groups of neurons coding specific and meaningful input patterns.
△ Less
Submitted 18 June, 2020;
originally announced June 2020.
-
Separation of Memory and Processing in Dual Recurrent Neural Networks
Authors:
Christian Oliva,
Luis F. Lago-Fernández
Abstract:
We explore a neural network architecture that stacks a recurrent layer and a feedforward layer that is also connected to the input, and compare it to standard Elman and LSTM architectures in terms of accuracy and interpretability. When noise is introduced into the activation function of the recurrent units, these neurons are forced into a binary activation regime that makes the networks behave muc…
▽ More
We explore a neural network architecture that stacks a recurrent layer and a feedforward layer that is also connected to the input, and compare it to standard Elman and LSTM architectures in terms of accuracy and interpretability. When noise is introduced into the activation function of the recurrent units, these neurons are forced into a binary activation regime that makes the networks behave much as finite automata. The resulting models are simpler, easier to interpret and get higher accuracy on different sample problems, including the recognition of regular languages, the computation of additions in different bases and the generation of arithmetic expressions.
△ Less
Submitted 17 May, 2020;
originally announced May 2020.
-
Charge ordering transition in GdBaCo2O5: evidence of reentrant behavior
Authors:
M. Allieta,
M. Scavini,
L. Lopresti,
M. Coduri,
L. Loconte,
S. Cappelli,
C. Oliva,
P. Ghigna,
P. Pattison,
V. Scagnoli
Abstract:
We present a detailed study on the charge ordering (CO) transition in GdBaCo2O5 system by combining high resolution synchrotron powder/single crystal diffraction with electron paramagnetic resonance (EPR) experiments as a function of temperature. We found a second order structural phase transition at TCO=247 K (Pmmm to Pmma) associated with the onset of long range CO. At Tmin = 1.2TCO, the EPR lin…
▽ More
We present a detailed study on the charge ordering (CO) transition in GdBaCo2O5 system by combining high resolution synchrotron powder/single crystal diffraction with electron paramagnetic resonance (EPR) experiments as a function of temperature. We found a second order structural phase transition at TCO=247 K (Pmmm to Pmma) associated with the onset of long range CO. At Tmin = 1.2TCO, the EPR linewidth rapidly broadens providing evidence of spin fluctuations due to magnetic interactions between Gd3+ ions and antiferromagnetic couplings of Co2+/Co3+ sublattices. This likely indicates that, analogously to manganites, the long-range antiferromagnetic order in GdBaCo2O5 sets in at TCO. Pair distribution function (PDF) analysis of diffraction data revealed signatures of structural inhomogeneities at low temperature. By comparing the average and local bond valences, we found that above TCO the local structure is consistent with a fully random occupation of Co2+ and Co3+ in a 1:1 ratio and with a complete charge ordering below TCO. Below T = 100 K the charge localization is partially melted at the local scale, suggesting a reentrant behavior of CO. This result is supported by the weakening of superstructure reflections and the temperature evolution of EPR linewidth that is consistent with paramagnetic (PM) reentrant behavior reported in the GdBaCo2O5.5 parent compound.
△ Less
Submitted 10 July, 2013;
originally announced July 2013.
