-
Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation
Authors:
Thang Do,
Sonja Hannibal,
Arnulf Jentzen
Abstract:
Deep learning methods - consisting of a class of deep neural networks (DNNs) trained by a stochastic gradient descent (SGD) optimization method - are nowadays key tools to solve data driven supervised learning problems. Despite the great success of SGD methods in the training of DNNs, it remains a fundamental open problem of research to explain the success and the limitations of such methods in ri…
▽ More
Deep learning methods - consisting of a class of deep neural networks (DNNs) trained by a stochastic gradient descent (SGD) optimization method - are nowadays key tools to solve data driven supervised learning problems. Despite the great success of SGD methods in the training of DNNs, it remains a fundamental open problem of research to explain the success and the limitations of such methods in rigorous theoretical terms. In particular, even in the standard setup of data driven supervised learning problems, it remained an open research problem to prove (or disprove) that SGD methods converge in the training of DNNs with the popular rectified linear unit (ReLU) activation function with high probability to global minimizers in the optimization landscape. In this work we answer this question negatively. Specifically, in this work we prove for a large class of SGD methods that the considered optimizer does with high probability not converge to global minimizers of the optimization problem. It turns out that the probability to not converge to a global minimizer converges at least exponentially quickly to one as the width of the first hidden layer of the ANN and the depth of the ANN, respectively, increase. The general non-convergence results of this work do not only apply to the plain vanilla standard SGD method but also to a large class of accelerated and adaptive SGD methods such as the momentum SGD, the Nesterov accelerated SGD, the Adagrad, the RMSProp, the Adam, the Adamax, the AMSGrad, and the Nadam optimizers.
△ Less
Submitted 14 February, 2025; v1 submitted 14 October, 2024;
originally announced October 2024.
-
Persistent oscillations of the order parameter and interaction quench phase diagram for a confined Bardeen-Cooper-Schrieffer Fermi gas
Authors:
S. Hannibal,
P. Kettmann,
M. D. Croitoru,
V. M. Axt,
T. Kuhn
Abstract:
We present a numerical study of the interaction quench dynamics in a superfluid ultracold Fermi gas confined in a three-dimensional cigar-shaped harmonic trap. In the present paper we investigate the amplitude mode of the superfluid order parameter after interaction quenches which start deep in the BCS phase and end in the BCS-BEC crossover regime. To this end, we exploit the Bogoliubov-de Gennes…
▽ More
We present a numerical study of the interaction quench dynamics in a superfluid ultracold Fermi gas confined in a three-dimensional cigar-shaped harmonic trap. In the present paper we investigate the amplitude mode of the superfluid order parameter after interaction quenches which start deep in the BCS phase and end in the BCS-BEC crossover regime. To this end, we exploit the Bogoliubov-de Gennes formalism which takes the confinement potential explicitly into account and provides a microscopic fully coherent description of the system. We find an anharmonic nonlinear oscillation of the modulus of the superfluid order parameter, i.e., of the Higgs mode. This oscillation persists for large times with only a small amplitude modulation being visible. We connect the frequency and the mean value of this oscillation with the breaking of Cooper pairs in the superfluid phase. Additionally, we demonstrate that the occurrence of this persistent oscillation is connected to the onset of chaotic dynamics in our model. Finally, we calculate an interaction quench phase diagram of the Higgs mode for quenches on the BCS side of the BCS-BEC crossover and discuss its properties as a function of the aspect ratio of the cigar-shaped trap.
△ Less
Submitted 29 May, 2018;
originally announced May 2018.
-
Dynamical vanishing of the order parameter in a confined Bardeen-Cooper-Schrieffer Fermi gas after an interaction quench
Authors:
S. Hannibal,
P. Kettmann,
M. D. Croitoru,
V. M. Axt,
T. Kuhn
Abstract:
We present a numerical study of the Higgs mode in an ultracold confined Fermi gas after an interaction quench and find a dynamical vanishing of the superfluid order parameter. Our calculations are done within a microscopic density-matrix approach in the Bogoliubov-de Gennes framework which takes the three-dimensional cigar-shaped confinement explicitly into account. In this framework, we study the…
▽ More
We present a numerical study of the Higgs mode in an ultracold confined Fermi gas after an interaction quench and find a dynamical vanishing of the superfluid order parameter. Our calculations are done within a microscopic density-matrix approach in the Bogoliubov-de Gennes framework which takes the three-dimensional cigar-shaped confinement explicitly into account. In this framework, we study the amplitude mode of the order parameter after interaction quenches starting on the BCS side of the BEC-BCS crossover close to the transition and ending in the BCS regime. We demonstrate the emergence of a dynamically vanishing superfluid order parameter in the spatiotemporal dynamics in a three-dimensional trap. Further, we show that the signal averaged over the whole trap mirrors the spatiotemporal behavior and allows us to systematically study the effects of the system size and aspect ratio on the observed dynamics. Our analysis enables us to connect the confinement-induced modifications of the dynamics to the pairing properties of the system. Finally, we demonstrate that the signature of the Higgs mode is contained in the dynamical signal of the condensate fraction, which, therefore, might provide a new experimental access to the nonadiabatic regime of the Higgs mode.
△ Less
Submitted 18 May, 2018;
originally announced May 2018.
-
Pure Goldstone mode in the quench dynamics of a confined ultracold Fermi gas in the BCS-BEC crossover regime
Authors:
P. Kettmann,
S. Hannibal,
M. D. Croitoru,
V. M. Axt,
T. Kuhn
Abstract:
We present a numerical study of the dynamic response of a confined superfluid Fermi gas to a rapid change of the scattering length (i.e., an interaction quench). Based on a fully microscopic time-dependent density-matrix approach within the full Bogoliubov-de Gennes formalism that includes a 3D harmonic confinement we simulate and identify the emergence of a Goldstone mode of the BCS gap in a ciga…
▽ More
We present a numerical study of the dynamic response of a confined superfluid Fermi gas to a rapid change of the scattering length (i.e., an interaction quench). Based on a fully microscopic time-dependent density-matrix approach within the full Bogoliubov-de Gennes formalism that includes a 3D harmonic confinement we simulate and identify the emergence of a Goldstone mode of the BCS gap in a cigar-shaped $^6$Li gas. By analyzing this Goldstone mode over a wide range of parameters, we show that its excitation spectrum is gapless and that its main frequency is not fixed by the trapping potential but that it is determined by the details of the quench. Thus, we report the emergence of a pure Goldstone mode of the BCS gap that --in contrast to situations in many previous studies-- maintains its gapless excitation spectrum predicted by the Goldstone theorem. Furthermore, we observe that the size-dependent superfluid resonances resulting from the atypical BCS-BEC crossover have a direct impact on this Goldstone mode. Finally, we find that the interaction quench-induced Goldstone mode leads to a low-frequency in-phase oscillation of the single-particle occupations with complete inversion of the lowest-lying single-particle states which could provide a convenient experimental access to the pure gapless Goldstone mode.
△ Less
Submitted 25 August, 2017;
originally announced August 2017.
-
Quench dynamics of a confined ultracold Fermi gas: Direct visibility of the Goldstone mode in the single-particle excitations
Authors:
P. Kettmann,
S. Hannibal,
M. D. Croitoru,
V. M. Axt,
T. Kuhn
Abstract:
We present a numerical study of a confined ultracold Fermi gas showing that the Goldstone mode of the BCS gap is directly visible in the dynamics of the single-particle excitations. To this end, we investigate the low-energy dynamic response of a confined Fermi gas to a rapid change of the scattering length (i.e., an interaction quench). Based on a fully microscopic time-dependent density-matrix a…
▽ More
We present a numerical study of a confined ultracold Fermi gas showing that the Goldstone mode of the BCS gap is directly visible in the dynamics of the single-particle excitations. To this end, we investigate the low-energy dynamic response of a confined Fermi gas to a rapid change of the scattering length (i.e., an interaction quench). Based on a fully microscopic time-dependent density-matrix approach within the Bogoliubov-de Gennes formalism that includes a 3D harmonic confinement we simulate and identify the emergence of the Goldstone mode in a cigar-shaped $^6$Li gas. We show that the quench leads to a low-frequency in-phase oscillation of the single-particle occupations. Complete inversion is achieved for occupations corresponding to the lowest-lying single-particle states.
△ Less
Submitted 28 August, 2017; v1 submitted 13 November, 2015;
originally announced November 2015.
-
Higgs mode in the quench dynamics of a confined ultracold Fermi gas in the BCS regime
Authors:
S. Hannibal,
P. Kettmann,
M. D. Croitoru,
A. V. Vagov,
V. M. Axt,
T. Kuhn
Abstract:
The Higgs amplitude mode of the order parameter of an ultracold confined Fermi gas in the BCS regime after a quench of the coupling constant is analyzed theoretically. Characteristic features are a damped oscillation which at a certain transition time changes into a rather irregular dynamics. We compare the numerical solution of the full set of nonlinear equations of motion for the normal and anom…
▽ More
The Higgs amplitude mode of the order parameter of an ultracold confined Fermi gas in the BCS regime after a quench of the coupling constant is analyzed theoretically. Characteristic features are a damped oscillation which at a certain transition time changes into a rather irregular dynamics. We compare the numerical solution of the full set of nonlinear equations of motion for the normal and anomalous Bogoliubov quasiparticle excitations with a linearized approximation. In doing so the transition time as well as the difference between resonant systems, i.e., systems where the Fermi energy is close to a subband minimum, and off-resonant systems can be well understood and traced back to the system and geometry parameters.
△ Less
Submitted 8 December, 2014;
originally announced December 2014.
