Symmetry Breaking in Neural Network Optimization: Insights from Input Dimension Expansion
Authors:
Jun-Jie Zhang,
Nan Cheng,
Fu-Peng Li,
Xiu-Cheng Wang,
Jian-Nan Chen,
Long-Gang Pang,
Deyu Meng
Abstract:
Understanding the mechanisms behind neural network optimization is crucial for improving network design and performance. While various optimization techniques have been developed, a comprehensive understanding of the underlying principles that govern these techniques remains elusive. Specifically, the role of symmetry breaking, a fundamental concept in physics, has not been fully explored in neura…
▽ More
Understanding the mechanisms behind neural network optimization is crucial for improving network design and performance. While various optimization techniques have been developed, a comprehensive understanding of the underlying principles that govern these techniques remains elusive. Specifically, the role of symmetry breaking, a fundamental concept in physics, has not been fully explored in neural network optimization. This gap in knowledge limits our ability to design networks that are both efficient and effective. Here, we propose the symmetry breaking hypothesis to elucidate the significance of symmetry breaking in enhancing neural network optimization. We demonstrate that a simple input expansion can significantly improve network performance across various tasks, and we show that this improvement can be attributed to the underlying symmetry breaking mechanism. We further develop a metric to quantify the degree of symmetry breaking in neural networks, providing a practical approach to evaluate and guide network design. Our findings confirm that symmetry breaking is a fundamental principle that underpins various optimization techniques, including dropout, batch normalization, and equivariance. By quantifying the degree of symmetry breaking, our work offers a practical technique for performance enhancement and a metric to guide network design without the need for complete datasets and extensive training processes.
△ Less
Submitted 12 September, 2024; v1 submitted 10 September, 2024;
originally announced September 2024.
General stationary solutions of the nonlocal nonlinear Schrödinger equation and their relevance to the PT-symmetric system
Authors:
Tao Xu,
Yang Chen,
Min Li,
De-Xin Meng
Abstract:
With the stationary solution assumption, we establish the connection between the nonlocal nonlinear Schrödinger (NNLS) equation and an elliptic equation. Then, we obtain the general stationary solutions and discuss the relevance of their smoothness and boundedness to some integral constants. Those solutions, which cover the known results in the literature, include the unbounded Jacobi elliptic-fun…
▽ More
With the stationary solution assumption, we establish the connection between the nonlocal nonlinear Schrödinger (NNLS) equation and an elliptic equation. Then, we obtain the general stationary solutions and discuss the relevance of their smoothness and boundedness to some integral constants. Those solutions, which cover the known results in the literature, include the unbounded Jacobi elliptic-function and hyperbolic-function solutions, the bounded sn-, cn- and dn-function solutions, as well as the hyperbolic soliton solutions. By the imaginary translation transformation of the NNLS equation, we also derive the complex-amplitude stationary solutions, in which all the bounded cases obey either the \PT- or anti-\PT-symmetric relation. In particular, the complex tanh-function solution can exhibit no spatial localization in addition to the dark and anti-dark soliton profiles, which is sharp contrast with the common dark soliton. Considering the physical relevance to \PT-symmetric system, we show that the complex-amplitude stationary solutions can yield a wide class of complex and time-independent \PT-symmetric potentials, and the symmetry breaking does not occur in the \PT-symmetric linear system with the associated potentials.
△ Less
Submitted 31 January, 2020; v1 submitted 26 June, 2019;
originally announced June 2019.
