A phase transition in sampling from Restricted Boltzmann Machines
Authors:
Youngwoo Kwon,
Qian Qin,
Guanyang Wang,
Yuchen Wei
Abstract:
Restricted Boltzmann Machines are a class of undirected graphical models that play a key role in deep learning and unsupervised learning. In this study, we prove a phase transition phenomenon in the mixing time of the Gibbs sampler for a one-parameter Restricted Boltzmann Machine. Specifically, the mixing time varies logarithmically, polynomially, and exponentially with the number of vertices depe…
▽ More
Restricted Boltzmann Machines are a class of undirected graphical models that play a key role in deep learning and unsupervised learning. In this study, we prove a phase transition phenomenon in the mixing time of the Gibbs sampler for a one-parameter Restricted Boltzmann Machine. Specifically, the mixing time varies logarithmically, polynomially, and exponentially with the number of vertices depending on whether the parameter $c$ is above, equal to, or below a critical value $c_\star\approx-5.87$. A key insight from our analysis is the link between the Gibbs sampler and a dynamical system, which we utilize to quantify the former based on the behavior of the latter. To study the critical case $c= c_\star$, we develop a new isoperimetric inequality for the sampler's stationary distribution by showing that the distribution is nearly log-concave.
△ Less
Submitted 10 October, 2024;
originally announced October 2024.
Spectral integrated neural networks (SINNs) for solving forward and inverse dynamic problems
Authors:
Lin Qiu,
Fajie Wang,
Wenzhen Qu,
Yan Gu,
Qing-Hua Qin
Abstract:
This paper proposes a novel neural network framework, denoted as spectral integrated neural networks (SINNs), for resolving three-dimensional forward and inverse dynamic problems. In the SINNs, the spectral integration method is applied to perform temporal discretization, and then a fully connected neural network is adopted to solve resulting partial differential equations (PDEs) in the spatial do…
▽ More
This paper proposes a novel neural network framework, denoted as spectral integrated neural networks (SINNs), for resolving three-dimensional forward and inverse dynamic problems. In the SINNs, the spectral integration method is applied to perform temporal discretization, and then a fully connected neural network is adopted to solve resulting partial differential equations (PDEs) in the spatial domain. Specifically, spatial coordinates are employed as inputs in the network architecture, and the output layer is configured with multiple outputs, each dedicated to approximating solutions at different time instances characterized by Gaussian points used in the spectral method. By leveraging the automatic differentiation technique and spectral integration scheme, the SINNs minimize the loss function, constructed based on the governing PDEs and boundary conditions, to obtain solutions for dynamic problems. Additionally, we utilize polynomial basis functions to expand the unknown function, aiming to enhance the performance of SINNs in addressing inverse problems. The conceived framework is tested on six forward and inverse dynamic problems, involving nonlinear PDEs. Numerical results demonstrate the superior performance of SINNs over the popularly used physics-informed neural networks in terms of convergence speed, computational accuracy and efficiency. It is also noteworthy that the SINNs exhibit the capability to deliver accurate and stable solutions for long-time dynamic problems.
△ Less
Submitted 7 January, 2024;
originally announced January 2024.
