-
Asymptotics of Learning with Deep Structured (Random) Features
Authors:
Dominik Schröder,
Daniil Dmitriev,
Hugo Cui,
Bruno Loureiro
Abstract:
For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of training samples are proportionally large. This characterization is formulated in terms of the population covariance of the features. Our work is partially motivated…
▽ More
For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of training samples are proportionally large. This characterization is formulated in terms of the population covariance of the features. Our work is partially motivated by the problem of learning with Gaussian rainbow neural networks, namely deep non-linear fully-connected networks with random but structured weights, whose row-wise covariances are further allowed to depend on the weights of previous layers. For such networks we also derive a closed-form formula for the feature covariance in terms of the weight matrices. We further find that in some cases our results can capture feature maps learned by deep, finite-width neural networks trained under gradient descent.
△ Less
Submitted 10 June, 2024; v1 submitted 21 February, 2024;
originally announced February 2024.
-
A different approach to introducing statistical mechanics
Authors:
Thomas A. Moore,
Daniel V. Schroeder
Abstract:
The basic notions of statistical mechanics (microstates, multiplicities) are quite simple, but understanding how the second law arises from these ideas requires working with cumbersomely large numbers. To avoid getting bogged down in mathematics, one can compute multiplicities numerically for a simple model system such as an Einstein solid -- a collection of identical quantum harmonic oscillators.…
▽ More
The basic notions of statistical mechanics (microstates, multiplicities) are quite simple, but understanding how the second law arises from these ideas requires working with cumbersomely large numbers. To avoid getting bogged down in mathematics, one can compute multiplicities numerically for a simple model system such as an Einstein solid -- a collection of identical quantum harmonic oscillators. A computer spreadsheet program or comparable software can compute the required combinatoric functions for systems containing a few hundred oscillators and units of energy. When two such systems can exchange energy, one immediately sees that some configurations are overwhelmingly more probable than others. Graphs of entropy vs. energy for the two systems can be used to motivate the theoretical definition of temperature, $T= (\partial S/\partial U)^{-1}$, thus bridging the gap between the classical and statistical approaches to entropy. Further spreadsheet exercises can be used to compute the heat capacity of an Einstein solid, study the Boltzmann distribution, and explore the properties of a two-state paramagnetic system.
△ Less
Submitted 24 February, 2015;
originally announced February 2015.
-
Interactive molecular dynamics
Authors:
Daniel V. Schroeder
Abstract:
Physics students now have access to interactive molecular dynamics simulations that can model and animate the motions of hundreds of particles, such as noble gas atoms, that attract each other weakly at short distances but repel strongly when pressed together. Using these simulations, students can develop an understanding of forces and motions at the molecular scale, nonideal fluids, phases of mat…
▽ More
Physics students now have access to interactive molecular dynamics simulations that can model and animate the motions of hundreds of particles, such as noble gas atoms, that attract each other weakly at short distances but repel strongly when pressed together. Using these simulations, students can develop an understanding of forces and motions at the molecular scale, nonideal fluids, phases of matter, thermal equilibrium, nonequilibrium states, the Boltzmann distribution, the arrow of time, and much more. This article summarizes the basic features and capabilities of such a simulation, presents a variety of student exercises using it at the introductory and intermediate levels, and describes some enhancements that can further extend its uses. A working simulation code, in HTML5 and JavaScript for running within any modern Web browser, is provided as an online supplement.
△ Less
Submitted 21 February, 2015;
originally announced February 2015.
