-
Two Tales of Single-Phase Contrastive Hebbian Learning
Authors:
Rasmus Kjær Høier,
Christopher Zach
Abstract:
The search for ``biologically plausible'' learning algorithms has converged on the idea of representing gradients as activity differences. However, most approaches require a high degree of synchronization (distinct phases during learning) and introduce substantial computational overhead, which raises doubts regarding their biological plausibility as well as their potential utility for neuromorphic…
▽ More
The search for ``biologically plausible'' learning algorithms has converged on the idea of representing gradients as activity differences. However, most approaches require a high degree of synchronization (distinct phases during learning) and introduce substantial computational overhead, which raises doubts regarding their biological plausibility as well as their potential utility for neuromorphic computing. Furthermore, they commonly rely on applying infinitesimal perturbations (nudges) to output units, which is impractical in noisy environments. Recently it has been shown that by modelling artificial neurons as dyads with two oppositely nudged compartments, it is possible for a fully local learning algorithm named ``dual propagation'' to bridge the performance gap to backpropagation, without requiring separate learning phases or infinitesimal nudging. However, the algorithm has the drawback that its numerical stability relies on symmetric nudging, which may be restrictive in biological and analog implementations. In this work we first provide a solid foundation for the objective underlying the dual propagation method, which also reveals a surprising connection with adversarial robustness. Second, we demonstrate how dual propagation is related to a particular adjoint state method, which is stable regardless of asymmetric nudging.
△ Less
Submitted 2 December, 2024; v1 submitted 13 February, 2024;
originally announced February 2024.
-
Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons
Authors:
Rasmus Høier,
D. Staudt,
Christopher Zach
Abstract:
Activity difference based learning algorithms-such as contrastive Hebbian learning and equilibrium propagation-have been proposed as biologically plausible alternatives to error back-propagation. However, on traditional digital chips these algorithms suffer from having to solve a costly inference problem twice, making these approaches more than two orders of magnitude slower than back-propagation.…
▽ More
Activity difference based learning algorithms-such as contrastive Hebbian learning and equilibrium propagation-have been proposed as biologically plausible alternatives to error back-propagation. However, on traditional digital chips these algorithms suffer from having to solve a costly inference problem twice, making these approaches more than two orders of magnitude slower than back-propagation. In the analog realm equilibrium propagation may be promising for fast and energy efficient learning, but states still need to be inferred and stored twice. Inspired by lifted neural networks and compartmental neuron models we propose a simple energy based compartmental neuron model, termed dual propagation, in which each neuron is a dyad with two intrinsic states. At inference time these intrinsic states encode the error/activity duality through their difference and their mean respectively. The advantage of this method is that only a single inference phase is needed and that inference can be solved in layerwise closed-form. Experimentally we show on common computer vision datasets, including Imagenet32x32, that dual propagation performs equivalently to back-propagation both in terms of accuracy and runtime.
△ Less
Submitted 7 June, 2023; v1 submitted 2 February, 2023;
originally announced February 2023.
-
AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks
Authors:
Huu Le,
Rasmus Kjær Høier,
Che-Tsung Lin,
Christopher Zach
Abstract:
We propose a new algorithm for training deep neural networks (DNNs) with binary weights. In particular, we first cast the problem of training binary neural networks (BiNNs) as a bilevel optimization instance and subsequently construct flexible relaxations of this bilevel program. The resulting training method shares its algorithmic simplicity with several existing approaches to train BiNNs, in par…
▽ More
We propose a new algorithm for training deep neural networks (DNNs) with binary weights. In particular, we first cast the problem of training binary neural networks (BiNNs) as a bilevel optimization instance and subsequently construct flexible relaxations of this bilevel program. The resulting training method shares its algorithmic simplicity with several existing approaches to train BiNNs, in particular with the straight-through gradient estimator successfully employed in BinaryConnect and subsequent methods. In fact, our proposed method can be interpreted as an adaptive variant of the original straight-through estimator that conditionally (but not always) acts like a linear mapping in the backward pass of error propagation. Experimental results demonstrate that our new algorithm offers favorable performance compared to existing approaches.
△ Less
Submitted 6 December, 2021;
originally announced December 2021.
-
Lifted Regression/Reconstruction Networks
Authors:
Rasmus Kjær Høier,
Christopher Zach
Abstract:
In this work we propose lifted regression/reconstruction networks (LRRNs), which combine lifted neural networks with a guaranteed Lipschitz continuity property for the output layer. Lifted neural networks explicitly optimize an energy model to infer the unit activations and therefore---in contrast to standard feed-forward neural networks---allow bidirectional feedback between layers. So far lifted…
▽ More
In this work we propose lifted regression/reconstruction networks (LRRNs), which combine lifted neural networks with a guaranteed Lipschitz continuity property for the output layer. Lifted neural networks explicitly optimize an energy model to infer the unit activations and therefore---in contrast to standard feed-forward neural networks---allow bidirectional feedback between layers. So far lifted neural networks have been modelled around standard feed-forward architectures. We propose to take further advantage of the feedback property by letting the layers simultaneously perform regression and reconstruction. The resulting lifted network architecture allows to control the desired amount of Lipschitz continuity, which is an important feature to obtain adversarially robust regression and classification methods. We analyse and numerically demonstrate applications for unsupervised and supervised learning.
△ Less
Submitted 7 May, 2020;
originally announced May 2020.
-
Real-time Accelerator Diagnostic Tools for the MAX IV Storage Rings
Authors:
B. Meirose,
V. Abelin,
F. Bertilsson,
B. E. Bolling,
M. Brandin,
M. Holz,
R. Høier,
A. Johansson,
P. Lilja,
J. S. Lundquist,
S. Molloy,
F. Persson,
J. E. Petersson,
H. Serodio,
R. Svärd,
D. Winchester
Abstract:
In this paper, beam diagnostic and monitoring tools developed by the MAX IV Operations Group are discussed. In particular, new beam position monitoring and accelerator tunes visualization software tools, as well as tools that directly influence the beam quality and stability are introduced. An availability and downtime monitoring application is also presented.
In this paper, beam diagnostic and monitoring tools developed by the MAX IV Operations Group are discussed. In particular, new beam position monitoring and accelerator tunes visualization software tools, as well as tools that directly influence the beam quality and stability are introduced. An availability and downtime monitoring application is also presented.
△ Less
Submitted 15 February, 2020;
originally announced February 2020.
