-
Hebbian Deep Learning Without Feedback
Authors:
Adrien Journé,
Hector Garcia Rodriguez,
Qinghai Guo,
Timoleon Moraitis
Abstract:
Recent approximations to backpropagation (BP) have mitigated many of BP's computational inefficiencies and incompatibilities with biology, but important limitations still remain. Moreover, the approximations significantly decrease accuracy in benchmarks, suggesting that an entirely different approach may be more fruitful. Here, grounded on recent theory for Hebbian learning in soft winner-take-all…
▽ More
Recent approximations to backpropagation (BP) have mitigated many of BP's computational inefficiencies and incompatibilities with biology, but important limitations still remain. Moreover, the approximations significantly decrease accuracy in benchmarks, suggesting that an entirely different approach may be more fruitful. Here, grounded on recent theory for Hebbian learning in soft winner-take-all networks, we present multilayer SoftHebb, i.e. an algorithm that trains deep neural networks, without any feedback, target, or error signals. As a result, it achieves efficiency by avoiding weight transport, non-local plasticity, time-locking of layer updates, iterative equilibria, and (self-) supervisory or other feedback signals -- which were necessary in other approaches. Its increased efficiency and biological compatibility do not trade off accuracy compared to state-of-the-art bio-plausible learning, but rather improve it. With up to five hidden layers and an added linear classifier, accuracies on MNIST, CIFAR-10, STL-10, and ImageNet, respectively reach 99.4%, 80.3%, 76.2%, and 27.3%. In conclusion, SoftHebb shows with a radically different approach from BP that Deep Learning over few layers may be plausible in the brain and increases the accuracy of bio-plausible machine learning. Code is available at https://github.com/NeuromorphicComputing/SoftHebb.
△ Less
Submitted 2 August, 2023; v1 submitted 23 September, 2022;
originally announced September 2022.
-
Short-Term Plasticity Neurons Learning to Learn and Forget
Authors:
Hector Garcia Rodriguez,
Qinghai Guo,
Timoleon Moraitis
Abstract:
Short-term plasticity (STP) is a mechanism that stores decaying memories in synapses of the cerebral cortex. In computing practice, STP has been used, but mostly in the niche of spiking neurons, even though theory predicts that it is the optimal solution to certain dynamic tasks. Here we present a new type of recurrent neural unit, the STP Neuron (STPN), which indeed turns out strikingly powerful.…
▽ More
Short-term plasticity (STP) is a mechanism that stores decaying memories in synapses of the cerebral cortex. In computing practice, STP has been used, but mostly in the niche of spiking neurons, even though theory predicts that it is the optimal solution to certain dynamic tasks. Here we present a new type of recurrent neural unit, the STP Neuron (STPN), which indeed turns out strikingly powerful. Its key mechanism is that synapses have a state, propagated through time by a self-recurrent connection-within-the-synapse. This formulation enables training the plasticity with backpropagation through time, resulting in a form of learning to learn and forget in the short term. The STPN outperforms all tested alternatives, i.e. RNNs, LSTMs, other models with fast weights, and differentiable plasticity. We confirm this in both supervised and reinforcement learning (RL), and in tasks such as Associative Retrieval, Maze Exploration, Atari video games, and MuJoCo robotics. Moreover, we calculate that, in neuromorphic or biological circuits, the STPN minimizes energy consumption across models, as it depresses individual synapses dynamically. Based on these, biological STP may have been a strong evolutionary attractor that maximizes both efficiency and computational power. The STPN now brings these neuromorphic advantages also to a broad spectrum of machine learning practice. Code is available at https://github.com/NeuromorphicComputing/stpn
△ Less
Submitted 28 June, 2022;
originally announced June 2022.
-
Performance Analysis of ANFIS in short term Wind Speed Prediction
Authors:
Ernesto Cortés Pérez,
Ignacio Algredo-Badillo,
Víctor Hugo García Rodríguez
Abstract:
Results are presented on the performance of Adaptive Neuro-Fuzzy Inference system (ANFIS) for wind velocity forecasts in the Isthmus of Tehuantepec region in the state of Oaxaca, Mexico. The data bank was provided by the meteorological station located at the University of Isthmus, Tehuantepec campus, and this data bank covers the period from 2008 to 2011. Three data models were constructed to carr…
▽ More
Results are presented on the performance of Adaptive Neuro-Fuzzy Inference system (ANFIS) for wind velocity forecasts in the Isthmus of Tehuantepec region in the state of Oaxaca, Mexico. The data bank was provided by the meteorological station located at the University of Isthmus, Tehuantepec campus, and this data bank covers the period from 2008 to 2011. Three data models were constructed to carry out 16, 24 and 48 hours forecasts using the following variables: wind velocity, temperature, barometric pressure, and date. The performance measure for the three models is the mean standard error (MSE). In this work, performance analysis in short-term prediction is presented, because it is essential in order to define an adequate wind speed model for eolian parks, where a right planning provide economic benefits.
△ Less
Submitted 11 December, 2012;
originally announced December 2012.
