The fine line between dead neurons and sparsity in binarized spiking neural networks
Authors:
Jason K. Eshraghian,
Wei D. Lu
Abstract:
Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increas…
▽ More
Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets, despite using binarized weights. Source code is available at https://github.com/jeshraghian/snn-tha/
△ Less
Submitted 27 January, 2022;
originally announced January 2022.
A Real-Time Retinomorphic Simulator Using a Conductance-Based Discrete Neuronal Network
Authors:
Jason K. Eshraghian,
Seungbum Baek,
Wesley Thio,
Yulia Sandamirskaya,
Herbert H. C. Iu,
Wei D. Lu
Abstract:
We present an optimized conductance-based retina microcircuit simulator which transforms light stimuli into a series of graded and spiking action potentials through photo transduction. We use discrete retinal neuron blocks based on a collation of single-compartment models and morphologically realistic formulations, and successfully achieve a biologically real-time simulator. This is done by optimi…
▽ More
We present an optimized conductance-based retina microcircuit simulator which transforms light stimuli into a series of graded and spiking action potentials through photo transduction. We use discrete retinal neuron blocks based on a collation of single-compartment models and morphologically realistic formulations, and successfully achieve a biologically real-time simulator. This is done by optimizing the numerical methods employed to solve the system of over 270 nonlinear ordinary differential equations and parameters. Our simulator includes some of the most recent advances in compartmental modeling to include five intrinsic ion currents of each cell whilst ensuring real-time performance, in attaining the ion-current and membrane responses of the photoreceptor rod and cone cells, the bipolar and amacrine cells, their laterally connected electrical and chemical synapses, and the output ganglion cell. It exhibits dynamical retinal behavior such as spike-frequency adaptation, rebound activation, fast-spiking, and subthreshold responsivity. Light stimuli incident at the photoreceptor rod and cone cells is modulated through the system of differential equations, enabling the user to probe the neuronal response at any point in the network. This is in contrast to many other retina encoding schemes which prefer to `black-box' the preceding stages to the spike train output. Our simulator is made available open source, with the hope that it will benefit neuroscientists and machine learning practitioners in better understanding the retina sub-circuitries, how retina cells optimize the representation of visual information, and in generating large datasets of biologically accurate graded and spiking responses.
△ Less
Submitted 26 December, 2019;
originally announced January 2020.
