-
HW/SW Codesign for Robust and Efficient Binarized SNNs by Capacitor Minimization
Authors:
Mikail Yayla,
Simon Thomann,
Ming-Liang Wei,
Chia-Lin Yang,
Jian-Jia Chen,
Hussam Amrouch
Abstract:
Using accelerators based on analog computing is an efficient way to process the immensely large workloads in Neural Networks (NNs). One example of an analog computing scheme for NNs is Integrate-and-Fire (IF) Spiking Neural Networks (SNNs). However, to achieve high inference accuracy in IF-SNNs, the analog hardware needs to represent current-based multiply-accumulate (MAC) levels as spike times, f…
▽ More
Using accelerators based on analog computing is an efficient way to process the immensely large workloads in Neural Networks (NNs). One example of an analog computing scheme for NNs is Integrate-and-Fire (IF) Spiking Neural Networks (SNNs). However, to achieve high inference accuracy in IF-SNNs, the analog hardware needs to represent current-based multiply-accumulate (MAC) levels as spike times, for which a large membrane capacitor needs to be charged for a certain amount of time. A large capacitor results in high energy use, considerable area cost, and long latency, constituting one of the major bottlenecks in analog IF-SNN implementations. In this work, we propose a HW/SW Codesign method, called CapMin, for capacitor size minimization in analog computing IF-SNNs. CapMin minimizes the capacitor size by reducing the number of spike times needed for accurate operation of the HW, based on the absolute frequency of MAC level occurrences in the SW. To increase the operation of IF-SNNs to current variation, we propose the method CapMin-V, which trades capacitor size for protection based on the reduced capacitor size found in CapMin. In our experiments, CapMin achieves more than a 14$\times$ reduction in capacitor size over the state of the art, while CapMin-V achieves increased variation tolerance in the IF-SNN operation, requiring only a small increase in capacitor size.
△ Less
Submitted 5 September, 2023;
originally announced September 2023.
-
HEP-BNN: A Framework for Finding Low-Latency Execution Configurations of BNNs on Heterogeneous Multiprocessor Platforms
Authors:
Leonard David Bereholschi,
Ching-Chi Lin,
Mikail Yayla,
Jian-Jia Chen
Abstract:
Binarized Neural Networks (BNNs) significantly reduce the computation and memory demands with binarized weights and activations compared to full-precision NNs. Executing a layer in a BNN on different devices of a heterogeneous multiprocessor platform consisting of CPU and GPU can affect the inference performance, i.e., accuracy and latency. Usually, a heterogeneous HW platform consisting of a CPU…
▽ More
Binarized Neural Networks (BNNs) significantly reduce the computation and memory demands with binarized weights and activations compared to full-precision NNs. Executing a layer in a BNN on different devices of a heterogeneous multiprocessor platform consisting of CPU and GPU can affect the inference performance, i.e., accuracy and latency. Usually, a heterogeneous HW platform consisting of a CPU and a GPU is available to execute the BNN workloads. However, to use the heterogeneous HW effectively, it is necessary to find an efficient strategy for BNN workload mapping. In this work, we propose a framework that generates efficient BNN layer-to-device mappings (i.e. suitable parallel configuration for each layer of the model) for execution platforms comprised of CPU and CUDA-capable GPU. We evaluate our proposed framework with two BNN architectures using two well-known datasets, Fashion-MNIST and CIFAR-10, on three hardware platforms with different characteristics. The results show that compared to running a fully-parallelized GPU implementation, our framework generates an efficient configuration up to 2x, 2.6x and 11.8x faster on our tested hardware respectively.
△ Less
Submitted 12 January, 2023;
originally announced January 2023.
-
Universal Approximation Theorems of Fully Connected Binarized Neural Networks
Authors:
Mikail Yayla,
Mario Günzel,
Burim Ramosaj,
Jian-Jia Chen
Abstract:
Neural networks (NNs) are known for their high predictive accuracy in complex learning problems. Beside practical advantages, NNs also indicate favourable theoretical properties such as universal approximation (UA) theorems. Binarized Neural Networks (BNNs) significantly reduce time and memory demands by restricting the weight and activation domains to two values. Despite the practical advantages,…
▽ More
Neural networks (NNs) are known for their high predictive accuracy in complex learning problems. Beside practical advantages, NNs also indicate favourable theoretical properties such as universal approximation (UA) theorems. Binarized Neural Networks (BNNs) significantly reduce time and memory demands by restricting the weight and activation domains to two values. Despite the practical advantages, theoretical guarantees based on UA theorems of BNNs are rather sparse in the literature. We close this gap by providing UA theorems for fully connected BNNs under the following scenarios: (1) for binarized inputs, UA can be constructively achieved under one hidden layer; (2) for inputs with real numbers, UA can not be achieved under one hidden layer but can be constructively achieved under two hidden layers for Lipschitz-continuous functions. Our results indicate that fully connected BNNs can approximate functions universally, under certain conditions.
△ Less
Submitted 4 February, 2021;
originally announced February 2021.
-
Bit Error Tolerance Metrics for Binarized Neural Networks
Authors:
Sebastian Buschjäger,
Jian-Jia Chen,
Kuan-Hsun Chen,
Mario Günzel,
Katharina Morik,
Rodion Novkin,
Lukas Pfahler,
Mikail Yayla
Abstract:
To reduce the resource demand of neural network (NN) inference systems, it has been proposed to use approximate memory, in which the supply voltage and the timing parameters are tuned trading accuracy with energy consumption and performance. Tuning these parameters aggressively leads to bit errors, which can be tolerated by NNs when bit flips are injected during training. However, bit flip trainin…
▽ More
To reduce the resource demand of neural network (NN) inference systems, it has been proposed to use approximate memory, in which the supply voltage and the timing parameters are tuned trading accuracy with energy consumption and performance. Tuning these parameters aggressively leads to bit errors, which can be tolerated by NNs when bit flips are injected during training. However, bit flip training, which is the state of the art for achieving bit error tolerance, does not scale well; it leads to massive overheads and cannot be applied for high bit error rates (BERs). Alternative methods to achieve bit error tolerance in NNs are needed, but the underlying principles behind the bit error tolerance of NNs have not been reported yet. With this lack of understanding, further progress in the research on NN bit error tolerance will be restrained.
In this study, our objective is to investigate the internal changes in the NNs that bit flip training causes, with a focus on binarized NNs (BNNs). To this end, we quantify the properties of bit error tolerant BNNs with two metrics. First, we propose a neuron-level bit error tolerance metric, which calculates the margin between the pre-activation values and batch normalization thresholds. Secondly, to capture the effects of bit error tolerance on the interplay of neurons, we propose an inter-neuron bit error tolerance metric, which measures the importance of each neuron and computes the variance over all importance values. Our experimental results support that these two metrics are strongly related to bit error tolerance.
△ Less
Submitted 2 February, 2021;
originally announced February 2021.
-
Towards Explainable Bit Error Tolerance of Resistive RAM-Based Binarized Neural Networks
Authors:
Sebastian Buschjäger,
Jian-Jia Chen,
Kuan-Hsun Chen,
Mario Günzel,
Christian Hakert,
Katharina Morik,
Rodion Novkin,
Lukas Pfahler,
Mikail Yayla
Abstract:
Non-volatile memory, such as resistive RAM (RRAM), is an emerging energy-efficient storage, especially for low-power machine learning models on the edge. It is reported, however, that the bit error rate of RRAMs can be up to 3.3% in the ultra low-power setting, which might be crucial for many use cases. Binary neural networks (BNNs), a resource efficient variant of neural networks (NNs), can toler…
▽ More
Non-volatile memory, such as resistive RAM (RRAM), is an emerging energy-efficient storage, especially for low-power machine learning models on the edge. It is reported, however, that the bit error rate of RRAMs can be up to 3.3% in the ultra low-power setting, which might be crucial for many use cases. Binary neural networks (BNNs), a resource efficient variant of neural networks (NNs), can tolerate a certain percentage of errors without a loss in accuracy and demand lower resources in computation and storage. The bit error tolerance (BET) in BNNs can be achieved by flipping the weight signs during training, as proposed by Hirtzlin et al., but their method has a significant drawback, especially for fully connected neural networks (FCNN): The FCNNs overfit to the error rate used in training, which leads to low accuracy under lower error rates. In addition, the underlying principles of BET are not investigated. In this work, we improve the training for BET of BNNs and aim to explain this property. We propose straight-through gradient approximation to improve the weight-sign-flip training, by which BNNs adapt less to the bit error rates. To explain the achieved robustness, we define a metric that aims to measure BET without fault injection. We evaluate the metric and find that it correlates with accuracy over error rate for all FCNNs tested. Finally, we explore the influence of a novel regularizer that optimizes with respect to this metric, with the aim of providing a configurable trade-off in accuracy and BET.
△ Less
Submitted 3 February, 2020;
originally announced February 2020.
