-
LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks
Authors:
Evgeny Hershkovitch Neiterman,
Gil Ben-Artzi
Abstract:
Training very deep convolutional networks is challenging, requiring significant computational resources and time. Existing acceleration methods often depend on specific architectures or require network modifications. We introduce LayerDropBack (LDB), a simple yet effective method to accelerate training across a wide range of deep networks. LDB introduces randomness only in the backward pass, maint…
▽ More
Training very deep convolutional networks is challenging, requiring significant computational resources and time. Existing acceleration methods often depend on specific architectures or require network modifications. We introduce LayerDropBack (LDB), a simple yet effective method to accelerate training across a wide range of deep networks. LDB introduces randomness only in the backward pass, maintaining the integrity of the forward pass, guaranteeing that the same network is used during both training and inference. LDB can be seamlessly integrated into the training process of any model without altering its architecture, making it suitable for various network topologies. Our extensive experiments across multiple architectures (ViT, Swin Transformer, EfficientNet, DLA) and datasets (CIFAR-100, ImageNet) show significant training time reductions of 16.93\% to 23.97\%, while preserving or even enhancing model accuracy. Code is available at \url{https://github.com/neiterman21/LDB}.
△ Less
Submitted 23 December, 2024;
originally announced December 2024.
-
ChannelDropBack: Forward-Consistent Stochastic Regularization for Deep Networks
Authors:
Evgeny Hershkovitch Neiterman,
Gil Ben-Artzi
Abstract:
Incorporating stochasticity into the training process of deep convolutional networks is a widely used technique to reduce overfitting and improve regularization. Existing techniques often require modifying the architecture of the network by adding specialized layers, are effective only to specific network topologies or types of layers - linear or convolutional, and result in a trained model that i…
▽ More
Incorporating stochasticity into the training process of deep convolutional networks is a widely used technique to reduce overfitting and improve regularization. Existing techniques often require modifying the architecture of the network by adding specialized layers, are effective only to specific network topologies or types of layers - linear or convolutional, and result in a trained model that is different from the deployed one. We present ChannelDropBack, a simple stochastic regularization approach that introduces randomness only into the backward information flow, leaving the forward pass intact. ChannelDropBack randomly selects a subset of channels within the network during the backpropagation step and applies weight updates only to them. As a consequence, it allows for seamless integration into the training process of any model and layers without the need to change its architecture, making it applicable to various network topologies, and the exact same network is deployed during training and inference. Experimental evaluations validate the effectiveness of our approach, demonstrating improved accuracy on popular datasets and models, including ImageNet and ViT. Code is available at \url{https://github.com/neiterman21/ChannelDropBack.git}.
△ Less
Submitted 23 November, 2024; v1 submitted 16 November, 2024;
originally announced November 2024.
-
Adaptive Enhancement of Extreme Low-Light Images
Authors:
Evgeny Hershkovitch Neiterman,
Michael Klyuchka,
Gil Ben-Artzi
Abstract:
Existing methods for enhancing dark images captured in a very low-light environment assume that the intensity level of the optimal output image is known and already included in the training set. However, this assumption often does not hold, leading to output images that contain visual imperfections such as dark regions or low contrast. To facilitate the training and evaluation of adaptive models t…
▽ More
Existing methods for enhancing dark images captured in a very low-light environment assume that the intensity level of the optimal output image is known and already included in the training set. However, this assumption often does not hold, leading to output images that contain visual imperfections such as dark regions or low contrast. To facilitate the training and evaluation of adaptive models that can overcome this limitation, we have created a dataset of 1500 raw images taken in both indoor and outdoor low-light conditions. Based on our dataset, we introduce a deep learning model capable of enhancing input images with a wide range of intensity levels at runtime, including ones that are not seen during training. Our experimental results demonstrate that our proposed dataset combined with our model can consistently and effectively enhance images across a wide range of diverse and challenging scenarios.
△ Less
Submitted 4 April, 2023; v1 submitted 7 December, 2020;
originally announced December 2020.
