-
Accelerating Deep Learning with Dynamic Data Pruning
Authors:
Ravi S Raju,
Kyle Daruwalla,
Mikko Lipasti
Abstract:
Deep learning's success has been attributed to the training of large, overparameterized models on massive amounts of data. As this trend continues, model training has become prohibitively costly, requiring access to powerful computing systems to train state-of-the-art networks. A large body of research has been devoted to addressing the cost per iteration of training through various model compress…
▽ More
Deep learning's success has been attributed to the training of large, overparameterized models on massive amounts of data. As this trend continues, model training has become prohibitively costly, requiring access to powerful computing systems to train state-of-the-art networks. A large body of research has been devoted to addressing the cost per iteration of training through various model compression techniques like pruning and quantization. Less effort has been spent targeting the number of iterations. Previous work, such as forget scores and GraNd/EL2N scores, address this problem by identifying important samples within a full dataset and pruning the remaining samples, thereby reducing the iterations per epoch. Though these methods decrease the training time, they use expensive static scoring algorithms prior to training. When accounting for the scoring mechanism, the total run time is often increased. In this work, we address this shortcoming with dynamic data pruning algorithms. Surprisingly, we find that uniform random dynamic pruning can outperform the prior work at aggressive pruning rates. We attribute this to the existence of "sometimes" samples -- points that are important to the learned decision boundary only some of the training time. To better exploit the subtlety of sometimes samples, we propose two algorithms, based on reinforcement learning techniques, to dynamically prune samples and achieve even higher accuracy than the random dynamic method. We test all our methods against a full-dataset baseline and the prior work on CIFAR-10 and CIFAR-100, and we can reduce the training time by up to 2x without significant performance loss. Our results suggest that data pruning should be understood as a dynamic process that is closely tied to a model's training trajectory, instead of a static step based solely on the dataset alone.
△ Less
Submitted 24 November, 2021;
originally announced November 2021.
-
End to End Bangla Speech Synthesis
Authors:
Prithwiraj Bhattacharjee,
Rajan Saha Raju,
Arif Ahmad,
M. Shahidur Rahman
Abstract:
Text-to-Speech (TTS) system is a system where speech is synthesized from a given text following any particular approach. Concatenative synthesis, Hidden Markov Model (HMM) based synthesis, Deep Learning (DL) based synthesis with multiple building blocks, etc. are the main approaches for implementing a TTS system. Here, we are presenting our deep learning-based end-to-end Bangla speech synthesis sy…
▽ More
Text-to-Speech (TTS) system is a system where speech is synthesized from a given text following any particular approach. Concatenative synthesis, Hidden Markov Model (HMM) based synthesis, Deep Learning (DL) based synthesis with multiple building blocks, etc. are the main approaches for implementing a TTS system. Here, we are presenting our deep learning-based end-to-end Bangla speech synthesis system. It has been implemented with minimal human annotation using only 3 major components (Encoder, Decoder, Post-processing net including waveform synthesis). It does not require any frontend preprocessor and Grapheme-to-Phoneme (G2P) converter. Our model has been trained with phonetically balanced 20 hours of single speaker speech data. It has obtained a 3.79 Mean Opinion Score (MOS) on a scale of 5.0 as subjective evaluation and a 0.77 Perceptual Evaluation of Speech Quality(PESQ) score on a scale of [-0.5, 4.5] as objective evaluation. It is outperforming all existing non-commercial state-of-the-art Bangla TTS systems based on naturalness.
△ Less
Submitted 1 August, 2021;
originally announced August 2021.
-
Aadhaar Card: Challenges and Impact on Digital Transformation
Authors:
Raja Siddharth Raju,
Sukhdev Singh,
Kiran Khatter
Abstract:
Objectives: This paper presents a brief review on Aadhaar card, and discusses the scope and advantages of linking Aadhaar card to various systems. Further we present various cases in which Aadhaar card may pose security threats. The observations of Supreme Court of India are also presented in this paper followed by a discussion on the loopholes in the existing system. Methods: We conducted literat…
▽ More
Objectives: This paper presents a brief review on Aadhaar card, and discusses the scope and advantages of linking Aadhaar card to various systems. Further we present various cases in which Aadhaar card may pose security threats. The observations of Supreme Court of India are also presented in this paper followed by a discussion on the loopholes in the existing system. Methods: We conducted literature survey based on the various research articles, leading newspapers, case studies and the observations of Supreme Court of India, and categorized the various cases into three categories. Findings: Aadhaar project is one of the significant projects in India to bring the universal trend of digital innovation. The launch of this project was focused on the inter-operability of various e-governance functionalities to ensure the optimal utilization of Information, Communication and Technology Infrastructure. Towards this Government of India has recently made Aadhaar card mandatory for many government applications, and also has promoted Aadhaar enabled transactions.
Improvements: There are many issues related to security and privacy of the Aadhaar data need to be addressed. This paper highlights such cases.
△ Less
Submitted 16 August, 2017;
originally announced August 2017.
