-
TQCompressor: improving tensor decomposition methods in neural networks via permutations
Authors:
V. Abronin,
A. Naumov,
D. Mazur,
D. Bystrov,
K. Tsarova,
Ar. Melnikov,
I. Oseledets,
S. Dolgov,
R. Brasher,
M. Perelshtein
Abstract:
We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associ…
▽ More
We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associated with factorization. We demonstrate this method applied to the GPT-2$_{small}$. The result of the compression is TQCompressedGPT-2 model, featuring 81 mln. parameters compared to 124 mln. in the GPT-2$_{small}$. We make TQCompressedGPT-2 publicly available. We further enhance the performance of the TQCompressedGPT-2 through a training strategy involving multi-step knowledge distillation, using only a 3.1% of the OpenWebText. TQCompressedGPT-2 surpasses DistilGPT-2 and KnGPT-2 in comparative evaluations, marking an advancement in the efficient and effective deployment of models in resource-constrained environments.
△ Less
Submitted 29 January, 2024;
originally announced January 2024.
-
Real-time on-device nod and shake recognition
Authors:
Elmar H. Langholz,
Reuben Brasher
Abstract:
We discuss methods for teaching systems to identify gestures such as head nod and shake. We use iPhone X depth camera to gather data and later use similar data as input for a working app. These methods have proved robust for training with limited datasets and thus we make the argument that similar methods could be adapted to learn other human to human non-verbal gestures. We showcase how to augmen…
▽ More
We discuss methods for teaching systems to identify gestures such as head nod and shake. We use iPhone X depth camera to gather data and later use similar data as input for a working app. These methods have proved robust for training with limited datasets and thus we make the argument that similar methods could be adapted to learn other human to human non-verbal gestures. We showcase how to augment Euler angle gesture sequences to train models with a relatively large number of parameters such as LSTM and GRU and gain better performance than reported for smaller models such as HMM. In the examples here, we demonstrate how to train such models with Keras and run the resulting models real time on device with CoreML.
△ Less
Submitted 12 June, 2018;
originally announced June 2018.
-
Sometimes You Want to Go Where Everybody Knows your Name
Authors:
Reuben Brasher,
Nat Roth,
Justin Wagle
Abstract:
We introduce a new metric for measuring how well a model personalizes to a user's specific preferences. We define personalization as a weighting between performance on user specific data and performance on a more general global dataset that represents many different users. This global term serves as a form of regularization that forces us to not overfit to individual users who have small amounts o…
▽ More
We introduce a new metric for measuring how well a model personalizes to a user's specific preferences. We define personalization as a weighting between performance on user specific data and performance on a more general global dataset that represents many different users. This global term serves as a form of regularization that forces us to not overfit to individual users who have small amounts of data. In order to protect user privacy, we add the constraint that we may not centralize or share user data. We also contribute a simple experiment in which we simulate classifying sentiment for users with very distinct vocabularies. This experiment functions as an example of the tension between doing well globally on all users, and doing well on any specific individual user. It also provides a concrete example of how to employ our new metric to help reason about and resolve this tension. We hope this work can help frame and ground future work into personalization.
△ Less
Submitted 30 January, 2018;
originally announced January 2018.
