-
Time-based Sequence Model for Personalization and Recommendation Systems
Authors:
Tigran Ishkhanov,
Maxim Naumov,
Xianjie Chen,
Yan Zhu,
Yuan Zhong,
Alisson Gusatti Azzolini,
Chonglin Sun,
Frank Jiang,
Andrey Malevich,
Liang Xiong
Abstract:
In this paper we develop a novel recommendation model that explicitly incorporates time information. The model relies on an embedding layer and TSL attention-like mechanism with inner products in different vector spaces, that can be thought of as a modification of multi-headed attention. This mechanism allows the model to efficiently treat sequences of user behavior of different length. We study t…
▽ More
In this paper we develop a novel recommendation model that explicitly incorporates time information. The model relies on an embedding layer and TSL attention-like mechanism with inner products in different vector spaces, that can be thought of as a modification of multi-headed attention. This mechanism allows the model to efficiently treat sequences of user behavior of different length. We study the properties of our state-of-the-art model on statistically designed data set. Also, we show that it outperforms more complex models with longer sequence length on the Taobao User Behavior dataset.
△ Less
Submitted 27 August, 2020;
originally announced August 2020.
-
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
Authors:
Antonio Ginart,
Maxim Naumov,
Dheevatsa Mudigere,
Jiyan Yang,
James Zou
Abstract:
Embedding representations power machine intelligence in many applications, including recommendation systems, but they are space intensive -- potentially occupying hundreds of gigabytes in large-scale settings. To help manage this outsized memory consumption, we explore mixed dimension embeddings, an embedding layer architecture in which a particular embedding vector's dimension scales with its que…
▽ More
Embedding representations power machine intelligence in many applications, including recommendation systems, but they are space intensive -- potentially occupying hundreds of gigabytes in large-scale settings. To help manage this outsized memory consumption, we explore mixed dimension embeddings, an embedding layer architecture in which a particular embedding vector's dimension scales with its query frequency. Through theoretical analysis and systematic experiments, we demonstrate that using mixed dimensions can drastically reduce the memory usage, while maintaining and even improving the ML performance. Empirically, we show that the proposed mixed dimension layers improve accuracy by 0.1% using half as many parameters or maintain it using 16X fewer parameters for click-through rate prediction task on the Criteo Kaggle dataset.
△ Less
Submitted 8 February, 2021; v1 submitted 25 September, 2019;
originally announced September 2019.
-
Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
Authors:
Hao-Jun Michael Shi,
Dheevatsa Mudigere,
Maxim Naumov,
Jiyan Yang
Abstract:
Modern deep learning-based recommendation systems exploit hundreds to thousands of different categorical features, each with millions of different categories ranging from clicks to posts. To respect the natural diversity within the categorical data, embeddings map each category to a unique dense representation within an embedded space. Since each categorical feature could take on as many as tens o…
▽ More
Modern deep learning-based recommendation systems exploit hundreds to thousands of different categorical features, each with millions of different categories ranging from clicks to posts. To respect the natural diversity within the categorical data, embeddings map each category to a unique dense representation within an embedded space. Since each categorical feature could take on as many as tens of millions of different possible categories, the embedding tables form the primary memory bottleneck during both training and inference. We propose a novel approach for reducing the embedding size in an end-to-end fashion by exploiting complementary partitions of the category set to produce a unique embedding vector for each category without explicit definition. By storing multiple smaller embedding tables based on each complementary partition and combining embeddings from each table, we define a unique embedding for each category at smaller memory cost. This approach may be interpreted as using a specific fixed codebook to ensure uniqueness of each category's representation. Our experimental results demonstrate the effectiveness of our approach over the hashing trick for reducing the size of the embedding tables in terms of model loss and accuracy, while retaining a similar reduction in the number of parameters.
△ Less
Submitted 28 June, 2020; v1 submitted 4 September, 2019;
originally announced September 2019.
-
On the Dimensionality of Embeddings for Sparse Features and Data
Authors:
Maxim Naumov
Abstract:
In this note we discuss a common misconception, namely that embeddings are always used to reduce the dimensionality of the item space. We show that when we measure dimensionality in terms of information entropy then the embedding of sparse probability distributions, that can be used to represent sparse features or data, may or not reduce the dimensionality of the item space. However, the embedding…
▽ More
In this note we discuss a common misconception, namely that embeddings are always used to reduce the dimensionality of the item space. We show that when we measure dimensionality in terms of information entropy then the embedding of sparse probability distributions, that can be used to represent sparse features or data, may or not reduce the dimensionality of the item space. However, the embeddings do provide a different and often more meaningful representation of the items for a particular task at hand. Also, we give upper bounds and more precise guidelines for choosing the embedding dimension.
△ Less
Submitted 7 January, 2019;
originally announced January 2019.
-
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
Authors:
Jongsoo Park,
Maxim Naumov,
Protonu Basu,
Summer Deng,
Aravind Kalaiah,
Daya Khudia,
James Law,
Parth Malani,
Andrey Malevich,
Satish Nadathur,
Juan Pino,
Martin Schatz,
Alexander Sidorov,
Viswanath Sivakumar,
Andrew Tulloch,
Xiaodong Wang,
Yiming Wu,
Hector Yuen,
Utku Diril,
Dmytro Dzhulgakov,
Kim Hazelwood,
Bill Jia,
Yangqing Jia,
Lin Qiao,
Vijay Rao
, et al. (3 additional authors not shown)
Abstract:
The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions…
▽ More
The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers.
△ Less
Submitted 29 November, 2018; v1 submitted 24 November, 2018;
originally announced November 2018.
-
On Periodic Functions as Regularizers for Quantization of Neural Networks
Authors:
Maxim Naumov,
Utku Diril,
Jongsoo Park,
Benjamin Ray,
Jedrzej Jablonski,
Andrew Tulloch
Abstract:
Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply th…
▽ More
Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply these functions component-wise and add the sum over the model parameters as a regularizer to the model loss during training. The frequency and amplitude hyper-parameters of these functions can be adjusted during training. The regularization pushes the weights into discrete points that can be encoded as integers. We show that using this technique the resulting quantized models exhibit the same accuracy as the original ones on CIFAR-10 and ImageNet datasets.
△ Less
Submitted 24 November, 2018;
originally announced November 2018.
-
Bandana: Using Non-volatile Memory for Storing Deep Learning Models
Authors:
Assaf Eisenman,
Maxim Naumov,
Darryl Gardner,
Misha Smelyanskiy,
Sergey Pupyrev,
Kim Hazelwood,
Asaf Cidon,
Sachin Katti
Abstract:
Typical large-scale recommender systems use deep learning models that are stored on a large amount of DRAM. These models often rely on embeddings, which consume most of the required memory. We present Bandana, a storage system that reduces the DRAM footprint of embeddings, by using Non-volatile Memory (NVM) as the primary storage medium, with a small amount of DRAM as cache. The main challenge in…
▽ More
Typical large-scale recommender systems use deep learning models that are stored on a large amount of DRAM. These models often rely on embeddings, which consume most of the required memory. We present Bandana, a storage system that reduces the DRAM footprint of embeddings, by using Non-volatile Memory (NVM) as the primary storage medium, with a small amount of DRAM as cache. The main challenge in storing embeddings on NVM is its limited read bandwidth compared to DRAM. Bandana uses two primary techniques to address this limitation: first, it stores embedding vectors that are likely to be read together in the same physical location, using hypergraph partitioning, and second, it decides the number of embedding vectors to cache in DRAM by simulating dozens of small caches. These techniques allow Bandana to increase the effective read bandwidth of NVM by 2-3x and thereby significantly reduce the total cost of ownership.
△ Less
Submitted 14 November, 2018; v1 submitted 14 November, 2018;
originally announced November 2018.
-
AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
Authors:
Aditya Devarakonda,
Maxim Naumov,
Michael Garland
Abstract:
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a new training approach that, rather than statically choosing a single batch size f…
▽ More
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a new training approach that, rather than statically choosing a single batch size for all epochs, adaptively increases the batch size during the training process. Our method delivers the convergence rate of small batch sizes while achieving performance similar to large batch sizes. We analyse our approach using the standard AlexNet, ResNet, and VGG networks operating on the popular CIFAR-10, CIFAR-100, and ImageNet datasets. Our results demonstrate that learning with adaptive batch sizes can improve performance by factors of up to 6.25 on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than 1% relative to training with fixed batch sizes.
△ Less
Submitted 13 February, 2018; v1 submitted 5 December, 2017;
originally announced December 2017.
