-
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference
Authors:
Yejin Lee,
Anna Sun,
Basil Hosmer,
Bilge Acun,
Can Balioglu,
Changhan Wang,
Charles David Hernandez,
Christian Puhrsch,
Daniel Haziza,
Driss Guessous,
Francisco Massa,
Jacob Kahn,
Jeffrey Wan,
Jeremy Reizenstein,
Jiaqi Zhai,
Joe Isaacson,
Joel Schlosser,
Juan Pino,
Kaushik Ram Sadagopan,
Leonid Shamis,
Linjian Ma,
Min-Jae Hwang,
Mingda Chen,
Mostafa Elhoushi,
Pedro Rodriguez
, et al. (5 additional authors not shown)
Abstract:
Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system design and optimization opportunities. The technology is capable of understanding and responding in multiple modalities. However, the advanced capability currently comes with significant system resource demands. To susta…
▽ More
Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system design and optimization opportunities. The technology is capable of understanding and responding in multiple modalities. However, the advanced capability currently comes with significant system resource demands. To sustainably scale generative AI capabilities to billions of users in the world, inference must be fast and efficient. This paper pinpoints key system design and optimization opportunities by characterizing a family of emerging multi-modal generation models on real systems. Auto-regressive token generation is a critical latency performance bottleneck, typically dominated by GPU idle time. In addition to memory-intensive attention across the generative AI models, linear operations constitute significant inference latency due to the feed forward networks in Transformer-based models. We demonstrate that state-of-the-art optimization levers, spanning from applications to system software and hardware, set a 3.88x better baseline.
△ Less
Submitted 9 May, 2025; v1 submitted 30 September, 2024;
originally announced October 2024.
-
Is Flash Attention Stable?
Authors:
Alicia Golden,
Samuel Hsia,
Fei Sun,
Bilge Acun,
Basil Hosmer,
Yejin Lee,
Zachary DeVito,
Jeff Johnson,
Gu-Yeon Wei,
David Brooks,
Carole-Jean Wu
Abstract:
Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantify…
▽ More
Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context when downstream effects are difficult to quantify. As a case study, we apply this framework to analyze the widely-adopted Flash Attention optimization. We find that Flash Attention sees roughly an order of magnitude more numeric deviation as compared to Baseline Attention at BF16 when measured during an isolated forward pass. We then use a data-driven analysis based on the Wasserstein Distance to provide upper bounds on how this numeric deviation impacts model weights during training, finding that the numerical deviation present in Flash Attention is 2-5 times less significant than low-precision training.
△ Less
Submitted 4 May, 2024;
originally announced May 2024.
-
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
Authors:
Mostafa Elhoushi,
Akshat Shrivastava,
Diana Liskovich,
Basil Hosmer,
Bram Wasti,
Liangzhen Lai,
Anas Mahmoud,
Bilge Acun,
Saurabh Agarwal,
Ahmed Roman,
Ahmed A Aly,
Beidi Chen,
Carole-Jean Wu
Abstract:
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exi…
▽ More
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and checkpoints at https://github.com/facebookresearch/LayerSkip.
△ Less
Submitted 18 October, 2024; v1 submitted 25 April, 2024;
originally announced April 2024.
-
CHAI: Clustered Head Attention for Efficient LLM Inference
Authors:
Saurabh Agarwal,
Bilge Acun,
Basil Hosmer,
Mostafa Elhoushi,
Yejin Lee,
Shivaram Venkataraman,
Dimitris Papailiopoulos,
Carole-Jean Wu
Abstract:
Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is one of the key components of LLMs, which can account for over 50% of LLMs memory and comput…
▽ More
Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is one of the key components of LLMs, which can account for over 50% of LLMs memory and compute requirement. We observe that there is a high amount of redundancy across heads on which tokens they pay attention to. Based on this insight, we propose Clustered Head Attention (CHAI). CHAI combines heads with a high amount of correlation for self-attention at runtime, thus reducing both memory and compute. In our experiments, we show that CHAI is able to reduce the memory requirements for storing K,V cache by up to 21.4% and inference time latency by up to 1.73x without any fine-tuning required. CHAI achieves this with a maximum 3.2% deviation in accuracy across 3 different models (i.e. OPT-66B, LLAMA-7B, LLAMA-33B) and 5 different evaluation datasets.
△ Less
Submitted 27 April, 2024; v1 submitted 12 March, 2024;
originally announced March 2024.
-
Generative AI Beyond LLMs: System Implications of Multi-Modal Generation
Authors:
Alicia Golden,
Samuel Hsia,
Fei Sun,
Bilge Acun,
Basil Hosmer,
Yejin Lee,
Zachary DeVito,
Jeff Johnson,
Gu-Yeon Wei,
David Brooks,
Carole-Jean Wu
Abstract:
As the development of large-scale Generative AI models evolve beyond text (1D) generation to include image (2D) and video (3D) generation, processing spatial and temporal information presents unique challenges to quality, performance, and efficiency. We present the first work towards understanding this new system design space for multi-modal text-to-image (TTI) and text-to-video (TTV) generation m…
▽ More
As the development of large-scale Generative AI models evolve beyond text (1D) generation to include image (2D) and video (3D) generation, processing spatial and temporal information presents unique challenges to quality, performance, and efficiency. We present the first work towards understanding this new system design space for multi-modal text-to-image (TTI) and text-to-video (TTV) generation models. Current model architecture designs are bifurcated into 2 categories: Diffusion- and Transformer-based models. Our systematic performance characterization on a suite of eight representative TTI/TTV models shows that after state-of-the-art optimization techniques such as Flash Attention are applied, Convolution accounts for up to 44% of execution time for Diffusion-based TTI models, while Linear layers consume up to 49% of execution time for Transformer-based models. We additionally observe that Diffusion-based TTI models resemble the Prefill stage of LLM inference, and benefit from 1.1-2.5x greater speedup from Flash Attention than Transformer-based TTI models that resemble the Decode phase. Since optimizations designed for LLMs do not map directly onto TTI/TTV models, we must conduct a thorough characterization of these workloads to gain insights for new optimization opportunities. In doing so, we define sequence length in the context of TTI/TTV models and observe sequence length can vary up to 4x in Diffusion model inference. We additionally observe temporal aspects of TTV workloads pose unique system bottlenecks, with Temporal Attention accounting for over 60% of total Attention time. Overall, our in-depth system performance characterization is a critical first step towards designing efficient and deployable systems for emerging TTI/TTV workloads.
△ Less
Submitted 5 May, 2024; v1 submitted 21 December, 2023;
originally announced December 2023.
