Big Bird: Transformers for Longer Sequences
Authors:
Manzil Zaheer,
Guru Guruganesh,
Avinava Dubey,
Joshua Ainslie,
Chris Alberti,
Santiago Ontanon,
Philip Pham,
Anirudh Ravula,
Qifan Wang,
Li Yang,
Amr Ahmed
Abstract:
Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that…
▽ More
Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.
△ Less
Submitted 8 January, 2021; v1 submitted 28 July, 2020;
originally announced July 2020.
ETC: Encoding Long and Structured Inputs in Transformers
Authors:
Joshua Ainslie,
Santiago Ontanon,
Chris Alberti,
Vaclav Cvicek,
Zachary Fisher,
Philip Pham,
Anirudh Ravula,
Sumit Sanghai,
Qifan Wang,
Li Yang
Abstract:
Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-…
▽ More
Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.
△ Less
Submitted 27 October, 2020; v1 submitted 17 April, 2020;
originally announced April 2020.
