Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov low
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-ofthe-art results of bpc/perplexity to 0.99 on en-wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch 1 .
this paper
works it cites
works citing it
node size = global citations · hover for the full title
Citation edge evidence
- Transformers: State-of-the-Art Natural Language Processing→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- Conformer: Convolution-augmented Transformer for Speech Recognition→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- Longformer: The Long-Document Transformer→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- Natural language processing: state of the art, current trends and challenges→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- Language Models as Knowledge Bases?→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- AutoML: A survey of the state-of-the-art→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- Pre-trained models for natural language processing: A survey→Transformer-XL: Attentive Language Models beyond a Fixed-Length ContextLegacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→Long Short-Term MemoryLegacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at S…Legacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→Neural Machine Translation by Jointly Learning to Align and TranslateLegacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→Recurrent neural network based language modelLegacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Mode…Legacy edge — source detail unavailable
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context→Neural Architecture Search with Reinforcement LearningLegacy edge — source detail unavailable
- Natural language processing: state of the art, current trends and challenges→Long Short-Term MemoryLegacy edge — source detail unavailable
- AutoML: A survey of the state-of-the-art→Neural Architecture Search with Reinforcement LearningLegacy edge — source detail unavailable
- Language Models as Knowledge Bases?→Long Short-Term MemoryLegacy edge — source detail unavailable
- Language Models as Knowledge Bases?→AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at S…Legacy edge — source detail unavailable
- Language Models as Knowledge Bases?→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- Language Models as Knowledge Bases?→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- Transformers: State-of-the-Art Natural Language Processing→AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at S…Legacy edge — source detail unavailable
- Transformers: State-of-the-Art Natural Language Processing→Longformer: The Long-Document TransformerLegacy edge — source detail unavailable
- Transformers: State-of-the-Art Natural Language Processing→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- Pre-trained models for natural language processing: A survey→Long Short-Term MemoryLegacy edge — source detail unavailable
- Pre-trained models for natural language processing: A survey→Neural Architecture Search with Reinforcement LearningLegacy edge — source detail unavailable
- Pre-trained models for natural language processing: A survey→AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at S…Legacy edge — source detail unavailable
- Pre-trained models for natural language processing: A survey→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- Pre-trained models for natural language processing: A survey→Language Models as Knowledge Bases?Legacy edge — source detail unavailable
- Longformer: The Long-Document Transformer→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- Conformer: Convolution-augmented Transformer for Speech Recognition→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning→AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at S…Legacy edge — source detail unavailable
- ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→Recurrent neural network based language modelLegacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→Long Short-Term MemoryLegacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→Language Models as Knowledge Bases?Legacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→Transformers: State-of-the-Art Natural Language ProcessingLegacy edge — source detail unavailable
- On the Opportunities and Risks of Foundation Models→Longformer: The Long-Document TransformerLegacy edge — source detail unavailable
What this paper cites, inside the corpus
What cites it, inside the corpus
Links
Download BibTeX · Download RIS · DOI · Open access copy · OpenAlex record
Topics
| Topic Modeling | Computer Science |
| Natural Language Processing Techniques | Computer Science |
| Speech Recognition and Synthesis | Computer Science |
Is this record sound?
complete
Nothing in this record contradicts itself and no field we check is missing.
- supports6 author record(s) attached.
- supports80 reference(s) recorded.
- neutralThe DOI carries no year to check against.
- supportsA title is present.
Provenance
sha256 a08467ae9504f237…