On the difficulty of training Recurrent Neural Networks

Razvan Pascanu; Tomáš Mikolov; Yoshua Bengio

doi:https://doi.org/10.48550/arxiv.1211.5063

On the difficulty of training Recurrent Neural Networks

2012 · arXiv (Cornell University) · 3,801 citations · 14 from inside this corpus

Razvan Pascanu, Tomáš Mikolov, Yoshua Bengio

The source holds an abstract for this work, but its best open-access copy is under no open licence, which does not permit us to republish the text. Read it at the source below.

20 of 20 neighbouring works in this corpus. Blue is what this paper cites; orange is what cites it, and a dashed line is one neighbour citing another. Only the largest labels are drawn — every node carries its full title on hover.

this paper works it cites works citing it node size = global citations · hover for the full title

Citation edge evidence

Each row is directed from the citing work to the cited work. Source status remains readable without the interactive graph.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Sh…→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Sequence to Sequence Learning with Neural Networks→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
[No title in the source record — Edinburgh Research Explorer (University of Edinburgh)]→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Bootstrap your own latent: A new approach to self-supervised Learning→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Character-level Convolutional Networks for Text Classification→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Conditional Random Fields as Recurrent Neural Networks→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Deep Learning: Methods and Applications→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
End-to-End Training of Deep Visuomotor Policies→On the difficulty of training Recurrent Neural NetworksLegacy edge — source detail unavailable
On the difficulty of training Recurrent Neural Networks→Long Short-Term MemoryLegacy edge — source detail unavailable
On the difficulty of training Recurrent Neural Networks→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
On the difficulty of training Recurrent Neural Networks→Adaptive Subgradient Methods for Online Learning and Stochastic OptimizationLegacy edge — source detail unavailable
On the difficulty of training Recurrent Neural Networks→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
On the difficulty of training Recurrent Neural Networks→Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communi…Legacy edge — source detail unavailable
On the difficulty of training Recurrent Neural Networks→Reservoir computing approaches to recurrent neural network trainingLegacy edge — source detail unavailable
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks→Long Short-Term MemoryLegacy edge — source detail unavailable
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Sh…→Adaptive Subgradient Methods for Online Learning and Stochastic OptimizationLegacy edge — source detail unavailable
Long Short-Term Memory→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Long Short-Term MemoryLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communi…Legacy edge — source detail unavailable
Deep learning in neural networks: An overview→Sequence to Sequence Learning with Neural NetworksLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Adaptive Subgradient Methods for Online Learning and Stochastic OptimizationLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Deep Learning: Methods and ApplicationsLegacy edge — source detail unavailable
Conditional Random Fields as Recurrent Neural Networks→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
Sequence to Sequence Learning with Neural Networks→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
Sequence to Sequence Learning with Neural Networks→Long Short-Term MemoryLegacy edge — source detail unavailable
Sequence to Sequence Learning with Neural Networks→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
Deep Learning: Methods and Applications→Long Short-Term MemoryLegacy edge — source detail unavailable
Deep Learning: Methods and Applications→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
End-to-End Training of Deep Visuomotor Policies→Deep learning in neural networks: An overviewLegacy edge — source detail unavailable
Character-level Convolutional Networks for Text Classification→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
Character-level Convolutional Networks for Text Classification→Long Short-Term MemoryLegacy edge — source detail unavailable
Reservoir computing approaches to recurrent neural network training→Long Short-Term MemoryLegacy edge — source detail unavailable
Reservoir computing approaches to recurrent neural network training→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
Reservoir computing approaches to recurrent neural network training→Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communi…Legacy edge — source detail unavailable
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network→Long Short-Term MemoryLegacy edge — source detail unavailable
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network→Sequence to Sequence Learning with Neural NetworksLegacy edge — source detail unavailable
[No title in the source record — Edinburgh Research Explorer (University of Edinburgh)]→Sequence to Sequence Learning with Neural NetworksLegacy edge — source detail unavailable
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF→Long Short-Term MemoryLegacy edge — source detail unavailable
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF→An Empirical Exploration of Recurrent Network ArchitecturesLegacy edge — source detail unavailable
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning→Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Sh…Legacy edge — source detail unavailable
Bootstrap your own latent: A new approach to self-supervised Learning→Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Sh…Legacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→Long Short-Term MemoryLegacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→Learning long-term dependencies with gradient descent is difficultLegacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communi…Legacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→Sequence to Sequence Learning with Neural NetworksLegacy edge — source detail unavailable

What this paper cites, inside the corpus

Paper	Year	Cited
Long Short-Term Memory	1997	101,359
Learning representations by back-propagating errors	1986	31,688
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization	2010	8,621
Learning long-term dependencies with gradient descent is difficult	1994	8,610
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communi…	2004	3,862
Reservoir computing approaches to recurrent neural network training	2009	2,992

What cites it, inside the corpus

Paper	Year	Cited
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Sh…	2015	24,404
Deep learning in neural networks: An overview	2014	18,236
Sequence to Sequence Learning with Neural Networks	2014	13,351
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning	2017	12,734
[No title in the source record — Edinburgh Research Explorer (University of Edinburgh)]		7,291
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network	2020	5,639
Bootstrap your own latent: A new approach to self-supervised Learning	2020	3,447
Character-level Convolutional Networks for Text Classification	2015	3,280
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF	2016	2,595
Conditional Random Fields as Recurrent Neural Networks	2015	2,437
Deep Learning: Methods and Applications	2014	1,801
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks	2015	1,646
An Empirical Exploration of Recurrent Network Architectures	2015	1,406
End-to-End Training of Deep Visuomotor Policies	2015	1,400

Links

Download BibTeX · Download RIS · DOI · Open access copy · OpenAlex record

Topics

Neural Networks and Applications	Computer Science
Model Reduction and Neural Networks	Physics and Astronomy
Domain Adaptation and Few-Shot Learning	Computer Science

Is this record sound?

complete

Nothing in this record contradicts itself and no field we check is missing.

supports3 author record(s) attached.
supports23 reference(s) recorded.
neutralThe DOI carries no year to check against.
supportsA title is present.

Provenance

Everything above was read from one stored OpenAlex payload, fetched 2026-09-04T03:58:46+00:00.

sha256 bba2969b3567609a…