A State-of-the-Art Survey on Deep Learning Theory and Architectures

Md Zahangir Alom; Tarek M. Taha; Chris Yakopcic; Stefan Westberg; Paheding Sidike; Mst Shamima Nasrin; Mahmudul Hasan; Brian C. Van Essen; Abdul Ahad S. Awwal; Vijayan K. Asari

doi:https://doi.org/10.3390/electronics8030292

A State-of-the-Art Survey on Deep Learning Theory and Architectures

2019 · Electronics · 1,619 citations · 0 from inside this corpus

Md Zahangir Alom, Tarek M. Taha low, Chris Yakopcic, Stefan Westberg low, Paheding Sidike low, Mst Shamima Nasrin low, Mahmudul Hasan, Brian C. Van Essen, Abdul Ahad S. Awwal low, Vijayan K. Asari

In recent years, deep learning has garnered tremendous success in a variety of application domains. This new field of machine learning has been growing rapidly and has been applied to most traditional application domains, as well as some new areas that present more opportunities. Different methods have been proposed based on different categories of learning, including supervised, semi-supervised, and un-supervised learning. Experimental results show state-of-the-art performance using deep learning when compared to traditional machine learning approaches in the fields of image processing, computer vision, speech recognition, machine translation, art, medical imaging, medical information processing, robotics and control, bioinformatics, natural language processing, cybersecurity, and many others. This survey presents a brief survey on the advances that have occurred in the area of Deep Learning (DL), starting with the Deep Neural Network (DNN). The survey goes on to cover Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), including Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). Additionally, we have discussed recent developments, such as advanced variant DL techniques based on these DL approaches. This work considers most of the papers published after 2012 from when the history of deep learning began. Furthermore, DL approaches that have been explored and evaluated in different application domains are also included in this survey. We also included recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys that have been published on DL using neural networks and a survey on Reinforcement Learning (RL). However, those papers have not discussed individual advanced techniques for training large-scale deep learning models and the recently developed method of generative models.

36 of 36 neighbouring works in this corpus. Blue is what this paper cites; orange is what cites it, and a dashed line is one neighbour citing another. Only the largest labels are drawn — every node carries its full title on hover.

this paper works it cites works citing it node size = global citations · hover for the full title

Citation edge evidence

Each row is directed from the citing work to the cited work. Source status remains readable without the interactive graph.

A State-of-the-Art Survey on Deep Learning Theory and Architectures→Dropout: a simple way to prevent neural networks from overfittingLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→A Survey on Transfer LearningLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Xception: Deep Learning with Depthwise Separable ConvolutionsLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Deep learning in neural networks: An overviewLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Mastering the game of Go with deep neural networks and tree searchLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→A survey on deep learning in medical image analysisLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Understanding the difficulty of training deep feedforward neural networksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Aggregated Residual Transformations for Deep Neural NetworksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→The perceptron: A probabilistic model for information storage and organization in the bra…Legacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Mastering the game of Go without human knowledgeLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Speech recognition with deep recurrent neural networksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Domain-Adversarial Training of Neural NetworksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Learning Deep Architectures for AILegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Recurrent neural network based language modelLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→A unified architecture for natural language processingLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Deep Reinforcement Learning: A Brief SurveyLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Deep Convolutional Neural Networks for Image Classification: A Comprehensive ReviewLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→On the importance of initialization and momentum in deep learningLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Reinforcement learning in robotics: A surveyLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Long short-term memory recurrent neural network architectures for large scale acoustic mo…Legacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Deep learning applications and challenges in big data analyticsLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?Legacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Disco…Legacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Stochastic Gradient Descent TricksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Acoustic Modeling Using Deep Belief NetworksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network ModelsLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→End-to-end training of deep visuomotor policiesLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Asynchronous Methods for Deep Reinforcement LearningLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial NetworksLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→Transfer Learning for Reinforcement Learning Domains: A SurveyLegacy edge — source detail unavailable
A State-of-the-Art Survey on Deep Learning Theory and Architectures→An Empirical Exploration of Recurrent Network ArchitecturesLegacy edge — source detail unavailable
On the importance of initialization and momentum in deep learning→Understanding the difficulty of training deep feedforward neural networksLegacy edge — source detail unavailable
On the importance of initialization and momentum in deep learning→Acoustic Modeling Using Deep Belief NetworksLegacy edge — source detail unavailable
On the importance of initialization and momentum in deep learning→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
On the importance of initialization and momentum in deep learning→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Understanding the difficulty of training deep feedforward neural networks→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
Understanding the difficulty of training deep feedforward neural networks→A unified architecture for natural language processingLegacy edge — source detail unavailable
Understanding the difficulty of training deep feedforward neural networks→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Domain-Adversarial Training of Neural Networks→Dropout: a simple way to prevent neural networks from overfittingLegacy edge — source detail unavailable
Acoustic Modeling Using Deep Belief Networks→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
Acoustic Modeling Using Deep Belief Networks→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→The perceptron: A probabilistic model for information storage and organization in the bra…Legacy edge — source detail unavailable
Deep learning in neural networks: An overview→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→A unified architecture for natural language processingLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Speech recognition with deep recurrent neural networksLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→A Survey on Transfer LearningLegacy edge — source detail unavailable
Deep learning in neural networks: An overview→Long short-term memory recurrent neural network architectures for large scale acoustic mo…Legacy edge — source detail unavailable
Deep learning in neural networks: An overview→Learning Deep Architectures for AILegacy edge — source detail unavailable
Dropout: a simple way to prevent neural networks from overfitting→Acoustic Modeling Using Deep Belief NetworksLegacy edge — source detail unavailable
Dropout: a simple way to prevent neural networks from overfitting→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
Dropout: a simple way to prevent neural networks from overfitting→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Reducing the Dimensionality of Data with Neural Networks→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Deep learning applications and challenges in big data analytics→Acoustic Modeling Using Deep Belief NetworksLegacy edge — source detail unavailable
Deep learning applications and challenges in big data analytics→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
Deep learning applications and challenges in big data analytics→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Speech recognition with deep recurrent neural networks→Acoustic Modeling Using Deep Belief NetworksLegacy edge — source detail unavailable
Long short-term memory recurrent neural network architectures for large scale acoustic mo…→Recurrent neural network based language modelLegacy edge — source detail unavailable
Long short-term memory recurrent neural network architectures for large scale acoustic mo…→Acoustic Modeling Using Deep Belief NetworksLegacy edge — source detail unavailable
Long short-term memory recurrent neural network architectures for large scale acoustic mo…→Speech recognition with deep recurrent neural networksLegacy edge — source detail unavailable
Xception: Deep Learning with Depthwise Separable Convolutions→Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningLegacy edge — source detail unavailable
Aggregated Residual Transformations for Deep Neural Networks→Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningLegacy edge — source detail unavailable
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks→Domain-Adversarial Training of Neural NetworksLegacy edge — source detail unavailable
A survey on deep learning in medical image analysis→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
A survey on deep learning in medical image analysis→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
A survey on deep learning in medical image analysis→What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?Legacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→On the importance of initialization and momentum in deep learningLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Understanding the difficulty of training deep feedforward neural networksLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Domain-Adversarial Training of Neural NetworksLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Deep learning in neural networks: An overviewLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Dropout: a simple way to prevent neural networks from overfittingLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Reducing the Dimensionality of Data with Neural NetworksLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→A Fast Learning Algorithm for Deep Belief NetsLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Xception: Deep Learning with Depthwise Separable ConvolutionsLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Aggregated Residual Transformations for Deep Neural NetworksLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→End-to-end training of deep visuomotor policiesLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningLegacy edge — source detail unavailable
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review→Learning Deep Architectures for AILegacy edge — source detail unavailable
Mastering the game of Go without human knowledge→Reinforcement learning in robotics: A surveyLegacy edge — source detail unavailable
Mastering the game of Go without human knowledge→Mastering the game of Go with deep neural networks and tree searchLegacy edge — source detail unavailable
Mastering the game of Go without human knowledge→Asynchronous Methods for Deep Reinforcement LearningLegacy edge — source detail unavailable
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models→Recurrent neural network based language modelLegacy edge — source detail unavailable
End-to-end training of deep visuomotor policies→Reinforcement learning in robotics: A surveyLegacy edge — source detail unavailable
End-to-end training of deep visuomotor policies→Deep learning in neural networks: An overviewLegacy edge — source detail unavailable
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning→On the importance of initialization and momentum in deep learningLegacy edge — source detail unavailable
Deep Reinforcement Learning: A Brief Survey→Learning representations by back-propagating errorsLegacy edge — source detail unavailable
Deep Reinforcement Learning: A Brief Survey→Mastering the game of Go with deep neural networks and tree searchLegacy edge — source detail unavailable
Deep Reinforcement Learning: A Brief Survey→End-to-end training of deep visuomotor policiesLegacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→On the importance of initialization and momentum in deep learningLegacy edge — source detail unavailable
An Empirical Exploration of Recurrent Network Architectures→Recurrent neural network based language modelLegacy edge — source detail unavailable

What this paper cites, inside the corpus

Paper	Year	Cited
Dropout: a simple way to prevent neural networks from overfitting	2014	34,236
Learning representations by back-propagating errors	1986	31,688
A Survey on Transfer Learning	2009	23,670
Reducing the Dimensionality of Data with Neural Networks	2006	21,247
Xception: Deep Learning with Depthwise Separable Convolutions	2017	19,457
Deep learning in neural networks: An overview	2014	18,236
A Fast Learning Algorithm for Deep Belief Nets	2006	16,540
Mastering the game of Go with deep neural networks and tree search	2016	16,014
A survey on deep learning in medical image analysis	2017	15,110
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning	2017	12,734
Understanding the difficulty of training deep feedforward neural networks	2010	12,673
Aggregated Residual Transformations for Deep Neural Networks	2017	12,015
The perceptron: A probabilistic model for information storage and organization in the bra…	1958	11,965
Mastering the game of Go without human knowledge	2017	9,313
Speech recognition with deep recurrent neural networks	2013	8,916
Domain-Adversarial Training of Neural Networks	2017	7,702
Learning Deep Architectures for AI	2009	6,951
Recurrent neural network based language model	2010	5,432
A unified architecture for natural language processing	2008	5,207
Deep Reinforcement Learning: A Brief Survey	2017	4,434
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review	2017	3,570
On the importance of initialization and momentum in deep learning	2013	3,523
Reinforcement learning in robotics: A survey	2013	3,154
Long short-term memory recurrent neural network architectures for large scale acoustic mo…	2014	2,986
Deep learning applications and challenges in big data analytics	2015	2,610
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?	2017	2,537
Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Disco…	2017	2,413
Stochastic Gradient Descent Tricks	2012	1,944
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation	2017	1,759
Acoustic Modeling Using Deep Belief Networks	2011	1,752
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models	2016	1,725
End-to-end training of deep visuomotor policies	2016	1,706
Asynchronous Methods for Deep Reinforcement Learning	2016	1,689
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks	2017	1,634
Transfer Learning for Reinforcement Learning Domains: A Survey	2009	1,564
An Empirical Exploration of Recurrent Network Architectures	2015	1,406

Links

Download BibTeX · Download RIS · DOI · Open access copy · OpenAlex record

Topics

Anomaly Detection Techniques and Applications	Computer Science
Advanced Neural Network Applications	Computer Science
Generative Adversarial Networks and Image Synthesis	Computer Science

Is this record sound?

complete

Nothing in this record contradicts itself and no field we check is missing.

supports10 author record(s) attached.
supports247 reference(s) recorded.
neutralThe DOI carries no year to check against.
supportsA title is present.

Provenance

Everything above was read from one stored OpenAlex payload, fetched 2026-09-04T03:58:56+00:00.

sha256 7bc26169e93749d1…