I am talking about the article of Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research team):

"Deep Residual Learning for Image Recognitione (2015)"

ResNets won the 1-st place at Large Scale Visual Recognition Challenge 2015 (ILSVRC2015)

In terms of architecture, do ResNets represent convolutional neural networks?


Residual networks (ResNets) are a type of recurrent neural network (RNN).

Jürgen Schmidhuber :

Microsoft Research dominated the ImageNet 2015 contest with a deep neural network of 150 layers 1. Congrats to Kaiming He & Xiangyu Zhang & Shaoqing Ren & Jian Sun on the great results [2]!

Their CNN layers compute G(F(x)+x), which is essentially a feedforward Long Short-Term Memory (LSTM) [3] without gates!

Their net is similar to the very deep Highway Networks [4] (with hundreds of layers), which are feedforward LSTMs with forget gates (= gated recurrent units) [5].

The authors mention the vanishing gradient problem, but do not mention my very first student Sepp Hochreiter (now professor) who identified and analyzed this fundamental deep learning problem in 1991, years before anybody else did [6].

Apart from the above, I liked the paper 1 a lot. LSTM concepts keep invading CNN territory [e.g., 7a-e], also through GPU-friendly multi-dimensional LSTMs [8].


1 Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual Learning for Image Recognition. arxiv:1512.03385

[2] ImageNet Large Scale Visual Recognition Challenge 2015 (ILSVRC2015): Results

[3] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997. Based on TR FKI-207-95, TUM (1995). PDF. Led to a lot of follow-up work, and is now heavily used by leading IT companies all over the world.

[4] R. K. Srivastava, K. Greff, J. Schmidhuber. Training Very Deep Networks. NIPS 2015; arxiv:1505.00387.

[5] F. A. Gers, J. Schmidhuber, F. Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10):2451-2471, 2000. PDF.

[6] S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich, 1991. Advisor: J. Schmidhuber. Overview.

[7a] 2011: First superhuman CNNs [7b] 2011: First human-competitive CNNs for handwriting [7c] 2012: First CNN to win segmentation contest [7d] 2012: First CNN to win contest on object discovery in large images [7e] Deep Learning. Scholarpedia, 10(11):32832, 2015

[8] M. Stollenga, W. Byeon, M. Liwicki, J. Schmidhuber. Parallel Multi-Dimensional LSTM, with Application to Fast Biomedical Volumetric Image Segmentation. NIPS 2015; arxiv:1506.07452.

http://www.asimovinstitute.org/neural-network-zoo/ :

Deep residual networks (DRN) are very deep FFNNs with extra connections passing input from one layer to a later layer (often 2 to 5 layers) as well as the next layer. Instead of trying to find a solution for mapping some input to some output across say 5 layers, the network is enforced to learn to map some input to some output + some input. Basically, it adds an identity to the solution, carrying the older input over and serving it freshly to a later layer. It has been shown that these networks are very effective at learning patterns up to 150 layers deep, much more than the regular 2 to 5 layers one could expect to train. However, it has been proven that these networks are in essence just RNNs without the explicit time based construction and they’re often compared to LSTMs without gates.

enter image description here

https://arxiv.org/pdf/1604.03640.pdf :

We begin with the observation that a shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers


Apparently residual networks share something in common with LSTMs and RNNs, however by definition residual networks are more of a type of convolutional neural network (CNN).

The key difference between a residual network and an unrolled RNN/LSTM is, as mentioned in https://arxiv.org/pdf/1604.03640.pdf

unrolling in (discrete) time the feedback system gives a deep residual network with the same (that is, shared) weights among the layers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.