Residual networks (ResNets) are a type of recurrent neural network (RNN).
Jürgen Schmidhuber :
Microsoft Research dominated the ImageNet 2015 contest with a deep
neural network of 150 layers 1. Congrats to Kaiming He & Xiangyu
Zhang & Shaoqing Ren & Jian Sun on the great results !
Their CNN layers compute G(F(x)+x), which is essentially a feedforward
Long Short-Term Memory (LSTM)  without gates!
Their net is similar to the very deep Highway Networks  (with
hundreds of layers), which are feedforward LSTMs with forget gates (=
gated recurrent units) .
The authors mention the vanishing gradient problem, but do not mention
my very first student Sepp Hochreiter (now professor) who identified
and analyzed this fundamental deep learning problem in 1991, years
before anybody else did .
Apart from the above, I liked the paper 1 a lot. LSTM concepts keep
invading CNN territory [e.g., 7a-e], also through GPU-friendly
multi-dimensional LSTMs .
1 Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual
Learning for Image Recognition. arxiv:1512.03385
 ImageNet Large Scale Visual Recognition Challenge 2015
 S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural
Computation, 9(8):1735-1780, 1997. Based on TR FKI-207-95, TUM (1995).
PDF. Led to a lot of follow-up work, and is now heavily used by
leading IT companies all over the world.
 R. K. Srivastava, K. Greff, J. Schmidhuber. Training Very Deep
Networks. NIPS 2015; arxiv:1505.00387.
 F. A. Gers, J. Schmidhuber, F. Cummins. Learning to Forget:
Continual Prediction with LSTM. Neural Computation, 12(10):2451-2471,
 S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen.
Diploma thesis, TU Munich, 1991. Advisor: J. Schmidhuber. Overview.
[7a] 2011: First superhuman CNNs [7b] 2011: First human-competitive
CNNs for handwriting [7c] 2012: First CNN to win segmentation contest
[7d] 2012: First CNN to win contest on object discovery in large
images [7e] Deep Learning. Scholarpedia, 10(11):32832, 2015
 M. Stollenga, W. Byeon, M. Liwicki, J. Schmidhuber. Parallel
Multi-Dimensional LSTM, with Application to Fast Biomedical Volumetric
Image Segmentation. NIPS 2015; arxiv:1506.07452.
Deep residual networks (DRN) are very deep FFNNs with extra
connections passing input from one layer to a later layer (often 2 to
5 layers) as well as the next layer. Instead of trying to find a
solution for mapping some input to some output across say 5 layers,
the network is enforced to learn to map some input to some output +
some input. Basically, it adds an identity to the solution, carrying
the older input over and serving it freshly to a later layer. It has
been shown that these networks are very effective at learning patterns
up to 150 layers deep, much more than the regular 2 to 5 layers one
could expect to train. However, it has been proven that these networks
are in essence just RNNs without the explicit time based construction
and they’re often compared to LSTMs without gates.
We begin with the observation that a shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers