Why do residual networks work? I have a few questions about the paper Deep Residual Learning for Image Recognition by
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.
The building blocks of residual networks can be viewed as follows: data passed to the right branch, convolution, scaling, convolution; then in the right branch: identity mapping or convolution; and after that, both branches' outputs are summed.

*

*Why does this allow the training of deep networks, escaping network saturation at deep levels? I didn't get the idea from the paper. Is this summation a reminder of what happened a few layers ago, a reference point? Or is it just clever regularization?


*How was the amount of right branch layers chosen?


*Why do we train scale layer on the right branch, according to this caffe architecture?
 A: In short (from my cellphone), it works because the gradient gets to every layer, with only a small number of layers in between it needs to differentiate through.
If you pick a layer from the bottom of your stack of layers, it  has a connection with the output layer which only goes through a couple of other layers. This means the gradient will be more pure.
It is a way to solve the vanishing gradient problem. And therefore models could be built even deeper.
A: 
Why does this allow the training of deep networks, escaping network saturation at deep levels?

We can treat a layer as a function, and adding a layer(with more parameters) leads to a new function with a larger hypothesis space.
There are two methods to adding a layer, and for the generic method, we just add a layer and this would result in the spaces depicted on the left side where a larger space does not guarantee to get closer to the truth(optima, either local or global optima) than before adding it.
However, if we add residual connections, it is like 'Taylor expansion' style parametrization as depicted on the right hand side. The more layers you add, the more approximate to the truth your parameters would possibly be.


Thus, only if larger function classes contain the smaller ones are we guaranteed that increasing them strictly increases the expressive power of the network. For deep neural networks, if we can train the newly-added layer into an identity function f(x) = x, the new model will be as effective as the original model. As the new model may get a better solution to fit the training dataset, the added layer might make it easier to reduce training errors.
At the heart of their proposed residual network (ResNet) is the idea that every additional layer should more easily contain the identity function as one of its elements.

References

*

*Deep Learning - ResNet

*Dive into deep learning
A: There is a cleaner answer for this question (found it on a discussion forum):

The point of shortcuts is to prevent vanishing gradients (rarely
  exploding ones). Imagine that during training the predicted output is
  not accurate, there is some error. For example, there is a Siberian
  cat in the picture, but the network predicts it as a European
  Shorthair cat. Not a big difference, the fur is shorter for the
  latter. Now, this difference must be back propagated through the whole
  network as a gradient. You can imagine that this difference, this
  gradient will be even smaller and smaller as we go back layer by layer
  towards the image itself, due to overall weights smaller than one.
  This is what we call "vanishing gradient" (just to mention, with
  weights greater than one, they would be exploding gradients, a quite
  bad thing). Too small gradients can be inaccurate and eventually they
  can be zero, so would not influence and train earlier layers at all.
These vanishing gradients can be avoided by these shortcuts. If you
  make shortcuts even just over one layer, the gradients can take a shorter path back, which will be
  roughly half of the original length. It can greatly help avoiding
  vanishing (or exploding) gradients.

