Why do residual networks work?

Question

I have a few questions about the paper Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.

The building blocks of residual networks can be viewed as follows: data passed to the right branch, convolution, scaling, convolution; then in the right branch: identity mapping or convolution; and after that, both branches' outputs are summed.

1. Why does this allow the training of deep networks, escaping network saturation at deep levels? I didn't get the idea from the paper. Is this summation a reminder of what happened a few layers ago, a reference point? Or is it just clever regularization?

2. How was the amount of right branch layers chosen?

3. Why do we train scale layer on the right branch, according to this caffe architecture?

Answer 1

In short (from my cellphone), it works because the gradient gets to every layer, with only a small number of layers in between it needs to differentiate through.

If you pick a layer from the bottom of your stack of layers, it has a connection with the output layer which only goes through a couple of other layers. This means the gradient will be more pure.

It is a way to solve the vanishing gradient problem. And therefore models could be built even deeper.

Answer 2

Why does this allow the training of deep networks, escaping network saturation at deep levels?

We can treat a layer as a function, and adding a layer(with more parameters) leads to a new function with a larger hypothesis space.

There are two methods to adding a layer, and for the generic method, we just add a layer and this would result in the spaces depicted on the left side where a larger space does not guarantee to get closer to the truth(optima, either local or global optima) than before adding it.

However, if we add residual connections, it is like 'Taylor expansion' style parametrization as depicted on the right hand side. The more layers you add, the more approximate to the truth your parameters would possibly be.

Thus, only if larger function classes contain the smaller ones are we guaranteed that increasing them strictly increases the expressive power of the network. For deep neural networks, if we can train the newly-added layer into an identity function f(x) = x, the new model will be as effective as the original model. As the new model may get a better solution to fit the training dataset, the added layer might make it easier to reduce training errors.

At the heart of their proposed residual network (ResNet) is the idea that every additional layer should more easily contain the identity function as one of its elements.

References

1. Deep Learning - ResNet
2. Dive into deep learning

Answer 3

There is a cleaner answer for this question (found it on a discussion forum):

The point of shortcuts is to prevent vanishing gradients (rarely exploding ones). Imagine that during training the predicted output is not accurate, there is some error. For example, there is a Siberian cat in the picture, but the network predicts it as a European Shorthair cat. Not a big difference, the fur is shorter for the latter. Now, this difference must be back propagated through the whole network as a gradient. You can imagine that this difference, this gradient will be even smaller and smaller as we go back layer by layer towards the image itself, due to overall weights smaller than one. This is what we call "vanishing gradient" (just to mention, with weights greater than one, they would be exploding gradients, a quite bad thing). Too small gradients can be inaccurate and eventually they can be zero, so would not influence and train earlier layers at all.

These vanishing gradients can be avoided by these shortcuts. If you make shortcuts even just over one layer, the gradients can take a shorter path back, which will be roughly half of the original length. It can greatly help avoiding vanishing (or exploding) gradients.