8
$\begingroup$

I have a few questions about the paper Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.

The building blocks of residual networks can be viewed as follows: data passed to the right branch, convolution, scaling, convolution; then in the right branch: identity mapping or convolution; and after that, both branches' outputs are summed.

  1. Why does this allow the training of deep networks, escaping network saturation at deep levels? I didn't get the idea from the paper. Is this summation a reminder of what happened a few layers ago, a reference point? Or is it just clever regularization?

  2. How was the amount of right branch layers chosen?

  3. Why do we train scale layer on the right branch, according to this caffe architecture?

$\endgroup$
3
  • $\begingroup$ It is not clear what paper you ae referring to. Can you clarify? $\endgroup$
    – mdewey
    Sep 27 '16 at 12:54
  • $\begingroup$ @mdewey added paper link to question $\endgroup$ Sep 27 '16 at 15:06
  • $\begingroup$ I think this video from Andrew Ng could help. $\endgroup$ Aug 8 at 9:34
4
$\begingroup$

In short (from my cellphone), it works because the gradient gets to every layer, with only a small number of layers in between it needs to differentiate through.

If you pick a layer from the bottom of your stack of layers, it has a connection with the output layer which only goes through a couple of other layers. This means the gradient will be more pure.

It is a way to solve the vanishing gradient problem. And therefore models could be built even deeper.

$\endgroup$
3
  • 3
    $\begingroup$ This is the exact text from the paper, where authors argue that it is NOT vanishing gradient problem: "We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish". Even I am wondering the real reason for success of ResNets ! $\endgroup$
    – MANU
    Mar 14 '19 at 22:34
  • 1
    $\begingroup$ Yeah, but for me those are two separate things. They say BN makes the norm not go to zero, which is correct, but it does not help the Signal-to-Noise ratio of the gradient. Each layer builds up in noise in which the signal is drowning after going many layers deep (which is another part of the vanishing gradient problem). Resnet kind of attacks the SNR-part of the problem. $\endgroup$
    – 317070
    Mar 15 '19 at 10:34
  • 1
    $\begingroup$ Ok, can you provide some reference for me to dive deeper regarding what you are trying to explain. I would like to understand the differences and better relate. $\endgroup$
    – MANU
    Mar 15 '19 at 13:22
1
$\begingroup$

There is a cleaner answer for this question (found it on a discussion forum):

The point of shortcuts is to prevent vanishing gradients (rarely exploding ones). Imagine that during training the predicted output is not accurate, there is some error. For example, there is a Siberian cat in the picture, but the network predicts it as a European Shorthair cat. Not a big difference, the fur is shorter for the latter. Now, this difference must be back propagated through the whole network as a gradient. You can imagine that this difference, this gradient will be even smaller and smaller as we go back layer by layer towards the image itself, due to overall weights smaller than one. This is what we call "vanishing gradient" (just to mention, with weights greater than one, they would be exploding gradients, a quite bad thing). Too small gradients can be inaccurate and eventually they can be zero, so would not influence and train earlier layers at all.

These vanishing gradients can be avoided by these shortcuts. If you make shortcuts even just over one layer, the gradients can take a shorter path back, which will be roughly half of the original length. It can greatly help avoiding vanishing (or exploding) gradients.

$\endgroup$
1
  • 6
    $\begingroup$ This is the exact text from the paper, where authors argue that it is NOT vanishing gradient problem: "We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish". Even I am wondering the real reason for success of ResNets ! $\endgroup$
    – MANU
    Mar 14 '19 at 22:34
0
$\begingroup$

Why does this allow the training of deep networks, escaping network saturation at deep levels?

We can treat a layer as a function, and adding a layer(with more parameters) leads to a new function with a larger hypothesis space.

There are two methods to adding a layer, and for the generic method, we just add a layer and this would result in the spaces depicted on the left side where a larger space does not guarantee to get closer to the truth(optima, either local or global optima) than before adding it.

However, if we add residual connections, it is like 'Taylor expansion' style parametrization as depicted on the right hand side. The more layers you add, the more approximate to the truth your parameters would possibly be.

enter image description here

Thus, only if larger function classes contain the smaller ones are we guaranteed that increasing them strictly increases the expressive power of the network. For deep neural networks, if we can train the newly-added layer into an identity function f(x) = x, the new model will be as effective as the original model. As the new model may get a better solution to fit the training dataset, the added layer might make it easier to reduce training errors.

At the heart of their proposed residual network (ResNet) is the idea that every additional layer should more easily contain the identity function as one of its elements.

References

  1. Deep Learning - ResNet
  2. Dive into deep learning
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.