3
$\begingroup$

I have few questions, that apperad reading through paper:

Building block of residual network can be viewed as following: data passed to right branch -> convolution, scaling, convolution and in right branch -> identity mapping or convolution, and after that both branches data are summed.

  1. So why it allows to train deep network, escaping network saturation at deep levels? I didn't get the idea from paper. This summation like reminder to network about what has happened few layers ago, reference point? Or just clever regularization?

  2. How amount of right branch layers was picked?

  3. Why we train scale layer on the right branch? According to caffe architecture https://github.com/KaimingHe/deep-residual-networks/blob/master/prototxt/ResNet-50-deploy.prototxt

UPD: This paper http://arxiv.org/abs/1512.03385

$\endgroup$
  • $\begingroup$ It is not clear what paper you ae referring to. Can you clarify? $\endgroup$ – mdewey Sep 27 '16 at 12:54
  • $\begingroup$ @mdewey added paper link to question $\endgroup$ – Il'ya Zhenin Sep 27 '16 at 15:06
5
$\begingroup$

In short (from my cellphone), it works because the gradient gets to every layer, with only a small number of layers in between it needs to differentiate through.

If you pick a layer from the bottom of your stack of layers, it has a connection with the output layer which only goes through a couple of other layers. This means the gradient will be more pure.

It is a way to solve the vanishing gradient problem. And therefore models could be built even deeper.

$\endgroup$
  • $\begingroup$ This is the exact text from the paper, where authors argue that it is NOT vanishing gradient problem: "We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish". Even I am wondering the real reason for success of ResNets ! $\endgroup$ – MANU Mar 14 at 22:34
  • $\begingroup$ Yeah, but for me those are two separate things. They say BN makes the norm not go to zero, which is correct, but it does not help the Signal-to-Noise ratio of the gradient. Each layer builds up in noise in which the signal is drowning after going many layers deep (which is another part of the vanishing gradient problem). Resnet kind of attacks the SNR-part of the problem. $\endgroup$ – 317070 Mar 15 at 10:34
  • $\begingroup$ Ok, can you provide some reference for me to dive deeper regarding what you are trying to explain. I would like to understand the differences and better relate. $\endgroup$ – MANU Mar 15 at 13:22
1
$\begingroup$

There is a cleaner answer for this question (found it on a discussion forum):

The point of shortcuts is to prevent vanishing gradients (rarely exploding ones). Imagine that during training the predicted output is not accurate, there is some error. For example, there is a Siberian cat in the picture, but the network predicts it as a European Shorthair cat. Not a big difference, the fur is shorter for the latter. Now, this difference must be back propagated through the whole network as a gradient. You can imagine that this difference, this gradient will be even smaller and smaller as we go back layer by layer towards the image itself, due to overall weights smaller than one. This is what we call "vanishing gradient" (just to mention, with weights greater than one, they would be exploding gradients, a quite bad thing). Too small gradients can be inaccurate and eventually they can be zero, so would not influence and train earlier layers at all.

These vanishing gradients can be avoided by these shortcuts. If you make shortcuts even just over one layer, the gradients can take a shorter path back, which will be roughly half of the original length. It can greatly help avoiding vanishing (or exploding) gradients.

$\endgroup$
  • 1
    $\begingroup$ This is the exact text from the paper, where authors argue that it is NOT vanishing gradient problem: "We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish". Even I am wondering the real reason for success of ResNets ! $\endgroup$ – MANU Mar 14 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.