4
$\begingroup$

I read some papers which said that the ResNet or Highway networks can mitigate the gradient vanishing/exploding problem in very deep neural networks. I'm not sure how the skip connections can solve the gradient exploding problem. Could anybody give some explanations or references? Thanks.

$\endgroup$
4
$\begingroup$

To my understanding, during backprop, skip connection's path will pass gradient update as well. Conceptually this update acts similar to synthetic gradient's purpose.

Instead of waiting for gradient to propagate back one layer at a time, skip connection's path allow gradient to reach those beginning nodes with greater magnitude by skipping some layers in between.

I personally do not find any improvement nor greater risk of encountering exploding gradient with skip connection.

$\endgroup$
  • $\begingroup$ Hi, thanks a lot for your kind reply! Yeah, the skip connections propagate the gradient flow. I thought it is easy to understand that they are helpful to overcome the gradient vanishing. But I'm not sure what they are helpful to the gradient exploding. As far as I know, the gradient exploding problem is usually solved by gradient clipping. $\endgroup$ – mining May 7 '18 at 12:07
  • $\begingroup$ I have not encounter enough exploding to be sure, but if skip connection does help, the intuition probably is similar to this scenario: Without skip: gradient is divided with the number of nodes of a layer (Lets say "x"). Gradient for each node would be ~ gradient / x With Skip: Gradient for each node would be ~ gradient / (x + number of skipped connections) If your gradient is large, this would help to distribute the gradient towards the beginning nodes. $\endgroup$ – kiryu nil May 8 '18 at 11:51
  • $\begingroup$ Yeah, I also have not meet the exploding. But is the gradient back-propagated as you said? If y=x1+x2, then dx1=dy and dx2=dy, so if dy is large, then dx1 and dx2 are also large. If x2 is the skip connection, then the large dx2 would be back-propagated to the previous network, which I think would speed up the exploding. $\endgroup$ – mining May 8 '18 at 14:04
  • $\begingroup$ I see what you are saying, but the cascading effect generally speaking lead to smaller and smaller value. Since most of the weights are set close to zero, its became something like 0.001(1st layer), 0.9(2nd), 0.002(3rd). If you update 0.9 with something large, it will go over 1, which will probably explode. If you share with 0.001 instead, the amount it increased, will have to multiply with the 2nd layer (0.9...) which in total is less. $\endgroup$ – kiryu nil May 9 '18 at 8:36
  • $\begingroup$ Yeah, I see... I didn't consider about the weights. That's the point. Greatly appreciate for your explanation! $\endgroup$ – mining May 9 '18 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.