I read some papers which said that the ResNet or Highway networks can mitigate the gradient vanishing/exploding problem in very deep neural networks. I'm not sure how the skip connections can solve the gradient exploding problem. Could anybody give some explanations or references? Thanks.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Thanks for the question. How does ResNet avoids exploding gradients please as we copy information deeper, so this seems like going to explode. What do you think please? $\endgroup$– AvvDec 31, 2021 at 4:56
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
To my understanding, during backprop, skip connection's path will pass gradient update as well. Conceptually this update acts similar to synthetic gradient's purpose.
Instead of waiting for gradient to propagate back one layer at a time, skip connection's path allow gradient to reach those beginning nodes with greater magnitude by skipping some layers in between.
I personally do not find any improvement nor greater risk of encountering exploding gradient with skip connection.
-
$\begingroup$ Hi, thanks a lot for your kind reply! Yeah, the skip connections propagate the gradient flow. I thought it is easy to understand that they are helpful to overcome the gradient vanishing. But I'm not sure what they are helpful to the gradient exploding. As far as I know, the gradient exploding problem is usually solved by gradient clipping. $\endgroup$– miningMay 7, 2018 at 12:07
-
$\begingroup$ I have not encounter enough exploding to be sure, but if skip connection does help, the intuition probably is similar to this scenario: Without skip: gradient is divided with the number of nodes of a layer (Lets say "x"). Gradient for each node would be ~ gradient / x With Skip: Gradient for each node would be ~ gradient / (x + number of skipped connections) If your gradient is large, this would help to distribute the gradient towards the beginning nodes. $\endgroup$ May 8, 2018 at 11:51
-
$\begingroup$ Yeah, I also have not meet the exploding. But is the gradient back-propagated as you said? If
y=x1+x2, thendx1=dyanddx2=dy, so ifdyis large, thendx1anddx2are also large. Ifx2is the skip connection, then the largedx2would be back-propagated to the previous network, which I think would speed up the exploding. $\endgroup$– miningMay 8, 2018 at 14:04 -
$\begingroup$ I see what you are saying, but the cascading effect generally speaking lead to smaller and smaller value. Since most of the weights are set close to zero, its became something like 0.001(1st layer), 0.9(2nd), 0.002(3rd). If you update 0.9 with something large, it will go over 1, which will probably explode. If you share with 0.001 instead, the amount it increased, will have to multiply with the 2nd layer (0.9...) which in total is less. $\endgroup$ May 9, 2018 at 8:36
-
$\begingroup$ Yeah, I see... I didn't consider about the weights. That's the point. Greatly appreciate for your explanation! $\endgroup$– miningMay 9, 2018 at 17:33
$\begingroup$
$\endgroup$
I'm not 100% sure, but I would guess that this is more referring to normalization like BatchNorm rather than skip connections. It's not like ResNets will not explode without any normalization and not like plain VGG-style network will explode if you properly place BatchNorms. Skip connections, I guess, only help make the function smoother and the logic of the function that neural networks compute less convoluted, but it's pretty unrelated to exploding gradients problem.
I found that having an activation function after, for example, BatchNorm, may also be crucial to prevent exploding gradients. Sometimes, when I didn't have it follow BatchNorm, or when I had it precede BatchNorm loss was blowing up.
$\begingroup$
$\endgroup$
A possible reasoning is that residual connections reduce the feature space that a network searches for, as mentioned here:
A neural network without residual parts explores more of the feature space. This makes it more vulnerable to perturbations that cause it to leave the manifold, and necessitates extra training data to recover.
Due to lower sensitivity to perturbations, the network can tend to have smaller loss values, leading to smaller gradients and hence prevent gradient explosion.
