I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: enter image description here

My confusion is: this seems to only explain how the gradients won't vanish. On one hand $\partial F / \partial x$ is small as assumed there, the gradient would not explode anyway without the identity matrix. On the other hand, if the partial derivative is not small, the identity matrix is not going to save potential explosions.

So can someone clarify this explanation? Thanks in advance.

  • $\begingroup$ Welcome to CV! On this site there's no need to say "thank you" at the end of your post - it might seem rude at first, but it's part of the philosophy of this site (tour) to "Ask questions, get answers, no distractions" and it means future readers of your question don't need to read through the pleasantries. $\endgroup$ – Jan Kukacka May 14 '18 at 15:11

You say if $\partial F / \partial x$ was small then the gradient wouldn't explode. However, it would vanish towards 0 in the nonresidual case, which is also problematic. On the other hand, it becomes close to the identity matrix in the residual case, which is good.

However, the claim that residual blocks are useful because they stop vanishing/exploding gradients is wrong. In the original ResNet paper, the authors say that nonresidual networks with batch norm layers exhibited healthy gradients which neither exploded nor vanished. Residual blocks were designed to solve some other optimization difficulty.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.