0
$\begingroup$

I am following deeplearning.ai's videos on Coursera. I have a couple of questions regarding vanishing and exploding gradients. The following is Prof Andrew Ngs lecture slides:

Slide 1

From what Prof Ng says in the lectures, having large weights leads to large gradients and having having small weights leads to small gradients and both of these are troublesome for training. My questions are:

1.) Assuming my activation function is Sigmoid (say g(x)), having very large weights (x) will lead to small gradients and slow learning through Gradient Descent. If, x is close to 0, then my gradients will be very large. Can this lead to overstepping in Gradient Descent (akin to a large Learning Rate)? I am guessing this is not the case as the gradient is due to the nature of the cost function while the learning rate is a parameter we set.

2.) If my activation function is ReLU, the slope will be constant for any x>0 and 0 for x<0. So, where is the notion of exploding or vanishing gradients coming from for ReLU? I thought ReLUs solve the vanishing and exploding gradient problem.

Thanks in advance.

$\endgroup$
1
$\begingroup$

1.) Assuming my activation function is Sigmoid (say g(x)), having very large weights (x) will lead to small gradients and slow learning through Gradient Descent. If, x is close to 0, then my gradients will be very large. Can this lead to overstepping in Gradient Descent (akin to a large Learning Rate)? I am guessing this is not the case as the gradient is due to the nature of the cost function while the learning rate is a parameter we set.

"If, x is close to 0, then my gradients will be very large" --> "very" is very relative here. It will be 0.25 if x=0.

2) If my activation function is ReLU, the slope will be constant for any x>0 and 0 for x<0. So, where is the notion of exploding or vanishing gradients coming from for ReLU? I thought ReLUs solve the vanishing and exploding gradient problem.

You can still lose the signal with ReLUs because the gradient will be 0 for negative inputs.

EDIT:

Alright. I understand how the vanishing gradients occur. But how is it ever possible to get "exploding" gradients if my gradients are never >1?

Consider the following neural network sketch:

enter image description here

For example, the partial derivative of the loss of the weight $w_{1,1}^{1}$, doesn't just rely on the derivative of the ReLU function. I.e., the partial derivative would be

$$\begin{aligned} \frac{\partial l}{\partial w_{1,1}^{(1)}}=\frac{\partial l}{\partial o} \cdot \frac{\partial o}{\partial a_{1}^{(2)}} & \cdot \frac{\partial a_{1}^{(2)}}{\partial a_{1}^{(1)}} \cdot \frac{\partial a_{1}^{(1)}}{\partial w_{1,1}^{(1)}} \\ &+\frac{\partial l}{\partial o} \cdot \frac{\partial o}{\partial a_{2}^{(2)}} \cdot \frac{\partial a_{2}^{(2)}}{\partial a_{1}^{(1)}} \cdot \frac{\partial a_{1}^{(1)}}{\partial w_{1,1}^{(1)}} \end{aligned}$$

$\endgroup$
  • $\begingroup$ Alright. I understand how the vanishing gradients occur. But how is it ever possible to get "exploding" gradients if my gradients are never >1? $\endgroup$ – Nitin Oct 20 '19 at 15:52
  • $\begingroup$ Sure, I edited the answer above to explain $\endgroup$ – resnet Oct 20 '19 at 16:23
  • $\begingroup$ I see. So vanishing gradients are an issue as they slow down learning, but what is the issue with exploding gradients? Also, does using ReLU activations solve the vanishing and exploding gradients problems? $\endgroup$ – Nitin Oct 20 '19 at 16:40
  • 1
    $\begingroup$ the vanishing gradient problem occurs if you have a long chain of multiplications that includes values smaller than 1. Vice versa, if you have values greater than 1 in the chain of multiplications, you will have exploding gradients. ReLU helps somewhat (except dead ReLUs) with the vanishing gradient, but as you can see from the example I included, the chain of multiplications is not just about the relu derivative, so yes, you can still have exploding gradients if several of the values there are greater than 1 and you have a very deep network $\endgroup$ – resnet Oct 20 '19 at 17:07
  • 1
    $\begingroup$ With exploding gradients, you will basically overshoot. I.e., say there is a local loss minimum you want to converge too, instead of taking a small step into that direction, you take a huge leap that is so large that you end up in a worse position than you started from. $\endgroup$ – resnet Oct 21 '19 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.