I am following deeplearning.ai's videos on Coursera. I have a couple of questions regarding vanishing and exploding gradients. The following is Prof Andrew Ngs lecture slides:
From what Prof Ng says in the lectures, having large weights leads to large gradients and having having small weights leads to small gradients and both of these are troublesome for training. My questions are:
1.) Assuming my activation function is Sigmoid (say g(x)), having very large weights (x) will lead to small gradients and slow learning through Gradient Descent. If, x is close to 0, then my gradients will be very large. Can this lead to overstepping in Gradient Descent (akin to a large Learning Rate)? I am guessing this is not the case as the gradient is due to the nature of the cost function while the learning rate is a parameter we set.
2.) If my activation function is ReLU, the slope will be constant for any x>0 and 0 for x<0. So, where is the notion of exploding or vanishing gradients coming from for ReLU? I thought ReLUs solve the vanishing and exploding gradient problem.
Thanks in advance.