I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the forward pass and back propagation essentially become a chain of multiplications of numbers $< 1$, resulting in increasingly small output values/gradients. This is obviously true for the forward pass, and also seems to make sense for the backpropagation:
As illustrated through the example made in this article, the partial derivative of the cost function w.r.t. the weight $w_{1}$, is given by:
$\frac{\delta J}{\delta w_{1}} = \frac{\delta J}{\delta \hat{y}} \frac{\delta \hat{y}}{\delta z_{2}} \frac{\delta z_{2}}{\delta a_{1}} \frac{\delta a_{1}}{\delta z_{1}} \frac{\delta z_{1}}{\delta w_{1}}$
Assuming that the sigmoid is used as the activation function function (i.e., $\hat{y} = \sigma(z_{2})$ / $a_{1} = \sigma(z_{1})$), whose derivative is given by $\sigma(x)(1-\sigma(x))$, then $\frac{\delta \hat{y}}{\delta z_{2}}$ and $\frac{\delta a_{1}}{\delta z_{1}}$ cannot be $> 1$ and will be close to zero if $z_{1}$ or $z_{2}$ are very big/small. Considering further that $z_{i} = w_{i}a_{i-1} + b_{i}$, it follows that $\frac{\delta z_{2}}{\delta a_{1}} = w_{2}$, and $\frac{\delta z_{1}}{\delta w_{1}} = x_{1}$, where $x_{1}$ is the input.
So, indeed there are no values $> 1$ in this calculation (except maybe the input $x_{1}$), and I can understand how the gradient would vanish and how this could be mitigated by using a different activation function such as the ReLU. But even using the ReLU, whose derivative is simply 1, the gradient would still diminish when the weights are $< 1$, right? Just not as quickly, I'm assuming.
I'm a bit confused about the exploding gradient problem. With very large weights, $\frac{\delta \hat{y}}{\delta z_{2}}$ and $\frac{\delta a_{1}}{\delta z_{1}}$ would still be $\leq 1$ and so only $\frac{\delta z_{2}}{\delta a_{1}}$ and $\frac{\delta z_{1}}{\delta w_{1}}$ could be $> 1$ (disregarding $\frac{\delta J}{\delta \hat{y}}$ for now). I see how in a deeper network, more weights $> 1$ would go into the gradient of early layers, but also the number of terms that would be $\leq 1$ would increase. Wouldn't the weights then have to be disproportionately big for the gradient to explode and can this actually happen? Also, wouldn't using the ReLU instead of the sigmoid increase the risk of running into this problem, rather than helping to avoid it?
Sorry for the lengthy post and thanks a lot for any kind of input!
