0
$\begingroup$

Problem

I am trying to understand the source of vanishing/exploding gradients in vanilla RNN. The update rule of vanilla RNN is

$$ \begin{aligned} &\mathbf{a}^{\left<t\right>}=\tanh(\mathbf{W}_ {aa}\mathbf{a}^{\left<t-1\right>}+\mathbf{W}_ {ax}\mathbf{x}^{\left< t\right>}+\mathbf{b}_ a)\\ &\mathbf{y}^{\left<t\right>}=g(\mathbf{W}_ y\mathbf{a}^{\left<t\right>}+\mathbf{b}_ y) \end{aligned} $$

To my understanding, only $\mathbf{W}_{aa}$ and $\mathbf{W}_y$ will cause the issues since they all require $\nabla_{\mathbf{a}^{\left< t\right>}}f$, where $\mathbf{a}^{\left<t\right>}$ is in turn dependent on previous steps, which result in multiplicative terms $(\mathbf{W}_{aa}^{T})^k$ and $(\mathbf{W}_{y}^{T})^k$ and thereby vanishing/exploding gradients. However, since $\nabla_{\mathbf{x}^{\left< t\right>}}f$ does not depend on previous steps. So $\mathbf{W}_{ax}$ will not cause the problem.

Is my understanding correct? Thank you in advance!

$\endgroup$
1
$\begingroup$

Yes, the gradient through $W_{ax}$ will stop there, and does not get propagated any further backwards in time. There is no gradient $dx_{t}/dx_{t-1}$ so there is no accumulated multiplicative terms. However, the gradient through $W_{aa}$ gets propagated through $da_{t}/da_{t-1}$ and will cause exploding/vanishing gradients through the accumulated multiplication terms.

$\endgroup$
1
$\begingroup$

The vanishing and exploding gradients in any neural network, not just RNN. I think they are caused by the multiplication of the gradients across layers (chain rule). If you have a neural network that represents as (not exact):

$$f(x) = A(B(C(x)))$$

The derivative or gradient calculation would be:

$$ \frac{df}{dx}=\frac{dA}{dB}\frac{dB}{dC}\frac{dC}{dx} $$

If the network has many layers, the multiplications add up, which causes either gradients vanishing or exploding, especially for RNN.

See here: https://medium.com/learn-love-ai/the-curious-case-of-the-vanishing-exploding-gradient-bf58ec6822eb

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.