# Source of vanishing/exploding gradients in RNN

## Problem

I am trying to understand the source of vanishing/exploding gradients in vanilla RNN. The update rule of vanilla RNN is

\begin{aligned} &\mathbf{a}^{\left}=\tanh(\mathbf{W}_ {aa}\mathbf{a}^{\left}+\mathbf{W}_ {ax}\mathbf{x}^{\left< t\right>}+\mathbf{b}_ a)\\ &\mathbf{y}^{\left}=g(\mathbf{W}_ y\mathbf{a}^{\left}+\mathbf{b}_ y) \end{aligned}

To my understanding, only $$\mathbf{W}_{aa}$$ and $$\mathbf{W}_y$$ will cause the issues since they all require $$\nabla_{\mathbf{a}^{\left< t\right>}}f$$, where $$\mathbf{a}^{\left}$$ is in turn dependent on previous steps, which result in multiplicative terms $$(\mathbf{W}_{aa}^{T})^k$$ and $$(\mathbf{W}_{y}^{T})^k$$ and thereby vanishing/exploding gradients. However, since $$\nabla_{\mathbf{x}^{\left< t\right>}}f$$ does not depend on previous steps. So $$\mathbf{W}_{ax}$$ will not cause the problem.

Is my understanding correct? Thank you in advance!

Yes, the gradient through $$W_{ax}$$ will stop there, and does not get propagated any further backwards in time. There is no gradient $$dx_{t}/dx_{t-1}$$ so there is no accumulated multiplicative terms. However, the gradient through $$W_{aa}$$ gets propagated through $$da_{t}/da_{t-1}$$ and will cause exploding/vanishing gradients through the accumulated multiplication terms.
$$f(x) = A(B(C(x)))$$
$$\frac{df}{dx}=\frac{dA}{dB}\frac{dB}{dC}\frac{dC}{dx}$$