0
$\begingroup$

So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with vanishing gradients, or improve the flow of information back deep in time."

From the explanation, the only thing that I can deduce is that only vanishing gradient pose a problem while learning long term dependencies. Is this correct?

does exploding gradient have any role in learning long term dependencies?

$\endgroup$

1 Answer 1

0
$\begingroup$

Exploding gradient is also an issue when vanilla RNN trying to learn the long-term dependency in the context. This phenomenon is related to the largest singular value $\lambda_1$ of weight matrix $W_h$ as well as $\frac{1}{\gamma}$, a term depends on activation function $\sigma$ and dimensionality, see 'On the difficulty of training recurrent neural networks' for further references.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.