1
$\begingroup$

I'm trying to train a RNN but I'm running into NaN errors while training. I think that this is due to the gradients exploding, although I can't confirm this. As a simple test, I fed random data with the same dimensions as my prepared data to the network, and I was surprised to see the NaNs go away.

Is it true that more random data are less likely to cause exploding gradients? I thought that more learnable data would be less likely to diverge, is this not the case?

Improve this question
$\endgroup$
1
  • $\begingroup$ Why don't you just alter your algorithm so it prints out the gradient it uses in every step? Most implementation has some control value you can change. $\endgroup$
    – usεr11852
    Commented Nov 8, 2015 at 6:17

1 Answer 1

Reset to default
1
$\begingroup$

I don't now what implementation are using, but you can have NAN's if you have missing data, or data with 0 variance (e.g all ones). Also you will usually get some warnings, when there are NANs produced, so it would be a good idea to read them.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.