I have a question and would like to hear what the community has to say. Suppose you are training a deep learning neural network. The implementation details are not relevant for my question. I know very well that if you choose a learning rate that is too big, you end up with a cost function that may becomes nan (if, for example, you use the sigmoid activation function). Suppose I am using the cross entropy as cost function. Typical binary classification (or even multi class with softmax) problem. I also know about why this happen. I often observe the following behaviour: my cost function decreases nicely, but after a certain number of epochs it becomes nan. Reducing the learning rate make this happen later (so after more epochs). Is this really because the (for example) gradient descent after getting very close to the minimum cannot stabilize itself and starts bouncing around wildly? I thought that the algorithm will not converge exactly to the minimum but should oscillates around it, remaining more or less stable there... Thoughts?

  • $\begingroup$ I have also encountered the same problem in which cost function turns out to a 'nan' value,the reason behind the 'nan' values may be invalid values inside the logarithmic function like zero or negative values. Try reducing learning rate for your function. $\endgroup$ – Dhruv Aditya Mittal Jun 11 '18 at 4:30
  • $\begingroup$ I had same problem. After going through many of suggestions available on web and implementing it (I don't remember exactly which one) I came to know that it happened because of convolution layer got input which is outside the intended input. Introducing batch normalization at the input layer solved the problem for me. $\endgroup$ – Mahender Nakrani Sep 22 '19 at 6:45
  • $\begingroup$ I just solve the same problem. In my case, it is the issue of a manual normalization during the forward computing. When computing $y = x/ {\parallel x \parallel}_2$, pay more attention to the divide and the sqrt ops. When the operand is close to 0, the bp gradients could be Nan. You can add a small epsilon to make these ops more stable. $\endgroup$ – Flying spaghetti Jun 30 '20 at 6:34
  • $\begingroup$ One quick reminder for anyone like me who forgot to check if the last layer is a softmax after modifying somebody's code. $\endgroup$ – tjysdsg Mar 15 at 14:28

Well, if you get NaN values in your cost function, it means that the input is outside of the function domain. E.g. the logarithm of 0. Or it could be in the domain analytically, but due to numerical errors we get the same problem (e.g. a small value gets rounded to 0). It has nothing to do with an inability to "settle".

So, you have to determine what the non-allowed function input values for your given cost function are. Then, you have to determine why you are getting that input to your cost function. You may have to change the scaling of the input data and the weight initialization. Or you just have to have an adaptive learning rate as suggested by Avis, as the cost function landscape may be quiet chaotic. Or it could be because of something else, like numerical issues with some layer in your architecture.

It is very difficult to say with deep networks, but I suggest you start looking at the progression of the input values to your cost function (the output of your activation layer), and try to determine a cause.

  • $\begingroup$ Thansk. I did all you suggested and I know the cause of it... Exactly what you describe... Just wanted to check with others if I missed something... $\endgroup$ – Umberto Feb 1 '18 at 9:51

Here are some of the things you could do:

  • When using SoftMax cross entropy function:

    the SoftMax numerator should never have zero-values due to the exponential. However, due to floating point precision, the numerator could be a very small value, say, exp(-50000), which essentially evaluates to zero.(ref.)

    • Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.(ref.)
  • You can use methods like "FastNorm" which improves numerical stability and reduces accuracy variance enabling higher learning rate and offering better convergence.(ref.)

  • Check weights initialization: If unsure, use Xavier or He initialization. Also, your initialization might be leading you to a bad local minimum, so try a different initialization and see if it helps.

  • Decrease the learning rate, especially if you are getting NaNs in the first 100 iterations.

  • NaNs can arise from division by zero or natural log of zero or negative number.

  • Try evaluating your network layer by layer and see where the NaNs appear.

some of the suggestions were taken from the references from the two great posts on StackOverflow & on KDnuggests

  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ – jbowman Mar 21 '19 at 18:38
  • $\begingroup$ @jbowman, thanks for highlighting it. I edited my post and added all the relevant answers to the question with respective URL references. $\endgroup$ – Anu Mar 21 '19 at 18:53
  • $\begingroup$ Some more checks are here: stackoverflow.com/questions/33962226/… $\endgroup$ – HARSH NILESH PATHAK Jul 15 '20 at 20:14

Possible reasons:

  1. Gradient blow up
  2. Your input contains nan (or unexpected values)
  3. Loss function not implemented properly
  4. Numerical instability in the Deep learning framework

You can check whether it always becomes nan when fed with a particular input or is it completely random.

Usual practice is to reduce the learning rate in step manner after every few iterations.

  • $\begingroup$ Thanks for the suggestions. Reducin the learning rate is what I am doing ;-) Just wanted to ask people here if there may be other causes other than what you mentioned. $\endgroup$ – Umberto Feb 1 '18 at 9:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.