Cost function turning into nan after a certain number of iterations

Question

I have a question and would like to hear what the community has to say. Suppose you are training a deep learning neural network. The implementation details are not relevant for my question. I know very well that if you choose a learning rate that is too big, you end up with a cost function that may becomes nan (if, for example, you use the sigmoid activation function). Suppose I am using the cross entropy as cost function. Typical binary classification (or even multi class with softmax) problem. I also know about why this happen. I often observe the following behaviour: my cost function decreases nicely, but after a certain number of epochs it becomes nan. Reducing the learning rate make this happen later (so after more epochs). Is this really because the (for example) gradient descent after getting very close to the minimum cannot stabilize itself and starts bouncing around wildly? I thought that the algorithm will not converge exactly to the minimum but should oscillates around it, remaining more or less stable there... Thoughts?

Answer 1

Well, if you get NaN values in your cost function, it means that the input is outside of the function domain. E.g. the logarithm of 0. Or it could be in the domain analytically, but due to numerical errors we get the same problem (e.g. a small value gets rounded to 0). It has nothing to do with an inability to "settle".

So, you have to determine what the non-allowed function input values for your given cost function are. Then, you have to determine why you are getting that input to your cost function. You may have to change the scaling of the input data and the weight initialization. Or you just have to have an adaptive learning rate as suggested by Avis, as the cost function landscape may be quiet chaotic. Or it could be because of something else, like numerical issues with some layer in your architecture.

It is very difficult to say with deep networks, but I suggest you start looking at the progression of the input values to your cost function (the output of your activation layer), and try to determine a cause.

Answer 2

Here are some of the things you could do:

• When using SoftMax cross entropy function:

  the SoftMax numerator should never have zero-values due to the exponential. However, due to floating point precision, the numerator could be a very small value, say, exp(-50000), which essentially evaluates to zero.(ref.)

  • Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.(ref.)

• You can use methods like "FastNorm" which improves numerical stability and reduces accuracy variance enabling higher learning rate and offering better convergence.(ref.)

• Check weights initialization: If unsure, use Xavier or He initialization. Also, your initialization might be leading you to a bad local minimum, so try a different initialization and see if it helps.

• Decrease the learning rate, especially if you are getting NaNs in the first 100 iterations.

• NaNs can arise from division by zero or natural log of zero or negative number.

• Try evaluating your network layer by layer and see where the NaNs appear.

some of the suggestions were taken from the references from the two great posts on StackOverflow & on KDnuggests

Answer 3

Possible reasons:

1. Gradient blow up
2. Your input contains nan (or unexpected values)
3. Loss function not implemented properly
4. Numerical instability in the Deep learning framework

You can check whether it always becomes nan when fed with a particular input or is it completely random.

Usual practice is to reduce the learning rate in step manner after every few iterations.