Loss function

Above is my loss function, the highlighted part is the output of a neural network. If that value is too big, the exponential becomes to high and I start getting NaNs.

How can I prevent this, or limit the output of the network as such it doesn't make the loss function fall apart because of the exponential?

Things I've tried:

Gradient clipping. This allows the network to keep learning without collapsing but the learning isn't good.

Using a larger weight decay in SGD. This helped but I don't know why. The model converged but it doesn't generalize well so I would like to use a smaller weight decay.

Any thoughts would be greatly appreciated. Even insight as to why using a larger weight decay helps this problem.


1 Answer 1


A common trick here is to factor out $e^{M}$ and using the product-sum rule of logarithms to simplify, where $M:=\max_jh_{\theta}(x_j)$, so rewriting $h'=h-M$ for each term prior to exponentiating.


$$\log(e^{100000}+e^{-5}+e^{99995}) =100000+\log(1+e^{-99995}+e^{-5}).$$

  • $\begingroup$ Thank you for a feedback but I'm sorry I might need a bit more of an explanation. I tried mulling over this a few days but I still don't understand. $\endgroup$ May 16, 2018 at 17:47
  • $\begingroup$ @HassanMuhammad: See example $\endgroup$
    – Alex R.
    May 16, 2018 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.