How can I limit a term in my loss function?

Question

Above is my loss function, the highlighted part is the output of a neural network. If that value is too big, the exponential becomes to high and I start getting NaNs.

How can I prevent this, or limit the output of the network as such it doesn't make the loss function fall apart because of the exponential?

Things I've tried:

Gradient clipping. This allows the network to keep learning without collapsing but the learning isn't good.

Using a larger weight decay in SGD. This helped but I don't know why. The model converged but it doesn't generalize well so I would like to use a smaller weight decay.

Any thoughts would be greatly appreciated. Even insight as to why using a larger weight decay helps this problem.

Answer 1

A common trick here is to factor out $e^{M}$ and using the product-sum rule of logarithms to simplify, where $M:=\max_jh_{\theta}(x_j)$, so rewriting $h'=h-M$ for each term prior to exponentiating.

Example

$$\log(e^{100000}+e^{-5}+e^{99995}) =100000+\log(1+e^{-99995}+e^{-5}).$$