Above is my loss function, the highlighted part is the output of a neural network. If that value is too big, the exponential becomes to high and I start getting NaNs.
How can I prevent this, or limit the output of the network as such it doesn't make the loss function fall apart because of the exponential?
Things I've tried:
Gradient clipping. This allows the network to keep learning without collapsing but the learning isn't good.
Using a larger weight decay in SGD. This helped but I don't know why. The model converged but it doesn't generalize well so I would like to use a smaller weight decay.
Any thoughts would be greatly appreciated. Even insight as to why using a larger weight decay helps this problem.