My question is about how is log softmax implemented in practice with the cross-entropy loss.
Softmax gives values between 0 and 1, which means log softmax will give values between -infinity and 0. This means that we cannot use one-hot encoding (one 1 and rest 0's) for our target labels anymore (correct me if I am wrong). Our labels should now contain one 0 (which is the target) and rest all as -infinity.
The cross-entropy loss function is given as:
$ - \Sigma \ t_i*log(o_i) $
where $ t_i $ is the target label for the $ i^{th} $ training sample,
and $ o_i $ is the predicted output for the $ i^{th} $ training sample.
Now, when we use our target label (which is in the range -infinity and 0), the loss will become +infinity because of the -infinity term in the target vector and thus becoming numerically unstable. What is the way around this?
Another question - How does Pytorch handle this? Also, the nn.CrossEntropyLoss()
function calculates the log_softmax
on the predicted outputs internally but I cannot find anywhere in their documentation as to where they convert the one-hot target labels (range 0 to 1); which we pass to the loss function; to a label with range -infinity to 0. Am I wrong in assuming that the target label needs to be changed?
Any help will be appreciated. Thanks!