I know that the Huber loss is usually applied on top of the L2 loss in order to prevent exploding gradients. Does it make sense to use the Huber loss on top of the cross entropy loss, though? I have a feeling that it is not very sensible, but a more scientific argument will be much more persuasive.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In practice, I have never had problems with large gradients / errors when using cross-entropy. That's because, when used in conjunction with a softmax layer, the gradients of the loss are constant with respect to both the predicted probability and the target labels. As opposed to gradients which grow linearly in the error, for L2 loss.
When the noise in a continuous random variable is normally distributed, then L2 loss is optimal in that it maximizes the likelihood (see this post for more detail).
However, if the noise is not normally distributed, you might get values which are very far from your prediction, causing a large error / gradient. So the Huber loss makes sense when you expect these extreme events to be more frequent. The L1 loss, which is close to the Huber loss, gives the maximum likelihood for the laplace distribution, which has tails of $O(e^{-|x|})$ as opposed to the normal distribution's $O(e^{-x^2})$.
However, there's not really an equivalent interpretation for cross-entropy, since our justifications above rely on the notions of heavy tailed noise, which doesn't appear in a categorical distribution. The closest thing might be noisy categorical labels, for which one approach is label smoothing (see this post)
