In the context of cross-entropy loss objectives for neural networks, I tend to think of label smoothing from the standpoint of directly manipulating the labels. This blog post explains how doing so is equivalent to adding an extra loss term, that is the the cross-entropy loss between the uniform noise distribution and the model output distribution:

enter image description here

I can follow this just fine.

But when I look at Regularizing Neural Networks By Penalizing Confident Output Distributions it talks about the extra term being a KL divergence instead (see section 3.2). In the latter case, I can't make the connection.

I have a related question on this.

But now I'm just totally confused about whether label smoothing is equivalent to adding a cross entropy term, or if it's equivalent to adding a KL divergence term.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.