I am reading the paper Regularizing Neural Networks by Penalizing Confident Output Distributions where the authors introduce label smoothing in section 3.2. For a neural network that produces a conditional distribution $p_\theta(y|x)$ over classes $y$ given an input $x$ through a softmax function, the label smoothing loss function is defined as:

$$\mathcal L(\theta) = -\sum \log p_\theta(y|x) - D_{\mathrm{KL}}(u||p_\theta(y|x))$$

where $D_{\mathrm{KL}}$ refers to the KL divergence and $u$ the uniform distribution. However my understanding is that minimising this expression would in fact attempt to maximise the KL divergence and, since this is a measure of the dissimilarity between the posterior distribution and the uniform distribution, this would encourage the opposite of smoothing. Where is my understanding falling down here?

Additional investigation

Trying to get to the bottom of this I noticed a few things. In the next line of the paper the authors mention that

By reversing the direction of the KL divergence, $D_{\mathrm{KL}}(p_\theta(y|x)‖u)$, we recover the confidence penalty.

Where, for entropy function $H$ and constant $\beta$, the confidence penalty is defined as $$\mathcal L(\theta) =−\sum \log p_\theta(y|x)−\beta H(p_\theta(y|x))$$

However when I do the derivation myself I obtain

$$\mathcal L(\theta) =−\sum \log p_\theta(y|x) + H(p_\theta(y|x))$$

Since the experiments all use positive valued $\beta$'s this suggests to me that perhaps the original equation is in fact a typo and should be adding the KL divergence rather than subtracting.

I have checked all the versions of the paper I could find online and the original label smoothing equation always subtracts the KL divergence.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.