In the context of cross-entropy loss objectives for neural networks, I tend to think of label smoothing from the standpoint of directly manipulating the labels. This blog post explains how doing so is equivalent to adding an extra loss term, that is the the cross-entropy loss between the uniform noise distribution and the model output distribution:

\begin{align}L^\prime &= \sum_{i=1}^n\left\{(1-\varepsilon)\left[-\sum_{y=1}^Kp(y|x_i)\log q_\theta(y|x_i)\right]+\varepsilon\left[-\sum_{y=1}^Ku(y|x_i)\log q_\theta(y|x_i)\right]\right\}\\ &= \sum_{i=1}^n\left [(1-\varepsilon)H_i(p,q_\theta)+ \varepsilon H_i(u, q_\theta)\right]\end{align}

I can follow this just fine.

But when I look at Regularizing Neural Networks By Penalizing Confident Output Distributions, it talks about the extra term being a KL divergence instead (see section 3.2). In the latter case, I can't make the connection.

I have a related question on this.

But now I'm just totally confused about whether label smoothing is equivalent to adding a cross entropy term, or if it's equivalent to adding a KL divergence term.


1 Answer 1


You are just one step away! Following Lei Mao's derivation $\eqref{eq1}$, we can replace the second term in the right hand with $\eqref{eq2}$. $$L^{'} = \sum_{i=1}^n \left[ (1-\epsilon)H_i(p, q_\theta) + \epsilon H_i(u, q_\theta)\right] \tag{1}\label{eq1}$$ $$H(u, q) = D_{KL}(u || q) + H(u) \tag{2} \label{eq2}$$ Voilà, we have $\eqref{eq3}$ in which the term $H(u)$ is a constant during the optimization when $u$ is a uniform distribution. Therefore, we can neglect it. Eventually, $\eqref{eq4}$ is what the paper Regularizing Neural Networks By Penalizing Confident Output Distributions actually refers to. $$L^{'} = \sum_{i=1}^n \left[ (1-\epsilon)H_i(p, q_\theta) + \epsilon D_{KL}(u || q_\theta) + \epsilon H(u) \right] \tag{3}\label{eq3}$$ $$L^{'} = \sum_{i=1}^n \left[ (1-\epsilon)H_i(p, q_\theta) + \epsilon D_{KL}(u || q_\theta) \right] \tag{4}\label{eq4}$$ However, the equation in Section 3.2 of the paper 1 omitted the smoothing factor for unknown reasons. It would be more understandable if reformulated as $\eqref{eq5}$ which is quite close to $\eqref{eq4}$. Hope this clarifies your confusion ;) $$\begin{equation}\begin{aligned} L(\theta) &= - \sum \left[ \text{log} q_{\theta}(\mathbf{y} | \mathbf{x}) - D_{KL}(u || q_\theta (\mathbf{y} | \mathbf{x})) \right] \\ &= \sum \left[ - \text{log} q_{\theta}(\mathbf{y} | \mathbf{x}) + D_{KL}(u || q_\theta (\mathbf{y} | \mathbf{x})) \right] \\ &= \sum \left[ H(p, q_\theta(\mathbf{y} | \mathbf{x})) + D_{KL}(u || q_\theta (\mathbf{y} | \mathbf{x})) \right] \end{aligned}\end{equation}\tag{5}\label{eq5}$$ Last but not least, back to your final question, both of your statements hold when the noise distribution $u$ is a uniform distribution.

  • 1
    $\begingroup$ Thanks! This was over a year ago so I'll need to find some time to have a look at it again before accepting your answer, but it seems like you got it :) $\endgroup$ Sep 9, 2022 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.