Label smoothing formula I recently came across the paper Regularizing Neural Networks by Penalizing Confident Output Distributions. In section 3.2, it claims that label smoothing loss is equivalent to adding the KL divergence of the uniform distribution $u$ from the network’s learned distribution $p$.
$$L(\theta) = -\sum \log(p_\theta(y|x))-D_{\mathrm{KL}}(u \parallel p_\theta(y|x))$$
On the other hand, we know that the label smoothing gives a probability of $1-\epsilon$ for the target and $\epsilon/(V-1)$ for the rest of the labels. Given this, I can't infer the formula above from the label smoothing value. Can anyone knows how to solve this problem?
 A: The softmax function assign to each object $x$ and to
each possible class $c$ a value $0 ≤ p(c|x) ≤ 1$. In predicting the class of the object $x$, using the NN, we simply take $ˆc = argmax_c(p(c|x))$; we do not care which is
the value of $p(cˆ|x)$, we care only that $ˆc$ is the class such
that $p(cˆ|x) > p(c|x)$ for all $c = cˆ$; furthermore, in measuring the accuracy of the prediction we do not care
how high is the $p(cˆ|x)$. 
For example, suppose there are
10 different classes, that is $K = 10$, and suppose that
$p(cˆ|x) = 0.2$ and $p(c|x) = 0.8/(K − 1)$ for $c = cˆ$. If
$cˆ = f(x)$ then we consider the network very accurate
in predicting the class of $x$, even tough $0.2$ is very far
from $1.0$. 
This is sometimes similar in what a human
do in recognize an image. Sometimes we have not the
certainty that the class of an image is a deer but we can
exclude that it is a dog and it is a horse, so we classify
it as a deer.
So, in order to apply this idea we use, as a loss function,
the cross-entropy between the soft max function  and
a probability distribution other than 
q(j|x) =  1  if f(x) = j
          0  otherwise.

If $0 ≤ γ ≤ 1$ then we may choose $q$ as
q(j|x) =  γ            if f(x) = j
          (1−γ)/(K −1) otherwise 


with the constraint that $γ > 1/K$. 
The above methods
can be implemented in existing software almost effortless. We made extensive experiments using state of the
art NN for classification on several data sets. We report
the results of the experiments in the following section.
We found that, in many settings, there is a value of γ
that improve the accuracy of the net with respect to the
categorical cross-entropy.
For further read
