I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the network’s predicted distribution p.

$L = -\sum{log(p)-D_{KL}(u, p)}$

$u$ is the uniform distribution. On the other hand we know that the label smoothing has a distribution of $1-\epsilon$ for the target and $\epsilon/(V-1)$ for the rest of the labels. Given this I can't infer the formula above from the label smoothing value. Can anyone knows how to solve this problem?


The softmax function assign to each object x and to each possible class c a value 0 ≤ p(c|x) ≤ 1. In predicting the class of the object x, using the NN, we simply take ˆc = argmaxc p(c|x); we do not care which is the value of p(cˆ|x), we care only that ˆc is the class such that p(cˆ|x) > p(c|x) for all c = cˆ; furthermore, in measuring the accuracy of the prediction we do not care how high is the p(cˆ|x). For example, suppose there are 10 different classes, that is K = 10, and suppose that p(cˆ|x) = 0.2 and p(c|x) = 0.8/(K − 1) for c = cˆ. If cˆ = f(x) then we consider the network very accurate in predicting the class of x, even tough 0.2 is very far from 1.0. This is sometimes similar in what a human do in recognize an image. Sometimes we have not the certainty that the class of an image is a deer but we can exclude that it is a dog and it is a horse, so we classify it as a deer. So, in order to apply this idea we use, as a loss function, the cross-entropy between the soft max function and a probability distribution other than

q(j|x) =  1  if f(x) = j
              0   otherwise.

If 0 ≤ γ ≤ 1 then we may choose q as

q(j|x) =   γ            if f(x) = j
          (1−γ)/(K −1) otherwise 

with the constraint that γ > 1/K. The above methods can be implemented in existing software almost effortless. We made extensive experiments using state of the art NN for classification on several data sets. We report the results of the experiments in the following section. We found that, in many settings, there is a value of γ that improve the accuracy of the net with respect to the categorical cross-entropy.

For further read


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.