In this introduction to neural networks (I enjoy it because it builds a digit-recognition neural network from scratch with just numpy, without any high-level NN library like pytorch or tensorflow; thus it helps to understand the internals), we import images from the well-known MNIST digits dataset, but instead of mapping the pixels grayscale to [0.00, 1.00], it does it to [0.01, 0.99]:

We map the values of the image data into the interval [0.01, 0.99] by dividing the train_data and test_data arrays by (255 * 0.99 + 0.01)

Then we create the one hot representation of digits, but again, avoiding 0.00 and 1.00:

We are ready now to turn our labelled images into one-hot representations. Instead of zeroes and one, we create 0.01 and 0.99, which will be better for our calculations:

lr = np.arange(no_of_different_labels)
# transform labels into one hot representation
train_labels_one_hot = (lr==train_labels).astype(np.float)
test_labels_one_hot = (lr==test_labels).astype(np.float)
# we don't want zeroes and ones in the labels neither:
train_labels_one_hot[train_labels_one_hot==0] = 0.01
train_labels_one_hot[train_labels_one_hot==1] = 0.99
test_labels_one_hot[test_labels_one_hot==0] = 0.01
test_labels_one_hot[test_labels_one_hot==1] = 0.99

I tried both:

  • with [0.00, 1.00] mapping => 94.0% accuracy
  • with [0.01, 0.99] mapping => 94.5% accuracy

so it seems to confirm that this little trick improves a little bit the accuracy.

Why and how does this work?


This appears to be a special case of label smoothing ($K=2, \epsilon=0.02$), which is used in some computer vision networks (but I suppose it could be used for other problems, too). The basic idea is that for some $0 < \epsilon \ll 1$, you use labels that are a convex combination of one-hot labels and a uniform weighting of all labels:

$$ \left[ \frac{\epsilon}{K}, \frac{\epsilon}{K}, \dots, 1-\frac{\epsilon(K-1)}{K}, \dots,\frac{\epsilon}{K} \right] $$

where $K$ is the number of classes. In probabilistic terms, this is a mixture of the Dirac distribution at the class label with a uniform distribution over all labels, with the mixture governed by $\epsilon$. The effect is to penalize a model which is overly confident in any single class, because the target value for a correct label is $1-\frac{\epsilon(K-1)}{K} < 1$ instead of 1 in the ordinary binary label case.

It's one of the tricks highlighted in this review and it appears to originate in "Rethinking the Inception Architecture for Computer Vision" by Christian Szegedy et al. as a certain kind of regularization.


For the inputs, it is equivalent to scaling the initial weight matrix. Neural networks can be sensitive to such things, but that is a reason why it is good practice to try different variances for the weight scale, rely on batch normalisation (or its cousins layer and weight normalisation).

For the one hot targets, it changes the loss function a little. Instead of just maximising the likelihood, it mixes in a term to follow the likelihood as if the label was wrong. I don't have a proper justification for this, but maybe it make gradients smoother.

I have never encountered both "tricks" before, and I am highly skeptical the author know what he/she is doing. While it makes sense for the outputs, as it keeps a $\log$ from exploding, I don't see why it should be done for the targets and for the inputs.

On a side, the transformation of the inputs does not place the inputs in the $[0.01, 0.99]$, but in $[0.01, 1.]$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.