Why map the pixel grayscale [0, 1] to [0.01, 0.99] before feeding to the neural network? (MNIST digit recognition)

Question

In this introduction to neural networks (I enjoy it because it builds a digit-recognition neural network from scratch with just numpy, without any high-level NN library like pytorch or tensorflow; thus it helps to understand the internals), we import images from the well-known MNIST digits dataset, but instead of mapping the pixels grayscale to [0.00, 1.00], it does it to [0.01, 0.99]:

We map the values of the image data into the interval [0.01, 0.99] by dividing the train_data and test_data arrays by (255 * 0.99 + 0.01)

Then we create the one hot representation of digits, but again, avoiding 0.00 and 1.00:

We are ready now to turn our labelled images into one-hot representations. Instead of zeroes and one, we create 0.01 and 0.99, which will be better for our calculations:

lr = np.arange(no_of_different_labels)
# transform labels into one hot representation
train_labels_one_hot = (lr==train_labels).astype(np.float)
test_labels_one_hot = (lr==test_labels).astype(np.float)
# we don't want zeroes and ones in the labels neither:
train_labels_one_hot[train_labels_one_hot==0] = 0.01
train_labels_one_hot[train_labels_one_hot==1] = 0.99
test_labels_one_hot[test_labels_one_hot==0] = 0.01
test_labels_one_hot[test_labels_one_hot==1] = 0.99

I tried both:

• with [0.00, 1.00] mapping => 94.0% accuracy
• with [0.01, 0.99] mapping => 94.5% accuracy

so it seems to confirm that this little trick improves a little bit the accuracy.

Why and how does this work?

Answer 1

1. We map the values of the image data into the interval [0.01, 0.99] by dividing the train_data and test_data arrays by (255 * 0.99 + 0.01)

This is just a different choice of scaling for the input data. Instead of inputs in the interval [0, 1], you have inputs in the interval [0.01, 1.0]. I doubt this has enormous consequences for the model. Instead, I think the primary driver of the differences you observe is due to the second choice, label smoothing.

2. Then we create the one hot representation of digits, but again, avoiding 0.00 and 1.00:

We are ready now to turn our labelled images into one-hot representations. Instead of zeroes and one, we create 0.01 and 0.99, which will be better for our calculations:

This appears to be a special case of label smoothing ($K=2, \epsilon=0.02$), which is used in some computer vision networks (but I suppose it could be used for other problems, too). The basic idea is that for some $0 < \epsilon \ll 1$, you use labels that are a convex combination of one-hot labels and a uniform weighting of all labels:

$$ \left[ \frac{\epsilon}{K}, \frac{\epsilon}{K}, \dots, 1-\frac{\epsilon(K-1)}{K}, \dots,\frac{\epsilon}{K} \right] $$

where $K$ is the number of classes. In probabilistic terms, this is a mixture of the Dirac distribution at the class label with a uniform distribution over all labels, with the mixture governed by $\epsilon$. The effect is to penalize a model which is overly confident in any single class, because the target value for a correct label is $1-\frac{\epsilon(K-1)}{K} < 1$ instead of 1 in the ordinary binary label case.

It's one of the tricks highlighted in this review ("How to Train Your ResNet 8: Bag of Tricks") and it appears to originate in "Rethinking the Inception Architecture for Computer Vision" by Christian Szegedy et al. as a certain kind of regularization.

Answer 2

For the inputs, it is equivalent to scaling the initial weight matrix. Neural networks can be sensitive to such things, but that is a reason why it is good practice to try different variances for the weight scale, rely on batch normalisation (or its cousins layer and weight normalisation).

For the one hot targets, it changes the loss function a little. Instead of just maximising the likelihood, it mixes in a term to follow the likelihood as if the label was wrong. I don't have a proper justification for this, but maybe it make gradients smoother.

I have never encountered both "tricks" before, and I am highly skeptical the author know what he/she is doing. While it makes sense for the outputs, as it keeps a $\log$ from exploding, I don't see why it should be done for the targets and for the inputs.

On a side, the transformation of the inputs does not place the inputs in the $[0.01, 0.99]$, but in $[0.01, 1.]$.