How to deal with anti-aliasing in MNIST images?

Question

I'm using MNIST dataset (http://yann.lecun.com/exdb/mnist/) to train a neural network. As pointed out in the site, images in the MNIST dataset were compressed from the original NIST images (which are bilevel: only black & white), resulting in grey-scale anti-aliasing artifacts in the images:

The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm.

Now, after training my neural network, I want to use it to recognize actual handwritten digits by me (in particular, I'm using a TFT touchscreen shield in an Arduino to capture this). My captured digits do not suffer from anti-aliasing because I normalize captured values to 28x28 in Arduino code. In other words, my images are binary/bilevel, like the original NIST images, but not the ones in MNIST.

MNIST seems to be the most popular choice for this type of problem, but I don't understand how people using it deal with the fact that new digits will likely not have the same anti-aliasing artifacts than the training dataset (or even not have anti-aliasing at all), and AFAIK this causes huge differences in how for example a neural network will perform.

I could capture a bigger image and compress it, thus generating anti-aliasing pixels on purpose, but I wouldn't know how to guarantee that the same anti-aliasing algorithm that was used by MNIST will be applied, resulting in more differences/noise in the capture data.

How should I approach this?

EDIT: I've found a related question in StackOverflow, I add it here also for more context: https://stackoverflow.com/questions/27925358/pre-processing-before-digit-recognition-for-nn-cnn-trained-with-mnist-dataset

Answer 1

The short answer to this is that if you train a neural network to recognize MNIST images, then it will work well for precisely that (i.e., recognize more MNIST images).

Now, it seems a little disappointing that we cannot use our awesome neural network that has probably an accuracy near 95% or even 99%, to recognize new handwritten digits. Who wants to be able to recognize only images in the MNIST dataset?

So, after researching a little bit more, the general consensus seems to be that if you want your neural network to be able to recognize your own handwritten digits, you have two options:

1) include enough of your own digits into the training set, along with the MNIST ones.

or

2) pre-process your own handwritten digits so they resemble engough to MNIST digits, thus fitting what the neural network has been trained for.

We could even generalize this not only for MNIST but for any domain. Either you include enough variations in your training set (enrich your training set), or you choose one data representation and pre-process any other variation to match the one you chose (pre-process your input).

I found this article precisely about how to do pre-processing to match MNIST data representation.

I personally cannot help but feel that pre-processing is kind of "cheating", because the whole point of neural networks is not having to resort to this "algorithmic" type of solutions. It would be nicer to be able to train a neural network to be able to generalize enough so any data representation (within the limits of a particular domain, like handwritten digits) will work. But it seems this are the current state-of-the-art approaches, and for some domains you'll have to go one way or the other to get good results.

EDIT: as pointed out by a StackOverflow user in the comments section of this question, this can be seen as a "Domain Adaptation" problem. I'm currently learning about Domain Adaptation techniques, in particular "Domain-Adversarial training of neural networks" (google this for several papers on the subject). In particular I'm looking at this GitHub repo that implements a Domain-Adversarial example using Tensorflow. I added this so others can be pointed in the same direction.

Answer 2

You can approach this in two different ways:

1. preprocess your data to look as similar as possible to the mnist data (at inference time)
2. fine tune (transfer learning) the pre-trained MNIST model you intend to use to better fit your data

In this case number 1 means adding some antialias, you could do this with upscale, blur, downscale or something like that.

Number 2 means to train again the model, freezing a few layers or not, with your data only for a few epochs.

In practice neural networks generalize well, in this case it probably does not make much difference if your images are antialiased or not. The only way to find out is to test your dataset with different preprocessing.

Other aspects, like size and location, may matter more. A good model was probably trained with a lot of augmented data that can be extremely different from the base digits (shear, blur, noise, masking, etc.)

If a little difference in antialias/compression artifacts would be enough to sway a model machine learning would be pretty useless.