Multilayer-Perceptron (MLP) performs poorly for digit recognition

Question

I am using nolearn with Lasagne to train a simple Multilayer-Perceptron (MLP) for the MNIST dataset. I get about 97% accuracy on the test set after training on the training set, which is a few thousand samples. Probably as good as it can get without using a Convolutional neural network (CNN).

I have created a simple paint program with pygame that lets me draw a digit and get it classified with my net.

Every time I want to classify a digit the following happens:

1. Resize the draw surface to 28x28 (what they are in MNIST)
2. Rotate and flip the draw surface to get correct orientation.
3. Transform surface to a numpy array of pixels.
4. Transform RGB to grayscale.
5. Divide the array by 255 to scale between 0 and 1 (as what I did with the MNIST training set)
6. Reshape array from 28x28 to 784x1 as what is expected of the net.

I have verified that my array is correct, in the sense that I have compared it against random training / test samples, and also written it out as a image to really be sure it is correctly transformed before using it.

I have tried to draw perfect digits, as well as possible and centered as possible with a good line width. I have tried to draw after samples in the MNIST dataset.

The problem is that I get very bad performance from my net of the drawn digits, way lower than expected. I can draw just one line in the middle and expect it to classify it as 1 with high probability, only to find out it classify it as a 7 or a 5 with mediocre probability. The inspiration for doing this comes from this video and clearly you can see how well it performs, though I think he does everything from scratch.

I have also tried different neural-network libraries but they perform the same.

Answer 1

I am facing the same problem with a neural net I built from scratch in C++. After running the MNIST training set through it, it will read numbers correctly from the test set around 70% of the time. But when I draw my own images (one of each digit), resize them to 28x28, and test these, it only reads one of them correctly.

I looked at my images beside MNIST images of the same digits to compare and noticed that my images were much more blurred, often with unwanted texture that made the numbers look ambiguous when zoomed in, though zoomed out they looked fine.

So, two strategies I found to get better results:

1. Paint your digits directly in the correct size of 28x28. This will avoid blurring caused by resizing. After doing this, my neural net reads 3 of my hand-drawn digits correctly (still not as good as 70%, but I'll take it).
2. Use the pencil tool instead of the paintbrush tool. This avoids unwanted textures. After this, my neural net reads 4 of my hand-drawn digits correctly (5 if I decrease the number of hidden nodes in my network, maybe this has to do with overfitting).

Sorry these are small numbers, so not as statistically reassuring as I would like, but I think these tips are an important step in the right direction.

I also noticed my drawings ended up with holes in them after formatting from .png or .jpg to .gray, which is what I'm using because it's very easy to read. So if you can avoid formatting, that may have potential for significantly better results.