2
$\begingroup$

I am using nolearn with Lasagne to train a simple Multilayer-Perceptron (MLP) for the MNIST dataset. I get about 97% accuracy on the test set after training on the training set, which is a few thousand samples. Probably as good as it can get without using a Convolutional neural network (CNN).

I have created a simple paint program with pygame that lets me draw a digit and get it classified with my net.

Every time I want to classify a digit the following happens:

  1. Resize the draw surface to 28x28 (what they are in MNIST)
  2. Rotate and flip the draw surface to get correct orientation.
  3. Transform surface to a numpy array of pixels.
  4. Transform RGB to grayscale.
  5. Divide the array by 255 to scale between 0 and 1 (as what I did with the MNIST training set)
  6. Reshape array from 28x28 to 784x1 as what is expected of the net.

I have verified that my array is correct, in the sense that I have compared it against random training / test samples, and also written it out as a image to really be sure it is correctly transformed before using it.

I have tried to draw perfect digits, as well as possible and centered as possible with a good line width. I have tried to draw after samples in the MNIST dataset.

The problem is that I get very bad performance from my net of the drawn digits, way lower than expected. I can draw just one line in the middle and expect it to classify it as 1 with high probability, only to find out it classify it as a 7 or a 5 with mediocre probability. The inspiration for doing this comes from this video and clearly you can see how well it performs, though I think he does everything from scratch.

I have also tried different neural-network libraries but they perform the same.

$\endgroup$
5
  • $\begingroup$ Please spell out your acronyms. What is MLP (multi-layer perceptron)? What is CNN? $\endgroup$ Commented Sep 29, 2015 at 18:02
  • 3
    $\begingroup$ CNN is a news network. $\endgroup$
    – Nick Cox
    Commented Sep 29, 2015 at 18:19
  • $\begingroup$ I have updated it, CNN is not a news network in this context. $\endgroup$
    – whereami
    Commented Sep 29, 2015 at 18:40
  • $\begingroup$ Try using T-SNE (lvdmaaten.github.io/tsne) to visualize your data. If you don't get clear separation in the visualization, your data may not be easily separable for some reason, and that may become clear given the clustering the T-SNE does. $\endgroup$ Commented Jun 18, 2016 at 23:36
  • $\begingroup$ @whereami Something's wrong with your model and how you trained it. You need to know this stunning fact : just a purely linearly model of the MNIST pixel could get test error down to 1.2%. This is reported in a recent paper from Google. $\endgroup$
    – horaceT
    Commented Jan 27, 2018 at 22:08

1 Answer 1

1
$\begingroup$

I am facing the same problem with a neural net I built from scratch in C++. After running the MNIST training set through it, it will read numbers correctly from the test set around 70% of the time. But when I draw my own images (one of each digit), resize them to 28x28, and test these, it only reads one of them correctly.

I looked at my images beside MNIST images of the same digits to compare and noticed that my images were much more blurred, often with unwanted texture that made the numbers look ambiguous when zoomed in, though zoomed out they looked fine.

So, two strategies I found to get better results:

  1. Paint your digits directly in the correct size of 28x28. This will avoid blurring caused by resizing. After doing this, my neural net reads 3 of my hand-drawn digits correctly (still not as good as 70%, but I'll take it).
  2. Use the pencil tool instead of the paintbrush tool. This avoids unwanted textures. After this, my neural net reads 4 of my hand-drawn digits correctly (5 if I decrease the number of hidden nodes in my network, maybe this has to do with overfitting).

Sorry these are small numbers, so not as statistically reassuring as I would like, but I think these tips are an important step in the right direction.

I also noticed my drawings ended up with holes in them after formatting from .png or .jpg to .gray, which is what I'm using because it's very easy to read. So if you can avoid formatting, that may have potential for significantly better results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.