5
$\begingroup$

I am trying to use a convolutional neural network (implemented with keras) to solve a modified version of the MNIST classification problem (I am trying the background variations as described here). I started from this example and played around a bit with the parameters to get better accuracies, but I seem to get stuck at about 90% accuracy on my validation set.

I've read papers who manage to get near-human accuracy on those datasets, but I seem not to be able to improve my network to get over 95% (something I would expect to be possible). Because I have only been guessing the parameters for the network thus far and I don't seem to find anything online, I was wondering whether there are any guidelines to find a good architecture and good parameters for convolutional neural networks.

Would anybody be aware of how to tackle the problem of finding good parameters (including architecture) for a CNN apart from trying?

|cite|improve this question
$\endgroup$
2
$\begingroup$

Would anybody be aware of how to tackle the problem of finding good parameters (including architecture) for a CNN apart from trying?

No. Hence, they are optimized by 'graduate student descent' :)

A standard-ish architecture for mnist is lenet-5, and closely related variants, eg Karpathy's convnetjs implementation, which uses the following layers:

type:'input', out_sx:24, out_sy:24, out_depth:1
type:'conv', sx:5, filters:8, stride:1, pad:2, activation:'relu'
type:'pool', sx:2, stride:2
type:'conv', sx:5, filters:16, stride:1, pad:2, activation:'relu'
type:'pool', sx:3, stride:3
type:'softmax', num_classes:10

(and I think it augments the data by cutting random 24x24 patches out of the original 28x28 images, which is a key part of obtaining higher accuracies on this (tiny) dataset).

There are meta-learning techniques, which is an open research area. For example "Neural architecture search with reinforcement learning", by Barret Zoph and Quoc Le, 2016, uses reinforcement learning to try different architectures, find out what works well. It does this in an automated way, without human intervention. Of course this needs a ton of GPU power...

|cite|improve this answer
$\endgroup$
0
$\begingroup$

You might decorrelate your data first using PCA, and then clamp the objects to your input nodes (i.e., input the PCs from PCA into the CNN). Did you select any features, or use everything? (don't know if the features were pre-selected and users are expected to use everything?).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I did not do any preprocessing thus far in the assumption that the CNN would learn interesting features (is that a bad assumption?) $\endgroup$ – Mr Tsjolder May 30 '16 at 19:28
  • $\begingroup$ Depends on whether or not your CNN approach has a "wrapper" for feature selection. If not, you can employ "filtering" and use an entirely separate technique to identify features. $\endgroup$ – JoleT Jun 1 '16 at 12:11
  • $\begingroup$ Rajiv Shah gave a very nice presentation at a tensor flow meetup about using pre-trained CNN's to very quickly get superior results. He did show how to apply this to mnist as well as another project. You will have to do a little googling to find the presentation, but I think it will answer your question. $\endgroup$ – aginensky Aug 18 '17 at 18:11
0
$\begingroup$

I would suggest you try another data set. MNIST data has been "over tuned" on test data set!!

You can try to run a test for "human accuracy" on this data, and you may not get over 95% accuracy. BTW, I tried, there are many digits are not quite recognizable. Here is an example, it can be 3 or 5.

enter image description here

In sum, today's NN tools are really good, that only needs few changes you may get very good results especially on classic data set. My suggestions to you would not be push the limit of the model on MINIST data but try more other data sets.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ As MNIST is a toy dataset, I just wanted to play around a bit and test how well I could train a CNN. I am very well aware that my CNN does not have that much of a purpose. Do you imply that human-accuracy can only be attained when overfitting? $\endgroup$ – Mr Tsjolder Dec 28 '16 at 7:36
  • $\begingroup$ @MrTsjolder I am saying you are over doing it. If you really want to learn NN trying other data sets may be better than doing this. Making it better than 95% is almost similar to trying to capturing the noise in data... $\endgroup$ – hxd1011 Dec 28 '16 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.