3
$\begingroup$

When I started with ANN I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even nearly beat my score on random forest!

I'm seeking some very general or not so general advice on what should one do to make his NN start capturing trends in data.

For implementing NN I use Theano Stacked Auto Encoder, the code from tutorial that works great (less than 5% error rate) on classifying MNIST dataset. It is a multilayer perceptron, with softmax layer on top with each hidden later being pre-trained as autoencoder (fully described at tutorial, chapter 8). The number of input features are ~50 and output classes ~10. NN has sigmoid neurons and all data normalized to [0,1]. I tried lots of different configurations: number of hidden layers and neurons in them (100->100->100, 60->60->60, 60->30->15, etc,etc), different learning and pre-train rates, etc.

And the best thing I can get is %20 error rate on validation set and %40 error rate on test set.

On the contrary, when I try to use Random Forest (from scikit-learn) I easily get %12 error rate on validation set and %25(!) on test set.

How can that be that my deep NN with pre-training behaves so badly? What should I try?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You do not say how many data-points you have but I suspect you need more data. Deep neural networks excel in huge datasets. $\endgroup$
    – usεr11852
    Commented Mar 3, 2015 at 21:01
  • $\begingroup$ Which dataset are you training on? Not every neural net works for every problem $\endgroup$
    – Bananin
    Commented Apr 16, 2020 at 15:02

1 Answer 1

Reset to default
2
$\begingroup$

The most obvious issue that I see is the size of your hidden layers. They're generally trained on a GPU and the training can take hours, days, and sometimes weeks.

The "capacity"of your network is not large enough to fully represent the underlying variability in the data (MNIST). Try increasing the size of your hidden layers.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.