7
$\begingroup$

I tried to train a deep belief network to recognize digits from the MNIST dataset. Everything works OK, I can train even quite a large network. The problem is that the best DBN is worse than a simple multilayer perceptron with less neurons (trained to the moment of stabilization). Is this normal behaviour or did I miss something?

Here is an example: DBN with layers 784-512-512-64-10 (red/green/black - 100/200/400 iterations of RBM) vs MLP 784-512-256-10 (blue line).

Plot of the error(iteration):

enter image description here

I tested many more configurations and the MLP seems to be better almost always (if the size of the MLP is not too large so it can be trained).

Improve this question
$\endgroup$
4
  • $\begingroup$ How exactly did you train the DBN? For classification, a DBN is typically only used to initialize an MLP. Is your problem that initializing with DBN parameters leads to worse results than initializing randomly? $\endgroup$
    – Lucas
    Jan 12, 2013 at 19:53
  • $\begingroup$ Also, are you showing us the training or test error? $\endgroup$
    – Lucas
    Jan 12, 2013 at 19:58
  • $\begingroup$ It's test error. And problem is exactly as said Lucas, random initialization is better than RBM pretraining. $\endgroup$
    – kolar
    Jan 12, 2013 at 20:19
  • $\begingroup$ maybe it's just some case of overfitting? How did you test performance? $\endgroup$
    – mrgloom
    Feb 26, 2014 at 8:41

2 Answers 2

Reset to default
3
$\begingroup$

My guess is that your training procedure is simply unable to find good parameters for such a big model. It is quite hard to get an MLP with more than three layers to work well on image classification problems, even when pretraining with a DBN. Most papers on image classification I have seen use three layers or even show that the performance decreases when you use more layers (this paper for example, table 6). So yes, this behavior is kind of normal.

It also fits with my experience with DBNs. After two or three layers, the generative performance of the DBN saturates or even decreases.

Another hint that you are likely facing an underfitting problem is your performance. Yann LeCun's website shows that you can get below 3% error even with fairly small 3-layer MLPs. Your error on the other hand seems to stay well above 5% for all of your models.

My suggestion would therefore be to stick with a smaller model or switch to more powerful optimization techniques.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks. I made mistake and figure from my question is test on MNIST with randomly rotated digits (I thought that dbn may be better on some harder dataset). I'll make new plot soon but my results from orginal MNIST was below 2% for MLP and about 2.2% for DBN which is comparable to results from cs.toronto.edu/~hinton/absps/fastnc.pdf $\endgroup$
    – kolar
    Jan 12, 2013 at 20:32
  • $\begingroup$ Maybe you can also add the training error. If there it's the other way round, it might actually be an overfitting and not an underfitting problem. $\endgroup$
    – Lucas
    Jan 12, 2013 at 20:43
  • $\begingroup$ Here are train and test errors: i46.tinypic.com/rkvvag.png $\endgroup$
    – kolar
    Jan 12, 2013 at 22:11
  • $\begingroup$ the 'more powerful optimization techniques' link (J. Martens, "Deep learning via Hessian-free optimization") seems to have moved to icml2010.haifa.il.ibm.com/papers/458.pdf $\endgroup$
    – Andre Holzner
    May 14, 2016 at 17:28
0
$\begingroup$

I'm seeing similar results but with a much simpler network using only a single hidden layer. I'm doing some tests on the CIFAR-10 image database. My network is 768-64-1, a binary classifier. If I use a RBM to pre-train layer 768-64 + backprop I get an F1 score of about 0.38 on validation/test. Where as using random weights with no pre-training and backprop I get an F1 score of about 0.45.

The weights learnt by the RBM are more meaningful visually, you can make out features that correspond to the data. Where as the random weights don't evolve past randomness much, yet perform better.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.