Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I tried to train a deep belief network to recognize digits from the MNIST dataset. Everything works OK, I can train even quite a large network. The problem is that the best DBN is worse than a simple multilayer perceptron with less neurons (trained to the moment of stabilization). Is this normal behaviour or did I miss something?

Here is an example: DBN with layers 784-512-512-64-10 (red/green/black - 100/200/400 iterations of RBM) vs MLP 784-512-256-10 (blue line).

Plot of the error(iteration):

enter image description here

I tested many more configurations and the MLP seems to be better almost always (if the size of the MLP is not too large so it can be trained).

share|improve this question
How exactly did you train the DBN? For classification, a DBN is typically only used to initialize an MLP. Is your problem that initializing with DBN parameters leads to worse results than initializing randomly? –  Lucas Jan 12 '13 at 19:53
Also, are you showing us the training or test error? –  Lucas Jan 12 '13 at 19:58
It's test error. And problem is exactly as said Lucas, random initialization is better than RBM pretraining. –  kolar Jan 12 '13 at 20:19
maybe it's just some case of overfitting? How did you test performance? –  mrgloom Feb 26 at 8:41

2 Answers 2

up vote 2 down vote accepted

My guess is that your training procedure is simply unable to find good parameters for such a big model. It is quite hard to get an MLP with more than three layers to work well on image classification problems, even when pretraining with a DBN. Most papers on image classification I have seen use three layers or even show that the performance decreases when you use more layers (this paper for example, table 6). So yes, this behavior is kind of normal.

It also fits with my experience with DBNs. After two or three layers, the generative performance of the DBN saturates or even decreases.

Another hint that you are likely facing an underfitting problem is your performance. Yann LeCun's website shows that you can get below 3% error even with fairly small 3-layer MLPs. Your error on the other hand seems to stay well above 5% for all of your models.

My suggestion would therefore be to stick with a smaller model or switch to more powerful optimization techniques.

share|improve this answer
Thanks. I made mistake and figure from my question is test on MNIST with randomly rotated digits (I thought that dbn may be better on some harder dataset). I'll make new plot soon but my results from orginal MNIST was below 2% for MLP and about 2.2% for DBN which is comparable to results from cs.toronto.edu/~hinton/absps/fastnc.pdf –  kolar Jan 12 '13 at 20:32
Maybe you can also add the training error. If there it's the other way round, it might actually be an overfitting and not an underfitting problem. –  Lucas Jan 12 '13 at 20:43
Here are train and test errors: i46.tinypic.com/rkvvag.png –  kolar Jan 12 '13 at 22:11

I'm seeing similar results but with a much simpler network using only a single hidden layer. I'm doing some tests on the CIFAR-10 image database. My network is 768-64-1, a binary classifier. If I use a RBM to pre-train layer 768-64 + backprop I get an F1 score of about 0.38 on validation/test. Where as using random weights with no pre-training and backprop I get an F1 score of about 0.45.

The weights learnt by the RBM are more meaningful visually, you can make out features that correspond to the data. Where as the random weights don't evolve past randomness much, yet perform better.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.