I have a large dataset (tens of thousands of predictors) on which I would like to perform feature reduction with the intent of better model-building for prediction. Deep Belief Networks seem to address the problem in such a way that they produce an output that bears some resemblance to that of PCA, i.e. a series of 'virtual' dimensions that can be used to explain variation in the data.

Unlike PCA however, it seems that every example of DBNs follow the original example of Hinton, being used to generate a series of starting weights for a feed-forward neural network. Is there a reason that nobody appears to use stacked RBMs as, say, input for linear regression or k-nearest neighbor?


1 Answer 1


Note that initialising neural nets with DBNs is more of a historical anecdote nowadays. Direct supervised training with dropout regularization and piecewise linear activation functions tend to work much better in the presence of many labeled training examples.

More direct answer to your question:

1) Using DBN features for linear regression or logistic regression is nothing but fine tuning the net. The difference is that you also adapt the parameters of the full hierarchy if you initialize a neural net.

2) KNN has been used on top of a neural net, see (Salakhutdinov, Ruslan, and Geoffrey E. Hinton. "Learning a nonlinear embedding by preserving class neighbourhood structure." International Conference on Artificial Intelligence and Statistics. 2007.) The whole method is more complicated though, as the DBN is further fine tuned for KNN usage.

3) DBNs have also been used as inputs to Gaussian processes (Salakhutdinov, Ruslan, and Geoffrey E. Hinton. "Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes." NIPS. 2007.)

4) And deep networks have also been used to learn the Kernel for SVMs. (Yichuan Tang "Deep Learning using Linear Support Vector Machines")

Other methods, such as GBMs or RFs, seem not to be considered since they are not differentiable and thus fine tuning the feature extractor is not possible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.