I'm following up on the question I'd asked earlier on RBMs. I see a lot of literature describing them but none that actually talks of regression (not even classification with labelled data). I get a feeling that it is used for unlabelled data only. Are there any resources for handling regression? Or is it as simple as adding another layer on top of the hidden layer and run the CD algorithm up and down? Thanks much in advance.
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
3
You are right about unlabeled data. RBMs are generative models and most commonly used as unsupervised learners.
When used for constructing a Deep Belief Network the most typical procedure is to simply train each each new RBM one at a time as they are stacked on top of each other. So contrastive divergence isn't going up and down in the sense that I think you mean. It is only working with one RBM at a time, using the hidden layer of the previous topmost RBM as the input for the new topmost RBM. After all this you can either treat the stack of RBM weights as the initial weights for a standard feed forward neural network and train using your labeled data and backpropagation or do something more exotic like use the wake-sleep algorithm. Notice that we haven't used any labeled data up until this last step, that is one of the benefits of these types of models. We can learn a good generative model using lots of unlabeled data and even if our ultimate goal is to have good discriminative performance, it should help.
On the other hand, there are several ways you can use RBMs for classification.
- Train an RBM or a stack of several RBMs. Use the topmost hidden layer as input to some other supervised learner.
- Train an RBM for each class and use the unnormalized energies as input to a discriminative classifier.
- Train the RBM to be a joint density model of P(X, Y). Then given some input x, just pick the class y which minimizes the energy function (normalization isn't a problem here like in the above since the constant Z is the same for all classes).
- Train a discriminative RBM
I would highly suggest you read through the technical report A Practical Guide to Training Restricted Boltzmann Machines by Geoff Hinton. It discusses several of these issues in much greater detail, provides invaluable tips, cites lots of relevant papers, and may help to clear up any other confusion you might have.
-
1$\begingroup$ so can RBMs be used for regression? $\endgroup$ Commented Aug 9, 2016 at 21:38
-
$\begingroup$ How do you train a RBM to be a joint density model? By providing both X and Y as non-hidden inputs, i.e. you train it to be P(X,Y|H) (H is the hidden layer)? $\endgroup$– AkiRossCommented Oct 13, 2016 at 16:11
-
$\begingroup$ I self-reply to my comment: yes, in "a Practical guide to training RBM" (Hinton, 2010), the 3 methods described above are explained in detail and, in the 3rd method, he talks about having "two sets of visible units", where, in addition to data, a softmax label unit is presented to represent the class. $\endgroup$– AkiRossCommented Oct 13, 2016 at 16:30
$\begingroup$
$\endgroup$
Please have a look at implementation , http://code.google.com/p/matrbm/ . It has a classification rmb fit function worth considering. Also let me know if you find better implementations of DBNs and RMBs for classification and regression.
$\begingroup$
$\endgroup$
I think one way you can test neural networks like the Restricted Boltzman Machine (RBM) is to apply them to new sets of data, or more commonly a subset of the total data set, and see how they perform. You can also test the model with cross validation. I have used a popular open source machine learning (ML) software toolkit that is simple to install and use, and it allows you to easily test many types of ML algorithms separately with a number of the model's regression statistics (http://www.cs.waikato.ac.nz/ml/weka/). That might give you a quick analysis without much intervention except the basics of running the program, although depending it may not be worth it if it will be difficult to format your data (although you can usually google "csv to arff convert online" for example for one-step formatting).
