1
$\begingroup$

I am reading a paper which uses a Restricted Boltzmann Machine to extract features from a dataset in an unsupervised way and then use those features to train a classifier (they use SVM but it could be every other). I am a little bit confused about what they call feature extraction and fine-tuning. I don't understand whether there is a difference in the two approaches or if they could be mixed together to reproduce their experiment.

To me, doing feature extraction is training the RBM on a dataset, find relevant hidden features and then use them as input of a classifier, which has no weights (parameters) in common with the RBM. On the other hand, fine tuning is to train the RBM on a dataset and then initialize the weights of a classifier with the same structure (imagine a feedforward NN) with those coming from the trained RBM.

Is there a cross line between those two definition? What I think they do in the paper is feature extraction in the sense I explained above, but they mention also fine-tuning with backpropagation and I would those concepts to be clarified.

|cite|improve this question
$\endgroup$
0
$\begingroup$

Here is a summary of what I get from the article to help clarify your ideas. The authors propose 2 steps :

1) Learning the relevant features from the data with a Deep Belief Network (DBN) which is a network made of stacked RBMs. This network is trained first unsupervisedly (pretraining to get good initial weights and biases). Then when the authors talk about backpropagation they are fine-tuning their network by turning the DBN into a feedforward NN.

In fact, I think they follow strictly the method they reference : Hinton & Salakhutdinov's Science paper. You may read it, it describes the unsupervised pretraining / supervised fine-tuning procedure, which has since become quite commonly used in machine learning.

2) Learning a mapping from the extracted features to the stock price. They set up another supervised classifier, a SVM, but this could again be a feedforward NN as you mention.

In a few lines of the article a few different concepts are used and mixed (discriminative/generative models, directed/undirected graphical models, unsupervised/supervised learning...) so things can indeed get messy!

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I understand well both points you mentioned, but I don't realize how they can be mixed together. Point 1 describes the process which involves pre-training+fine-tuning, while point 2 involves to learn some features and then use those features to train a model with a different structure. To me, it looks like with pretraining you care about weights for better initialization, while with feature extraction you care about discovered hidden features. Looking at the dataflow picture they seem to do feature extraction, because they use hidden features together with prices as input of a new classifier. $\endgroup$ – Alexbrini May 20 at 15:59
  • $\begingroup$ Ok, the article is a bit unclear about what is the input of what. As I get it : in point 1) you actually extract the features into more "abstract" or "pertinent" ones that the DBN automatically discover (this is described in the first part of their Section 3). In point 2, the SVM inputs are those extracted features (this is what appears in their scheme and in the last bullet point of the Results section). $\endgroup$ – TheCG May 20 at 16:33
  • $\begingroup$ This point is what I understand from the paper and I think it can be referred to as feature extraction. By the way, the paper of Hinton suggests the approach of fine tuning after pretraining, which I don't think is covered in the paper I am analyzing. It is cited but not covered. This was the fact that I wanted to underline with the previous answer. $\endgroup$ – Alexbrini May 21 at 9:43
  • $\begingroup$ "backpropagation can be utilized to fine-tune theweights and biases to improve the extracting performance [7]" yeah, it is indeed unclear whether they did it or not.. $\endgroup$ – TheCG May 21 at 12:20
  • $\begingroup$ Yes, "it can be". Anyway, I was using this paper just as an example to understand the conceptual difference in doing pretraining and feature extraction, but as far as I understood there's no fixed line between the two approaches and they can be easily mixed. $\endgroup$ – Alexbrini May 21 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.