16
$\begingroup$

I am reading this paper http://zli115.web.engr.illinois.edu/wp-content/uploads/2016/10/0479.pdf

It distinguishes between feature extraction and fine tuning in deep learning. I am not getting the difference as feature extraction is just the same as fine tuning:

As per my understanding:

You train a model on a dataset, use it for training on another dataset. This is fine tuning. This is the same as feature extraction from the first trained model, like in feature extraction also you take the first model and train it on a new dataset.

Is there any difference between the two in the ml literature?

Joint training is a third category I understand as there you train on all data simultaneously.

$\endgroup$
16
$\begingroup$

As shown in figure 2 of {1}, in the fine-tuning strategy all weights are changed when training on the new task (except for the weights of the last layers for the original task), whereas in the feature extraction strategy only the weights of the newly added last layers change during the training phase:

enter image description here


References:

$\endgroup$
3
  • 2
    $\begingroup$ I wonder why they call it feature extraction, better call it feature freezing. thanks for the explanation. $\endgroup$
    – Rafael
    Jan 9 '17 at 21:58
  • $\begingroup$ @Rafael If I'm not mistaken (I'm a beginner on neural networks), I think you may be confusing features with weights, which I believe are not the same thing. Features would be what the model detected as useful stuff from the input you provided it. Weights are what changes or not, which are one of the things used to generate the features (aside from activation functions, for example). Feature Extraction vs Fine-tunning in the middle of this page: learndatasci.com/tutorials/hands-on-transfer-learning-keras - it's helping me a lot to understand these concepts $\endgroup$
    – DADi590
    Jun 7 at 20:59
  • $\begingroup$ [Probably what I wrote is no longer useful for the OP, but for anyone else that might come across - if I'm correct in what I wrote, at least.] $\endgroup$
    – DADi590
    Jun 7 at 21:00
0
$\begingroup$

I agree it's a wording error. Feature extraction in this context should mean participation of outputs from model a) in the inputs to some other model (not necessarily of the same type). For example, we could run textual inputs through transformer model like Bert, and use resulting vector as features along with simple text stats as words count, avg length etc to predict sentiment of the text using, let's say, xgboost model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.