Fine Tuning vs Joint Training vs Feature Extraction

Question

I am reading this paper http://zli115.web.engr.illinois.edu/wp-content/uploads/2016/10/0479.pdf

It distinguishes between feature extraction and fine tuning in deep learning. I am not getting the difference as feature extraction is just the same as fine tuning:

As per my understanding:

You train a model on a dataset, use it for training on another dataset. This is fine tuning. This is the same as feature extraction from the first trained model, like in feature extraction also you take the first model and train it on a new dataset.

Is there any difference between the two in the ml literature?

Joint training is a third category I understand as there you train on all data simultaneously.

Answer 1

As shown in figure 2 of {1}, in the fine-tuning strategy all weights are changed when training on the new task (except for the weights of the last layers for the original task), whereas in the feature extraction strategy only the weights of the newly added last layers change during the training phase:

---

References:

• {1} Li, Zhizhong, and Derek Hoiem. "Learning without forgetting." In European Conference on Computer Vision, pp. 614-629. Springer International Publishing, 2016. https://arxiv.org/abs/1606.09282 ; https://doi.org/10.1007/978-3-319-46493-0_37

Answer 2

I agree it's a wording error. Feature extraction in this context should mean participation of outputs from model a) in the inputs to some other model (not necessarily of the same type). For example, we could run textual inputs through transformer model like Bert, and use resulting vector as features along with simple text stats as words count, avg length etc to predict sentiment of the text using, let's say, xgboost model.