I understand that there are two ways of leveraging BERT for some NLP classification task:

  1. BERT might perform ‘feature extraction’ and its output is input further to another (classification) model
  2. The other way is fine-tuning BERT on some text classification task by adding an output layer or layers to pretrained BERT and retraining the whole (with varying number of BERT layers fixed)

However, if in the second case, we fix all the layers and add ALL the layers from the classification model will be added, 1st and 2nd approaches are effectively the same, am I right?

  • $\begingroup$ I was asking myself the same question. Did you get some insight on this point? $\endgroup$
    – Tau
    Jun 25, 2020 at 20:44

1 Answer 1


Indeed. You might want to read this paper comparing the 2 approaches on various NLP tasks: To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

The main conclusion is that results are quite similar in most cases.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.