Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus for me to be able to train my own embedding (for example legal texts, or medicine, etc...).

I would like to use a pretrained embedding like the ones available in BERT (trained on all of Wikipedia) or fastText, but I want to somehow restrict the vocabulary to that in my domain specific corpus.

Is this possible at all? Or does the task amount to retraining on the target corpus anyway - i.e there is no way to do it efficiently?

If it is, how can it be done?

  • $\begingroup$ "Restrict" in what sense? $\endgroup$
    – Tim
    Jun 18, 2019 at 14:57

1 Answer 1


This is called transfer learning. There are many methods for doing this, including adding new layers at the end, deleting or retraining some of your DNN’s final layers, freezing or lowering learning rated on later layers, etc. I’d guess that a high percentage of Bert use involves at least some new tweaking before deployment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.