Is it possible to take a pretrained word embedding, trained on a general vocabulary, and make it domain specific?

Question

Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus for me to be able to train my own embedding (for example legal texts, or medicine, etc...).

I would like to use a pretrained embedding like the ones available in BERT (trained on all of Wikipedia) or fastText, but I want to somehow restrict the vocabulary to that in my domain specific corpus.

Is this possible at all? Or does the task amount to retraining on the target corpus anyway - i.e there is no way to do it efficiently?

If it is, how can it be done?

Answer 1

This is called transfer learning. There are many methods for doing this, including adding new layers at the end, deleting or retraining some of your DNN’s final layers, freezing or lowering learning rated on later layers, etc. I’d guess that a high percentage of Bert use involves at least some new tweaking before deployment.