Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus for me to be able to train my own embedding (for example legal texts, or medicine, etc...).
I would like to use a pretrained embedding like the ones available in BERT (trained on all of Wikipedia) or fastText, but I want to somehow restrict the vocabulary to that in my domain specific corpus.
Is this possible at all? Or does the task amount to retraining on the target corpus anyway - i.e there is no way to do it efficiently?
If it is, how can it be done?