Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus for me to be able to train my own embedding (for example legal texts, or medicine, etc...).
So I would like to use a pertained embedding like the ones available in BERT (Trained on all of Wikipedia) or fastText, but I want to somehow restrict the vocabulary to that in my domain specific corpus.
Is this possible at all? Or does the task amount to retraining on the target corpus anyway - i.e there is no way to do it efficiently?
If it is, how can it be done?