# How do I add a missing word to a pretrained embedding?

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that?

I think to just randomly initialize the vector is not a good idea.

I heard something about calculating the average - but how do I exactly do that? The average of what exactly?

• What are you using your embeddings for? – Aaron Jul 27 '18 at 4:54
• I am using the embedding to train a model to categorize issues mails into "Bug" and "not bug / future request". – Dieshe Aug 24 '18 at 9:15
• I think the best solution to this problem is to use a language model that is able to generate embedding vectors even when it does not know the exact word. This is the case with fasttext and ELMO embeddings / models. See here: allennlp.org/elmo – Dieshe Sep 20 '18 at 15:52

That is, if the embedding for the new word is $$v_{w}$$ and the average is $$u_{w} = \alpha \sum_{\mathrm{sentences \ s \ containing \ w} \\ \mathrm{words \ w' \ in \ s}} v_{w'}$$ with $$\alpha$$ the appropriate normalizing constant, then we have $$v_{w} = A u_{w}$$ where $$A$$ is a matrix that is determined uniquely by the corpus itself. In particular, $$A$$ can be determined by linear regression (we have known pairs $$v_{w}$$, $$u_{w}$$ for all of the words already in the embedding).