How do I add a missing word to a pretrained embedding?

Question

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that?

I think to just randomly initialize the vector is not a good idea.

I heard something about calculating the average - but how do I exactly do that? The average of what exactly?

Answer 1

You might consider "a la carte" word embeddings. Essentially, the word vector of any new word can be derived from the average vector of the other words appearing alongside it, as you suggest. However, note that the "derived" word vector is not simply the average, but rather a linear transformation of the average.

That is, if the embedding for the new word is $v_{w}$ and the average is $$ u_{w} = \alpha \sum_{\mathrm{sentences \ s \ containing \ w} \\ \mathrm{words \ w' \ in \ s}} v_{w'}$$ with $\alpha$ the appropriate normalizing constant, then we have $$ v_{w} = A u_{w}$$ where $A$ is a matrix that is determined uniquely by the corpus itself. In particular, $A$ can be determined by linear regression (we have known pairs $v_{w}$, $u_{w}$ for all of the words already in the embedding).

To be clear, I have not used "a la carte" embeddings before, so I don't know how well it works in practice.

Fun fact: the official publication date of the linked paper is roughly one week before this question was posted.