I have a pretrained word embedding and want to add missing words to it. How exactly should I do that?

I think to just randomly initialize the vector is not a good idea.

I heard something about calculating the average - but how do I exactly do that? The average of what exactly?

  • 1
    $\begingroup$ What are you using your embeddings for? $\endgroup$ – Aaron Jul 27 '18 at 4:54
  • $\begingroup$ I am using the embedding to train a model to categorize issues mails into "Bug" and "not bug / future request". $\endgroup$ – Dieshe Aug 24 '18 at 9:15
  • 1
    $\begingroup$ I think the best solution to this problem is to use a language model that is able to generate embedding vectors even when it does not know the exact word. This is the case with fasttext and ELMO embeddings / models. See here: allennlp.org/elmo $\endgroup$ – Dieshe Sep 20 '18 at 15:52

You might consider "a la carte" word embeddings. Essentially, the word vector of any new word can be derived from the average vector of the other words appearing alongside it, as you suggest. However, note that the "derived" word vector is not simply the average, but rather a linear transformation of the average.

That is, if the embedding for the new word is $v_{w}$ and the average is $$ u_{w} = \alpha \sum_{\mathrm{sentences \ s \ containing \ w} \\ \mathrm{words \ w' \ in \ s}} v_{w'}$$ with $\alpha$ the appropriate normalizing constant, then we have $$ v_{w} = A u_{w}$$ where $A$ is a matrix that is determined uniquely by the corpus itself. In particular, $A$ can be determined by linear regression (we have known pairs $v_{w}$, $u_{w}$ for all of the words already in the embedding).

To be clear, I have not used "a la carte" embeddings before, so I don't know how well it works in practice.

Fun fact: the official publication date of the linked paper is roughly one week before this question was posted.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.