1
$\begingroup$

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that?

I think to just randomly initialize the vector is not a good idea.

I heard something about calculating the average - but how do I exactly do that? The average of what exactly?

$\endgroup$
3
  • 1
    $\begingroup$ What are you using your embeddings for? $\endgroup$
    – Aaron
    Jul 27, 2018 at 4:54
  • $\begingroup$ I am using the embedding to train a model to categorize issues mails into "Bug" and "not bug / future request". $\endgroup$
    – Dieshe
    Aug 24, 2018 at 9:15
  • 1
    $\begingroup$ I think the best solution to this problem is to use a language model that is able to generate embedding vectors even when it does not know the exact word. This is the case with fasttext and ELMO embeddings / models. See here: allennlp.org/elmo $\endgroup$
    – Dieshe
    Sep 20, 2018 at 15:52

1 Answer 1

1
$\begingroup$

You might consider "a la carte" word embeddings. Essentially, the word vector of any new word can be derived from the average vector of the other words appearing alongside it, as you suggest. However, note that the "derived" word vector is not simply the average, but rather a linear transformation of the average.

That is, if the embedding for the new word is $v_{w}$ and the average is $$ u_{w} = \alpha \sum_{\mathrm{sentences \ s \ containing \ w} \\ \mathrm{words \ w' \ in \ s}} v_{w'}$$ with $\alpha$ the appropriate normalizing constant, then we have $$ v_{w} = A u_{w}$$ where $A$ is a matrix that is determined uniquely by the corpus itself. In particular, $A$ can be determined by linear regression (we have known pairs $v_{w}$, $u_{w}$ for all of the words already in the embedding).

To be clear, I have not used "a la carte" embeddings before, so I don't know how well it works in practice.

Fun fact: the official publication date of the linked paper is roughly one week before this question was posted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.