What is the motivation to train one's own word embedding model?

Question

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I found out that it is possible to train your own model.

• Why would you do that ? does it make it more domain-specific ?
• How much data examples would I need to feed the model to achieve a decent result, knowing that my documents are very short (3-4 words max) ?

• Is it possible to get an estimation of the training time ? say for a million data examples of 3 words each.

• Bonus : is continuing the training of a pre-trained model a good strategy ? will it be able to capture the "essence" of the new data even if there are way less data examples than during the first, original training ?

Thank you so much !

Answer 1

For a partial answer, mostly about why using "pre-trained" embeddings. In Word2Vec for instance, each word gets its embedding by looking at the context in which the words occur (i.e, by using the neighboring words). However, it has two major problems: 1) you can not achieve word disambiguation -the word jaguar will have the same embedding whether we are talking about the animal or the car (clearly that's wrong) and 2) which is linked to 1), your embeddings will not change depending on where they appear in the sentence at task time. Thus by using something like BERT, each word embedding is unique depending on the particular context at hand during the task to perform, rather than being a fixed representation such as Word2vec.