Performing Word Embeddings with domain-specific data

Question

I am new to word-embeddings and have only worked with older approaches like bag of words/tf-idf. Unlike td-idf or bag of words, I have to first train a model to perform the embeddings.

If working with domain-specific documents, can I train word2vec on those documents and then perform word-embedding with the trained model instead of an easily available, pre-trained model? If not, what are suggestions to embed domain-specific data for classification, since similar data may not be readily available?

Answer 1

Yes, you can use your own corpus entirely when trying to create word embeddings. This is the path you to want to pursue if you have any words in your corpus that would be out-of-vocabulary for pretrained embeddings, or if the corpus used to create those pretrained embeddings is ill-suited for your problem at hand.

If you did want to include pretrained embeddings with your additional vocabulary, there are ways around it. One example would be training a model on your limited corpus to get vectors for the words that are out of vocabulary and concatenating them to the pretrained set. I haven't tried and can't speak to how successful/useful that approach is, but it's an option.

I've only used gensim in the past to train word2vec models, so I can't speak to other libraries out there that have implemented some flavor of word embedding. Here's the doc for gensim's word2vec implementation, which generally gives you a lot of levers to pull in terms of how your vectors are generated.

Answer 2

This depends upon the domain that you want to use the word-embeddings for and the size of your training data. For example, for the Biomedical classification task that I had at hand, I tried 3 ways.

1. Using pre-trained Google's word embeddings (Google word2vec)
2. Using pre-trained Biomedical word embeddings from open-access subset from Medline database (PubMed embeddings)
3. Training word embeddings from the "little" corpus I had available with me.

Strangely, the word embeddings trained using the biomedical corpora did not perform any better than the google's general word-embeddings. Additionally, the embeddings I had trained using my own little corpus, performed even worse.

Later, I found this study that experimented with the word embeddings from biomedical corpora and this too found that biomedical word embeddings did not perform any better on the downstream NLP tasks (like classification) compared to general word embeddings.

Shockingly, tf-idf and Bag-of-words models performed better than the word2vec (With a 3-fold cross validation).

tf-idf > Bag-of-words > word2vec