2
$\begingroup$

I am new to word-embeddings and have only worked with older approaches like bag of words/tf-idf. Unlike td-idf or bag of words, I have to first train a model to perform the embeddings.

If working with domain-specific documents, can I train word2vec on those documents and then perform word-embedding with the trained model instead of an easily available, pre-trained model? If not, what are suggestions to embed domain-specific data for classification, since similar data may not be readily available?

$\endgroup$

3 Answers 3

2
$\begingroup$

Yes, you can use your own corpus entirely when trying to create word embeddings. This is the path you to want to pursue if you have any words in your corpus that would be out-of-vocabulary for pretrained embeddings, or if the corpus used to create those pretrained embeddings is ill-suited for your problem at hand.

If you did want to include pretrained embeddings with your additional vocabulary, there are ways around it. One example would be training a model on your limited corpus to get vectors for the words that are out of vocabulary and concatenating them to the pretrained set. I haven't tried and can't speak to how successful/useful that approach is, but it's an option.

I've only used gensim in the past to train word2vec models, so I can't speak to other libraries out there that have implemented some flavor of word embedding. Here's the doc for gensim's word2vec implementation, which generally gives you a lot of levers to pull in terms of how your vectors are generated.

$\endgroup$
0
$\begingroup$

This depends upon the domain that you want to use the word-embeddings for and the size of your training data. For example, for the Biomedical classification task that I had at hand, I tried 3 ways.

  1. Using pre-trained Google's word embeddings (Google word2vec)
  2. Using pre-trained Biomedical word embeddings from open-access subset from Medline database (PubMed embeddings)
  3. Training word embeddings from the "little" corpus I had available with me.

Strangely, the word embeddings trained using the biomedical corpora did not perform any better than the google's general word-embeddings. Additionally, the embeddings I had trained using my own little corpus, performed even worse.

Later, I found this study that experimented with the word embeddings from biomedical corpora and this too found that biomedical word embeddings did not perform any better on the downstream NLP tasks (like classification) compared to general word embeddings.

Shockingly, tf-idf and Bag-of-words models performed better than the word2vec (With a 3-fold cross validation).

tf-idf > Bag-of-words > word2vec

$\endgroup$
0
$\begingroup$

The study mentioned by PinkBanter has too many flaws in my opinion.

Comparing Apples to Oranges:

  1. GloVe Wikipedia 100 dim vector, 400K words, GloVe is a completely different method than Word2Vec
  2. Word2Vec Google News 300 dim vector, 3 million words
  3. Domain-Specific Word2Vec 60 dim vector, 2 million words
  4. Domain-Specific Word2Vec 100 dim vector, 100K words

Questionable tests:

  • One of the 3 downstream applications used no machine learning, just some sort of simple SQL-like query
  • None of the machine learning applications used deep learning. Who does significant high-quality NLP anymore without deep learning?
  • Two of the ML-based downstream applications focused on general terms so domain-specific terms had little value

Finally, the paper's conclusion has two seemingly contradictory statements

First, the word embeddings trained on EHR and MedLit can capture the semantics of medical terms better and find semantically relevant medical terms closer to human experts’ judgments than those trained on GloVe and Google News.

Finally, the word embeddings trained on biomedical domain corpora do not necessarily have better performance than those trained on general domain corpora for any downstream biomedical NLP task.

And that last statement extrapolates that the article's simple (and in my opinion greatly flawed tasks) is a valid proxy for ANY biomedical task. ANY??? Because so many people just read the short summary, they will probably get the wrong conclusion than if the authors were more restrained and had written "any downstream biomedical NLP task in this paper", which also would not have been true but at least far less hyperbolic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.