37
$\begingroup$

How do I use a word embedding to map a document to a feature vector, suitable for use with supervised learning?

A word embedding maps each word $w$ to a vector $v \in \mathbb{R}^d$, where $d$ is some not-too-large number (e.g., 500). Popular word embeddings include word2vec and Glove.

I want to apply supervised learning to classify documents. I'm currently mapping each document to a feature vector using the bag-of-words representation, then applying an off-the-shelf classifier. I'd like replace the bag-of-words feature vector with something based on an existing pre-trained word embedding, to take advantage of the semantic knowledge that's contained in the word embedding. Is there a standard way to do that?

I can imagine some possibilities, but I don't know if there's something that makes the most sense. Candidate approaches I've considered:

  • I could compute the vector for each word in the document, and average all of them. However, this seems like it might lose a lot of information. For instance, with the bag-of-words representation, if there are a few words that are highly relevant to classification task and most words are irrelevant, the classifier can easily learn that; if I average the vectors for all the words in the document, the classifier has no chance.

  • Concatenating the vectors for all the words doesn't work, because it doesn't lead to a fixed-size feature vector. Also it seems like a bad idea because it will be overly sensitive to the specific placement of a word.

  • I could use the word embedding to cluster the vocabulary of all words into a fixed set of clusters, say, 1000 clusters, where I use cosine similarity on the vectors as a measure of word similarity. Then, instead of a bag-of-words, I could have a bag-of-clusters: the feature vector I supply to the classifer could be a 1000-vector, where the $i$th component counts the number of words in the document that are part of cluster $i$.

  • Given a word $w$, these word embeddings let me compute a set of the top 20 most similar words $w_1,\dots,w_{20}$ and their similarity score $s_1,\dots,s_{20}$. I could adapt the bag-of-words-like feature vector using this. When I see the word $w$, in addition to incrementing the element corresponding to word $w$ by $1$, I could also increment the element corresponding to word $w_1$ by $s_1$, increment the element corresponding to word $w_2$ by $s_2$, and so on.

Is there any specific approach that is likely to work well for document classification?


I'm not looking for paragraph2vec or doc2vec; those require training on a large data corpus, and I don't have a large data corpus. Instead, I want to use an existing word embedding.

$\endgroup$
23
$\begingroup$

One simple technique that seems to work reasonably well for short texts (e.g., a sentence or a tweet) is to compute the vector for each word in the document, and then aggregate them using the coordinate-wise mean, min, or max.

Based on results in one recent paper, it seems that using the min and the max works reasonably well. It's not optimal, but it's simple and about as good or better as other simple techniques. In particular, if the vectors for the $n$ words in the document are $v^1,v^2,\dots,v^n \in \mathbb{R}^d$, then you compute $\min(v^1,\dots,v^n)$ and $\max(v^1,\dots,v^n)$. Here we're taking the coordinate-wise minimum, i.e., the minimum is a vector $u$ such that $u_i = \min(v^1_i, \dots, v^n_i)$, and similarly for the max. The feature vector is the concatenation of these two vectors, so we obtain a feature vector in $\mathbb{R}^{2d}$. I don't know if this is better or worse than a bag-of-words representation, but for short documents I suspect it might perform better than bag-of-words, and it allows using pre-trained word embeddings.

TL;DR: Surprisingly, the concatenation of the min and max works reasonably well.

Reference:

Representation learning for very short texts using weighted word embedding aggregation. Cedric De Boom, Steven Van Canneyt, Thomas Demeester, Bart Dhoedt. Pattern Recognition Letters; arxiv:1607.00570. abstract, pdf. See especially Tables 1 and 2.

Credits: Thanks to @user115202 for bringing this paper to my attention.

$\endgroup$
  • 3
    $\begingroup$ for short text, avg/min/max might work well, but what if long text, such as news article? $\endgroup$ – avocado Apr 25 '17 at 5:06
  • $\begingroup$ For anyone who reads through that paper and gets as confused as me: the paper doesn't focus on the approach mentioned by @D.W., they only mention it briefly under "5.1. Baselines" as a baseline approach. The body of the paper focuses on their own technique, which involves training a classifier using embeddings, which is much more complex than the approach outlined here! $\endgroup$ – Miguel Aug 31 at 6:25
16
$\begingroup$

You can use doc2vec similar to word2vec and use a pre-trained model from a large corpus. Then use something like .infer_vector() in gensim to construct a document vector. The doc2vec training doesn't necessary need to come from the training set.

Another method is to use an RNN, CNN or feed forward network to classify. This effectively combines the word vectors into a document vector.

You can also combine sparse features (words) with dense (word vector) features to complement each other. So your feature matrix would be a concatenation of the sparse bag of words matrix with the average of word vectors. https://research.googleblog.com/2016/06/wide-deep-learning-better-together-with.html

Another interesting method is to use a similar algorithm to word2vec but instead of predicting a target word, you can predict a target label. This directly tunes the word vectors to the classification task. http://arxiv.org/pdf/1607.01759v2.pdf

For more ad hoc methods, you might try weighing the words differently depending on syntax. For example, you can weigh verbs more strongly than determiners.

$\endgroup$
6
$\begingroup$

If you are working with English text and want pre-trained word embeddings to begin with, then please see this: https://code.google.com/archive/p/word2vec/

This is the original C version of word2vec. Along with this release, they also released a model trained on 100 billion words taken from Google News articles (see subsection titled: "Pre-trained word and phrase vectors").

In my opinion and experience of working on word embeddings, for document classification, a model like doc2vec (with CBOW) works much better than bag of words.

Since, you have a small corpus, I suggest, you initialize your word embedding matrix by the pre-trained embeddings mentioned above. Then train for the paragraph vector in the doc2vec code. If you are comfortable with python, you can checkout the gensim version of it, which is very easy to modify.

Also check this paper that details the inner workings of word2vec/doc2vec: http://arxiv.org/abs/1411.2738. This will make understanding the gensim code very easy.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the suggestions. I'm not asking for a word embedding; I already know how to get a pre-trained word embedding (I mentioned word2vec in my question). My question is how to construct feature vectors from a pre-trained word embedding. I appreciate the reference to doc2vec, but my corpus is quite small and so I suspect/fear that trying to train doc2vec codes will overfit and perform poorly (even if I initialize the matrix with pre-trained embeddings). $\endgroup$ – D.W. Jul 2 '16 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.