Apply word embeddings to entire document, to get a feature vector

Question

How do I use a word embedding to map a document to a feature vector, suitable for use with supervised learning?

A word embedding maps each word $w$ to a vector $v \in \mathbb{R}^d$, where $d$ is some not-too-large number (e.g., 500). Popular word embeddings include word2vec and Glove.

I want to apply supervised learning to classify documents. I'm currently mapping each document to a feature vector using the bag-of-words representation, then applying an off-the-shelf classifier. I'd like replace the bag-of-words feature vector with something based on an existing pre-trained word embedding, to take advantage of the semantic knowledge that's contained in the word embedding. Is there a standard way to do that?

I can imagine some possibilities, but I don't know if there's something that makes the most sense. Candidate approaches I've considered:

• I could compute the vector for each word in the document, and average all of them. However, this seems like it might lose a lot of information. For instance, with the bag-of-words representation, if there are a few words that are highly relevant to classification task and most words are irrelevant, the classifier can easily learn that; if I average the vectors for all the words in the document, the classifier has no chance.

• Concatenating the vectors for all the words doesn't work, because it doesn't lead to a fixed-size feature vector. Also it seems like a bad idea because it will be overly sensitive to the specific placement of a word.

• I could use the word embedding to cluster the vocabulary of all words into a fixed set of clusters, say, 1000 clusters, where I use cosine similarity on the vectors as a measure of word similarity. Then, instead of a bag-of-words, I could have a bag-of-clusters: the feature vector I supply to the classifer could be a 1000-vector, where the $i$th component counts the number of words in the document that are part of cluster $i$.

• Given a word $w$, these word embeddings let me compute a set of the top 20 most similar words $w_1,\dots,w_{20}$ and their similarity score $s_1,\dots,s_{20}$. I could adapt the bag-of-words-like feature vector using this. When I see the word $w$, in addition to incrementing the element corresponding to word $w$ by $1$, I could also increment the element corresponding to word $w_1$ by $s_1$, increment the element corresponding to word $w_2$ by $s_2$, and so on.

Is there any specific approach that is likely to work well for document classification?

---

I'm not looking for paragraph2vec or doc2vec; those require training on a large data corpus, and I don't have a large data corpus. Instead, I want to use an existing word embedding.

Answer 1

One simple technique that seems to work reasonably well for short texts (e.g., a sentence or a tweet) is to compute the vector for each word in the document, and then aggregate them using the coordinate-wise mean, min, or max.

Based on results in one recent paper, it seems that using the min and the max works reasonably well. It's not optimal, but it's simple and about as good or better as other simple techniques. In particular, if the vectors for the $n$ words in the document are $v^1,v^2,\dots,v^n \in \mathbb{R}^d$, then you compute $\min(v^1,\dots,v^n)$ and $\max(v^1,\dots,v^n)$. Here we're taking the coordinate-wise minimum, i.e., the minimum is a vector $u$ such that $u_i = \min(v^1_i, \dots, v^n_i)$, and similarly for the max. The feature vector is the concatenation of these two vectors, so we obtain a feature vector in $\mathbb{R}^{2d}$. I don't know if this is better or worse than a bag-of-words representation, but for short documents I suspect it might perform better than bag-of-words, and it allows using pre-trained word embeddings.

TL;DR: Surprisingly, the concatenation of the min and max works reasonably well.

Reference:

Representation learning for very short texts using weighted word embedding aggregation. Cedric De Boom, Steven Van Canneyt, Thomas Demeester, Bart Dhoedt. Pattern Recognition Letters; arxiv:1607.00570. abstract, pdf. See especially Tables 1 and 2.

Credits: Thanks to @user115202 for bringing this paper to my attention.

Answer 2

You can use doc2vec similar to word2vec and use a pre-trained model from a large corpus. Then use something like .infer_vector() in gensim to construct a document vector. The doc2vec training doesn't necessary need to come from the training set.

Another method is to use an RNN, CNN or feed forward network to classify. This effectively combines the word vectors into a document vector.

You can also combine sparse features (words) with dense (word vector) features to complement each other. So your feature matrix would be a concatenation of the sparse bag of words matrix with the average of word vectors. https://research.googleblog.com/2016/06/wide-deep-learning-better-together-with.html

Another interesting method is to use a similar algorithm to word2vec but instead of predicting a target word, you can predict a target label. This directly tunes the word vectors to the classification task. http://arxiv.org/pdf/1607.01759v2.pdf

For more ad hoc methods, you might try weighing the words differently depending on syntax. For example, you can weigh verbs more strongly than determiners.

Answer 3

If you are working with English text and want pre-trained word embeddings to begin with, then please see this: https://code.google.com/archive/p/word2vec/

This is the original C version of word2vec. Along with this release, they also released a model trained on 100 billion words taken from Google News articles (see subsection titled: "Pre-trained word and phrase vectors").

In my opinion and experience of working on word embeddings, for document classification, a model like doc2vec (with CBOW) works much better than bag of words.

Since, you have a small corpus, I suggest, you initialize your word embedding matrix by the pre-trained embeddings mentioned above. Then train for the paragraph vector in the doc2vec code. If you are comfortable with python, you can checkout the gensim version of it, which is very easy to modify.

Also check this paper that details the inner workings of word2vec/doc2vec: http://arxiv.org/abs/1411.2738. This will make understanding the gensim code very easy.

Answer 4

I'm impressed no one mentioned it, but other best practices are to pad the sentences into a fixed size, initialize an embedding layer with the weights of Word2Vec and feed it into an LSTM. So it is basically what OP mentioned here, but including padding for handling the different lengths:

Concatenating the vectors for all the words doesn't work, because it doesn't lead to a fixed-size feature vector.

Example

Consider the following sentence (taken from the Toxic Comment Classification Challenge):

"Explanation Why the edits made under my username Hardcore Metallica Fan were reverted? They weren't vandalisms, just closure on some GAs after I voted at New York Dolls FAC. And please don't remove the template from the talk page since I'm retired now.89.205.38.27"

First, we clean such sentence:

"explanation why the edits made under my username hardcore metallica fan were reverted ? they weren ' t vandalisms , just closure on some gas after i voted at new york dolls fac . and please don ' t remove the template from the talk page since i ' m retired now . ipaddress"

Next, we encode their words into integers:

776 92 2 161 153 212 44 754 4597 9964 1290 104 399 34 57 2292 10 29 14515 3 66 6964 22 75 2730 173 5 2952 47 136 1298 16686 2615 1 8 67 73 10 29 290 2 398 45 2 60 43 164 5 10 81 4030 107 1 216

And finally, if we perform padding with a length of 200, it would look like this:

array([    0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,     0,     0,     0,     0,     0,     0,     0,
           0,     0,   776,    92,     2,   161,   153,   212,    44,
         754,  4597,  9964,  1290,   104,   399,    34,    57,  2292,
          10,    29, 14515,     3,    66,  6964,    22,    75,  2730,
         173,     5,  2952,    47,   136,  1298, 16686,  2615,     1,
           8,    67,    73,    10,    29,   290,     2,   398,    45,
           2,    60,    43,   164,     5,    10,    81,  4030,   107,
           1,   216], dtype=int32)

We can force all sentences to have a maximum of 200 words, fill with zeros if they have less, or cut words that come later if they have more.

Next, we initialize an embedding model with the weights of word2vec, here's an example using Keras:

model.add(Embedding(nb_words, WV_DIM, weights=[wv_matrix], input_length=MAX_SEQUENCE_LENGTH, trainable=False))

wv_matrix contains a matrix with shape $ℝ^{nd}$ (number of unique words versus embedding dimension).

And finally we add an LSTM layer after that, for example:

embedded_sequences = SpatialDropout1D(0.2)(embedded_sequences)
x = Bidirectional(CuDNNLSTM(64, return_sequences=False))(embedded_sequences)

References

• The full implementation of the above example if implemented in this Kaggle's Kernel. The code comes from this post.

Answer 5

I would suggest to use window-size approach. Given window-size=1024 (token) and you pre-define says 10 windows, then concatenating all vectors of the windows. This is similar to your solution 2, but rather than using word vectors, using window vectors. With this approach, you can try with other embedding such as BERT or similar as these have limited size of token length.

If using Word2Vec, or word vector, would you consider to use a linear combination with the word weighting such as TFIDF and the word vectors. I found it's outperformed compared with word vectors without weightings.