Can I apply word2vec to find document similarity?

Question

I appreciate word2vec is used more to find the semantic similarities between words in a corpus, but here is my idea.

1. Train the word2vec model on a corpus

2. For each document in the corpus, find the Term Frequency (Tf) of each word (the same Tf in TfIDF)

3. Multiply the Tf of each word in a document by its corresponding word vector. Do this for each document

4. Sum the word vectors together element wise to get a single vector and return the $L2$ norm of this vector

Essentially, we are weighing and summing the word vectors for each word in a document.

Each document is now represented by an $N\times 1$ dimensional vector, where $N$ is the number of features chosen in word2vec (the dimensionality hyperparamter - and I have mine set quite low at 150).

Now that all documents create a sort of unit ball in the $N$ dimensional space, we can now find clusters of similar documents / the most similar documents given an input query document, using k-nearest neighbors or k-means.

My question is, is this method viable? I have tried doc2vec, TfIDF, LDA and used appropriate similarity metrics for each (with good results), but my documents are quite short (20-100 tokens) and word2vec has worked very well alone. So I want to know if I can apply the method above or is there anything blatantly wrong with what I am doing here? Any other tips + advice would also be much appreciated.

Answer 1

Some time ago I tried this idea on 20 newsgroups data. I used GloVe embeddings from the authors site (Wikipedia ones).

Aggregating word embeddings using TF-IDF doesn't give good results. It is actually worse than just using TF-IDF features. See results in this notebook (Accuracy on tfidf data vs Accuracy on weighted embedded words).

I also made plots of truncated SVD/PCA of the encoded documents - it seems like aggregated embeddings just make everything close to everything. To illustrate this I tried to find closest words for document encodings in Word embeddings space - it seems like they just lie close to common words (see Closest $10$ words to mean-aggregated texts).

That being said, This notebook is just a toy example and it only suggests that the simplest approach won't work for this data. For instance I didn't try to filter out common words based on some threshold. Also maybe it would make more sense to first extract summaries from the documents first (for example TextRank sort of retrieves most informative paragraphs based partly on TF-IDF score of their words).

If you want to try more elaborate techniques, I think that Gensim covers much of this stuff (for example extractive summarization via TextRank and similar algorithms).

Answer 2

There is nothing wrong with the method, it has been explored in the literature a lot. This is for instance the way that many papers use to evaluate extrinsically word embeddings with tasks like classification. One would expect however to lose in terms of accuracy as the length of the documents increases. Models like doc2vec have been proposed to address such limitations, but it is always better to test them in your benchmark.

Answer 3

Basically the word2vec method intrinsically takes into account the tf (term frequency) of each word. There is no need to emphasis it twice. on the other hand maybe it is a good idea to emphasis on the words with high tf-idf owing the fact that these words are not seen enough in the training phase. I think the way to do that is not simple multiplication however you can feed the network with the context of high tf-idf words more than the other contexts.