Measuring similarity between document and category

Question

Lets say I have a word embedding model, a set of documents and N categories. Lets say the categories are "cars" and "planes". I want to categorize the documents as either being about cars or planes. So, I pick a bag of words from the document from which I remove stop words and the like.

I take the resulting list of words and average out the distance between the word vector for "car" and the words in my list. I do the same for the word "plane". Finally, I categorize the document as belonging to the category which gives me the smallest average distance between the category and the words in the document.

The reason I am asking this is because I see that "cosine similarity" is a standard feature in gensim. Is that a better way to do classification of documents? I always thought like words in a well-trained word embedding model would have shorter euclidean distance than words that are different definition-wise. When is it better to use cosine similarity? Also, are there other methods I should consider?

Answer 1

Simplifying a little bit in order to stay general (the math gets tedious once a lot more documents and topics are involved).

Let's assume for a second there are only pure car and pure plane documents - those that load 100% onto their respective topics. Let's say your average "car" document has a car score of 10, i.e. an average vector length of 10 (because say, "car" happens to occur 3 times in the average document). Think of average documents like "There is one car, there is another car, there is yet another car..."

What happens if you have a document that only has the word "car" in it once? This might happen if a document is much shorter than average, for example a sentence like "This is a car." Just looking at this, you know it is about cars. It should be in the "car" topic.

Euclidian distance is a measure of magnitude.

Euclidian distance between the average car document and the short document is 9 - in one dimension, its simply the arithmetic difference, and we are in one dimension, because we just assumed there are only pure topics so the vector does not have any value in "planes". Even though the short document is about cars, the euclidian distance is quite large, so you might judge them as not being very similar. You might notice in this case, but you will deal with much larger and complex documents, and you cant check everything by hand.

To make things harder, your documents will likely never be 100% plane and 0% car or vice versa. Some might be a mix. There might be one that is absolutely about planes and flying, such as a pilot's diary, but contains the word car 3 times (pilot describing how he gets to the airport, etc). Is this document in the car topic? You would say no, but the math says "a little bit". By euclidian distance, the pilot document might end up being closer to the car topic than the short car document. This might lead you to misclassify it or misinterpret the topics.

This issue gets more and more pronounced with the amount of topics and documents. There will be hundreds of words, dozens of topics and potentially millions of documents, all of different length. If you then want to do any sort of analysis as to how similar some documents are, you quickly run into trouble.

Why use cosine similarity instead?

Cosine similarity measures the difference in vector orientation

Cosine similarity uses the cosine of the angle between two vectors. If there is no difference between vector orientation - assume pure topics again - as it would be between the average car document and the short "this is a car" document, no matter the vector length, then the angle is zero. The cosine of zero is 1 - their cosine similarity is high, therefore being a potentially better measure of document similarity than euclidian distance (and more easily interpreted difference as it is [1,-1] - good luck figuring out what a euclidian distance of 12453523 in a 245962-dimensional vector space means).

If we go back to the pilot document, its vector has a 3 on the "cars" dimension, and a higher score of "planes". Its cosine similarity with the average car document will be lower than 1, since they are somewhat dissimilar. When it comes to measuring similarity, that is exactly the result you want.

You can check out this tutorial on cosine similarity for a deeper dive with python examples. There are other distance, i.e. similarity, metrics - see this notebook by the gensim team.

Answer 2

If you're looking for alternatives for supervised classification, it seems you might find zero-shot learning useful.

Zero-shot learning is a family of methods that uses prior information on labels (like features for categories that are defined with words that you mentioned) to learn mapping from input to labels by jointly learning label and input representations. The Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly might be a good start.

For an example of ZSL method you might find Embarrasingly Simple Zero Shot Learning (yes, it's the actual name). EZSL is a method that learns mapping from features $X$ to labels $S$ by finding an alignment matrix $V$ such that

$\|X^TVS \|_F$ is minimized. Note this is similar, and more general, than the situation where you'd have embeddings for input features and labels from the same space and just use cosine similarity - the matrix $V$ makes possible to compare $x_i$ with $s_j$, and you can define your prediction by $s = \underset{s_j}{argmin} (x_iVs_j) $.