Is cosine similarity a classification or a clustering technique?

Question

In document classification, is cosine similarity considered a classification or a clustering technique? But you need training data with the cosine similarity for creation of the centroid right?

Answer 1

No.

Cosine similarity can be computed amongst arbitrary vectors. It is a similarity measure (which can be converted to a distance measure, and then be used in any distance based classifier, such as nearest neighbor classification.)

$$\cos \varphi = \frac{a\cdot b}{||a|| ||b||} $$

Where $a$ and $b$ are whatever vectors you want to compare.

If you want to do NN classification, you would use $a$ as your new document, and $b$ as your known sample documents, then classify the new document based on the most similar sample(s).

Alternatively, you could compute a centroid for a whole class, but that would assume that the class is very consistent in itself, and that the centroid is a reasonable estimator for the cosine distances (I'm not sure about this!). NN classification is much easier for you, and less dependent on your corpus to be very consistent in itself.

Say you have the topic "sports". Some documents will talk about Soccer, others about Basketball, others about American Football. The centroid will probably be quite meaningless. Keeping a number of good sample documents for NN classification will likely work much better.

This happens commonly when one class consists of multiple clusters. It's an often misunderstood thing, classes do not necessarily equal clusters. Multiple classes may be one big cluster when they are hard to discern in the data. And on the other hand a class may well have multiple clusters if it is not very uniform.

Clustering can work well for finding good sample documents from your training data, but there are other more appropriate methods. In a supervised context, supervised methods will always perform better than unsupervised.

Answer 2

I think you have not yet understood the difference between clustering and classification.

Document classification (or supervised learning) requires a set of documents and a class information for each document (example: the topic of the document). The goal of classification is to build a model which predicts the class for documents where the class (in this example the topic) is not known. When models are applied on documents where the class is known, they can be evaluated by comparing the predicted with the true class (hence supervised). The data used for training but not evaluating the model is called training data.

Document clustering (or unsupervised learning) requires a set of documents but not a class information. The goal is to find groups / clusters in the data, so that

• documents which are similar according to a specified distance function are in one cluster. Example: Documents which roughly contains the same keywords.
• documents which are not similar according to a specified distance function are in different clusters

The resulting clusters cannot be evaluated like a classification model, because the true clusters are not known (hence unsupervised). Hence there is no such thing as training data, you simply use all data to build the clusters.

See also: Classification vs clustering

Now the connection between both techniques, and imho the source of your confusion: By defining the clusters generated by document clustering as class, one can train a classification model on the data. Example: If you cluster documents by words, you may detect that the resulting clusters are indeed describing topics. Now you can build a classification model for that automatically derived class.

Finally, as put by Anony-Mousse et al., the cosine similarity can be used both for

• clustering, by defining 1-cosine as distance function (which may not be a metric). Maybe you want use the loosely related Jaccard distance instead
• classification, by using it in e.g. k-nearest-neighbor

Answer 3

A cosine similarity function returns the cosine between vectors. A cosine is a cosine, and should not depend upon the data. However, how we decide to represent an object, like a document, as a vector may well depend upon the data.

Often, we represent an document as a vector where each dimension corresponds to a word. If the word does not appear we assign a value of 0 to that dimension. If the word does appear the value corresponds to the number of times that word appears in the document normalized by the number of times that word appears in all the documents with our data. This is the general idea behind TF/IDF. Since different sets of documents will have a different distribution of words, the TF/IDF vector representations of documents depend upon the particular document set you are working with.

Many classification and clustering methods depend upon some measure of distance and similarity or distance between objects. If they do, then they can use cosine similarity.

Answer 4

Similarity measures are not machine learning algorithm per se, but they play an integral part. After features are extracted from the raw data, the classes are selected or clusters defined implicitly by the properties of the similarity measure.

It might help to consider the Euclidean distance instead of cosine similarity. Is the Euclidean distance a learning algorithm? No, but you can use it to define one.

Answer 5

Yes. In document clustering problem, cosine similarity sometimes get better result. Check this paper: "concept decompositions for large sparse text data using clustering", Dhillon, Modha, 2000.