7
$\begingroup$

In document classification, is cosine similarity considered a classification or a clustering technique? But you need training data with the cosine similarity for creation of the centroid right?

$\endgroup$
  • $\begingroup$ In document clustering problems, cosine similarity sometimes gets better results. Check this paper: "concept decompositions for large sparse text data using clustering", Dhillon, Modha, 2000. $\endgroup$ – user22401 Mar 22 '13 at 20:25
24
$\begingroup$

No.

Cosine similarity can be computed amongst arbitrary vectors. It is a similarity measure (which can be converted to a distance measure, and then be used in any distance based classifier, such as nearest neighbor classification.)

$$\cos \varphi = \frac{a\cdot b}{\|a\| \, \|b\|} $$

Where $a$ and $b$ are whatever vectors you want to compare.

If you want to do NN classification, you would use $a$ as your new document, and $b$ as your known sample documents, then classify the new document based on the most similar sample(s).

Alternatively, you could compute a centroid for a whole class, but that would assume that the class is very consistent in itself, and that the centroid is a reasonable estimator for the cosine distances (I'm not sure about this!). NN classification is much easier for you, and less dependent on your corpus to be very consistent in itself.

Say you have the topic "sports". Some documents will talk about Soccer, others about Basketball, others about American Football. The centroid will probably be quite meaningless. Keeping a number of good sample documents for NN classification will likely work much better.

This happens commonly when one class consists of multiple clusters. It's an often misunderstood thing, classes do not necessarily equal clusters. Multiple classes may be one big cluster when they are hard to discern in the data. And on the other hand a class may well have multiple clusters if it is not very uniform.

Clustering can work well for finding good sample documents from your training data, but there are other more appropriate methods. In a supervised context, supervised methods will always perform better than unsupervised.

$\endgroup$
  • $\begingroup$ If the cosine btw two vectors is 0, they are considered to be similar. Isn't cosine similarity a distance measure then? $\endgroup$ – bayerj May 14 '12 at 12:58
  • 3
    $\begingroup$ It doesn't satisfy the triangle equality, so it is not a distance function. It also is not well-defined for the vector 0. $\endgroup$ – Anony-Mousse May 14 '12 at 14:11
  • $\begingroup$ you said No to "But you need training data with the cosine similarity for creation of the centroid right?"?. If so how do you create the centroid? $\endgroup$ – samsamara May 16 '12 at 3:03
  • $\begingroup$ What do you need a centroid for? Cosine similarity is defined on TWO VECTORS, with no centroids involved: en.wikipedia.org/wiki/Cosine_similarity $\endgroup$ – Anony-Mousse May 16 '12 at 9:54
  • 1
    $\begingroup$ You could use a centroid, but if you are doing NN classification, you would just keep a number of relevant sample documents for each topic to compare it with. $\endgroup$ – Anony-Mousse May 17 '12 at 17:45
11
$\begingroup$

I think you have not yet understood the difference between clustering and classification.

Document classification (or supervised learning) requires a set of documents and a class information for each document (example: the topic of the document). The goal of classification is to build a model which predicts the class for documents where the class (in this example the topic) is not known. When models are applied on documents where the class is known, they can be evaluated by comparing the predicted with the true class (hence supervised). The data used for training but not evaluating the model is called training data.

Document clustering (or unsupervised learning) requires a set of documents but not a class information. The goal is to find groups / clusters in the data, so that

  • documents which are similar according to a specified distance function are in one cluster. Example: Documents which roughly contains the same keywords.
  • documents which are not similar according to a specified distance function are in different clusters

The resulting clusters cannot be evaluated like a classification model, because the true clusters are not known (hence unsupervised). Hence there is no such thing as training data, you simply use all data to build the clusters.

See also: Classification vs clustering

Now the connection between both techniques, and imho the source of your confusion: By defining the clusters generated by document clustering as class, one can train a classification model on the data. Example: If you cluster documents by words, you may detect that the resulting clusters are indeed describing topics. Now you can build a classification model for that automatically derived class.

Finally, as put by Anony-Mousse et al., the cosine similarity can be used both for

  • clustering, by defining 1-cosine as distance function (which may not be a metric). Maybe you want use the loosely related Jaccard distance instead
  • classification, by using it in e.g. k-nearest-neighbor
$\endgroup$
6
$\begingroup$

A cosine similarity function returns the cosine between vectors. A cosine is a cosine, and should not depend upon the data. However, how we decide to represent an object, like a document, as a vector may well depend upon the data.

Often, we represent an document as a vector where each dimension corresponds to a word. If the word does not appear we assign a value of 0 to that dimension. If the word does appear the value corresponds to the number of times that word appears in the document normalized by the number of times that word appears in all the documents with our data. This is the general idea behind TF/IDF. Since different sets of documents will have a different distribution of words, the TF/IDF vector representations of documents depend upon the particular document set you are working with.

Many classification and clustering methods depend upon some measure of distance and similarity or distance between objects. If they do, then they can use cosine similarity.

$\endgroup$
  • $\begingroup$ Then why cosine similarity comes as a similarity measure for text document clustering? $\endgroup$ – samsamara May 13 '12 at 15:42
  • $\begingroup$ Because documents are often represented as vectors. Cosine similarity is a measure of whether these vectors are pointing in the same direction. In practice, this often works as a measure of whether documents are about the same thing. $\endgroup$ – fgregg May 13 '12 at 17:41
  • $\begingroup$ "A cosine is a cosine, and should not depend upon the data." !? If the data change, the cosine will change. Maybe you mean basis? $\endgroup$ – jtlz2 Jan 11 '18 at 9:15
4
$\begingroup$

Similarity measures are not machine learning algorithm per se, but they play an integral part. After features are extracted from the raw data, the classes are selected or clusters defined implicitly by the properties of the similarity measure.

It might help to consider the Euclidean distance instead of cosine similarity. Is the Euclidean distance a learning algorithm? No, but you can use it to define one.

$\endgroup$
  • $\begingroup$ I'm doing some tests in document clustering in weka with Euclidean distance as the distance function. But my problem is even if I give a mix of unrelated (topically) and related instances during tests (even some empty instances), they all get clustered into the same group. What could be the reason for this? $\endgroup$ – samsamara May 15 '12 at 15:59
  • $\begingroup$ You are not getting good separation in the feature space? Can you see the clusters with PCA or does it look like a jumble? $\endgroup$ – Emre May 15 '12 at 16:07
  • $\begingroup$ I'm using weka. I checked it but there is no option for that. Do you know of a different tool kit with those options? So if I test the model with same instances from the train set and empty instances shouldn't they get clustered into different groups? $\endgroup$ – samsamara May 15 '12 at 16:26
  • $\begingroup$ WEKA PCA example $\endgroup$ – Emre May 15 '12 at 16:29
  • $\begingroup$ Thanks for the link. I did a try but it was taking so long time when I stopped. I'm wondering if I have some empty instances in my test set shouldn't they get clustered into a different cluster? $\endgroup$ – samsamara May 17 '12 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.