1
$\begingroup$

I have been reading up on some clustering evaluation techniques in this Stanford NLP textbook

On page 359, it defines each of TP, TN, FP & FN. I am having trouble understanding why the definition of TP holds.

It is:

A true positive (TP) decision assigns two similar documents to the same cluster

Initially this made sense to me until I got to the example. (with associated Figure 16.4 on page 357)

Of these, the x pairs in cluster 1, the o pairs in cluster 2, the ⋄ pairs in cluster 3, and the x pair in cluster 3 are true positives.

So, keeping both of these quotes in mind, any cluster with 2 documents from the same True Category are TP. I realise that the TP definition has to change from how it is used with classification, but this seems too lenient of an assumption for a TP, particularly as N gets large. I.e. How can the x values be TP in two different clusters?

Any help would be appreciated,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

It's all about pairs. Clusters themselves are not distinguishable.

So two objects x, y. If they are in the same cluster, they are a pair $(x,y)$. Otherwise they are not.

If this pair exists in both, TP. If it exists in neither, TN. But if one clustering says x and y are dimilar, and the other says they are dissimilar, then that is a disagreement.

it is actually the same as in classification. Except that not objects, bit pairs are used. There are two classes: Class A: object pair (x,y) is similar ("exists"). Class B: object pair (x,y) is dissimilar ("does not exist"). Now you compute the confusion matrix of predicting, whether two objects are similar (A), or not (B).

Improve this answer
$\endgroup$
3
  • $\begingroup$ So to be clear in terms of document clustering, a pair is two documents? If this is the case, then what I don't understand is why a pair of documents counts as TP. For example, if I have 2 categories that I know a-priori (a,b) and 6 documents. I cluster the documents using some algorithm, and it comes back with all 6 documents in one cluster. In this situation I would have 6 TP would I not? This is what seems incorrect to me, because intuitively I would have though half of these would be considered incorrect. Perhaps it is incorrect to be looking at TP independently in this instance. $\endgroup$
    – SamPassmore
    Sep 24, 2015 at 21:29
  • 1
    $\begingroup$ Assuming that 3 are in category 1, 3 are in category 2, you have 3*2 "category pairs" each. Your clustering has 6*5=30 pairs. You have 100% recall (12TP), but 18 FP+FN, a pretty bad precision of 40%. $\endgroup$
    – Has QUIT--Anony-Mousse
    Sep 24, 2015 at 23:05
  • 2
    $\begingroup$ Never only look at TP. Indeed, putting everything into the same cluster means every pair exist, so 100% recall. That's why you do not use recall with clustering. Use ARI. $\endgroup$
    – Has QUIT--Anony-Mousse
    Sep 24, 2015 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.