I try to compare different binary objects, lets say:

data <- read.table(header = TRUE, row.names = 1, text = 
"User   va1   var2   var3  var4
abc1     0     0     1     1
abc2     1     1     0     0
abc3     0     0     1     0")

I want to have similar objects in one cluster so I first use the jaccard coefficient to compute similarity:

distance <- dist(data, method = "binary")

Which results in :

     abc1 abc2 abc3
abc1  0.0      
abc2  1.0   0   
abc3  0.5   1   0.0

Now, what I do not understand here in the first place is: abc1 and abc2 are in my opinion most dissimilar, because they do not have any match. Anyhow, the jaccard coefficient puts them as most similar?.

Now I want to cluster theses guys using hierarchical clustering:

hc <- hclust(distance, method = "ward.D")

The result again confused me, as abc1 and abc3 are first clustered together before building a bigger cluster with abc2.

So my questions are:
1. Why is jaccard index telling me that abc1 and abc2 are very equal, when they are the most unequal in the dataset?
2. Why is the clustering (based on the distances computed by the jaccard coefficient) then clustering first abc1 and abc3 together (which is right in my opinion) and then merges them with abc2?
3. Do I just need another coefficient?

  • $\begingroup$ ask only one question per thread $\endgroup$ – Antoine Nov 25 '16 at 10:02
  • 3
    $\begingroup$ Your matrix of Jaccard coefficients is distance matrix, 1-jaccard. Jaccard is similarity. $\endgroup$ – ttnphns Nov 25 '16 at 10:03
  • $\begingroup$ Ok, so when I put it right then, abc1 and abc2 are most dissimilar in the distance matrix, which then results in the right clustering $\endgroup$ – hoisiter Nov 25 '16 at 10:15
  • $\begingroup$ ward is only to be used with squared Euclidean distance. jaccard distance and jaccard similarity both exist. Dist=1-sim is their relationship. $\endgroup$ – Anony-Mousse Nov 27 '16 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.