I try to compare different binary objects, lets say:
data <- read.table(header = TRUE, row.names = 1, text = "User va1 var2 var3 var4 abc1 0 0 1 1 abc2 1 1 0 0 abc3 0 0 1 0")
I want to have similar objects in one cluster so I first use the jaccard coefficient to compute similarity:
distance <- dist(data, method = "binary")
Which results in :
abc1 abc2 abc3 abc1 0.0 abc2 1.0 0 abc3 0.5 1 0.0
Now, what I do not understand here in the first place is: abc1 and abc2 are in my opinion most dissimilar, because they do not have any match. Anyhow, the jaccard coefficient puts them as most similar?.
Now I want to cluster theses guys using hierarchical clustering:
hc <- hclust(distance, method = "ward.D") plot(hc)
The result again confused me, as abc1 and abc3 are first clustered together before building a bigger cluster with abc2.
So my questions are:
1. Why is jaccard index telling me that abc1 and abc2 are very equal, when they are the most unequal in the dataset?
2. Why is the clustering (based on the distances computed by the jaccard coefficient) then clustering first abc1 and abc3 together (which is right in my opinion) and then merges them with abc2?
3. Do I just need another coefficient?