0
$\begingroup$

I am trying to isolate the most similar groups from a set of binary variables while minimizing variation within the clusters.

Is it valid to use Ward's method for clustering and measure similarly by Dice's coefficient for binary data? If anyone happens to have any examples of published data where this method as been used I would be extremely grateful.

$\endgroup$
3
  • $\begingroup$ I made few changes in your question -- feel free to edit them or revert if you feel that they are inadequate. I would encourage you to make your question more precise and more detailed, usually it facilitates higher-quality answers. $\endgroup$
    – Tim
    Jan 15, 2016 at 13:43
  • 1
    $\begingroup$ The answer is no. Ward's demands (squared) euclidean distances. It requires scale variables (although some people would do it with binary data, that usage is much questionned). Dice measure is not distance/dissimilarity at all, it is similarity. Please find/read more about Ward's method on this site. $\endgroup$
    – ttnphns
    Jan 15, 2016 at 13:56
  • $\begingroup$ How many binary variables have you got? For $k$ such variables, their cross-combinations automatically define $2^k$ groups, many of which may not occur. Why not just look at the cross-combinations that occur, e.g. male unemployed non-statisticians? $\endgroup$
    – Nick Cox
    Jan 16, 2016 at 18:38

1 Answer 1

2
$\begingroup$

Ward's method is about minimizing squared deviations, similar to k-means. Squared deviations are related to squared Euclidean distance.

To better understand the relationship, have a look at the underlying math and variance. Variance, by its concept, is meant for continuous variables, not for binary data (where the mean is not a meaningful data point anymore).

This implies it should not be used with other distances, unless you can prove them to be equivalent to squared Euclidean distance in some kernel space, or maybe a Bregman divergence, etc.

For a distance (or similarity) of binary variables such as Dice or Hamming or Jaccard, other linkages such as single-linkage and complete-linkage are more meaningful and interpretable. Precisely, single linkage will still mean that every point in the cluster is connected by steps of at most this distance, and in complete linkage every point in a cluster is connected to every other point in the cluster with at most this distance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.