7
$\begingroup$

I am doing a cluster analyis and I was wondering whether it is possible to remove duplicates from the data set - in order to increase performance.

I work on tables where objects are in rows and variables are in columns.

If two rows are identical, is it possible to delete them? Would this change the result?

I am working with both hierarchical clustering and k-means / k-modes

$\endgroup$
  • 2
    $\begingroup$ Duplicates are in essense cases (objects) with integer frequency weight. So, your question is about the effect of weighting. It definitely affects K-means results. As for hierarchical clustering - whether weighting will play a role or not depends on the agglomerative method. See this answer for more. $\endgroup$ – ttnphns May 18 '15 at 9:58
14
$\begingroup$

It changes the results. With k-means this should be straightforward to see: the mean of 0, 0 and 1 is different from 0 and 1. Usually this will also be the case for hierarchical clustering, but it depends on the linkage criteria, For example, complete linkage shouldn't be affected.

Speaking generally, I would argue for leaving it in. Having duplicates indicate that those are particularly likely combinations of variable values, which should get a higher weight because of that. This means observations with the same values do not become redundant.

Do you really have performance problems with these two algorithms?

| cite | improve this answer | |
$\endgroup$
8
$\begingroup$

If you remove duplicates, you need to add weights to your data instead, otherwise the result may change (except for single-linkage clustering, I guess).

If your data set has few duplicates, this will likely cost you some runtime.

If your data set has lots of duplicates, it can accelerate the processing a lot to merge them and use weights instead. If you have on average 10 duplicates of each object, and an algorithm with quadratic runtime, the speedup can be 100 fold. That is substantial, and well worth the effort to merge duplicates.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.