Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am doing a cluster analyis and I was wondering whether it is possible to remove duplicates from the data set - in order to increase performance.
I work on tables where objects are in rows and variables are in columns.
If two rows are identical, is it possible to delete them? Would this change the result?
I am working with both hierarchical clustering and k-means / k-modes
It changes the results. With k-means this should be straightforward to see: the mean of 0, 0 and 1 is different from 0 and 1. Usually this will also be the case for hierarchical clustering, but it depends on the linkage criteria, For example, complete linkage shouldn't be affected.
Speaking generally, I would argue for leaving it in. Having duplicates indicate that those are particularly likely combinations of variable values, which should get a higher weight because of that. This means observations with the same values do not become redundant.
Do you really have performance problems with these two algorithms?
If you remove duplicates, you need to add weights to your data instead, otherwise the result may change (except for single-linkage clustering, I guess).
If your data set has few duplicates, this will likely cost you some runtime.
If your data set has lots of duplicates, it can accelerate the processing a lot to merge them and use weights instead. If you have on average 10 duplicates of each object, and an algorithm with quadratic runtime, the speedup can be 100 fold. That is substantial, and well worth the effort to merge duplicates.
