1
$\begingroup$

I have a dataset which has ~200k rows and looks like the following -

id - ids which are unique col1 - categorical with 3 levels col2 - categorical with 5 levels col3 - numerical col4 - categorical with 5 levels

However, the number of unique rows in this dataset once the id column is removed is only ~1000 rows.

I am trying to use PARTITIONING AROUND MEDOIDS (PAM) clustering in R which requires Gower distance matrix to be calculated. However, calculating this matrix is intensive and the final result will not fit in memory.

I looked at adding weights to each of these duplicated rows when using Gower distance but it looks like from this question - https://stackoverflow.com/questions/21334677/how-do-i-weight-variables-with-gower-distance-in-r, that weights can be set for columns but not rows.

I also looked at Do I need to remove duplicate objects for cluster analysis of objects? but I am not able to find a solution/best practice.

Any suggestions on how duplicate rows are handled in clustering?

Improve this question
$\endgroup$
1
  • $\begingroup$ In methods like k medoids or k means duplicated objects clearly affect results. If weighting option is not or cannot be implemented then you are left with the usual problem of too-big-matrix-to-analyze. Try to do your clustering on a random subset(s) of your 200k object dataset. $\endgroup$
    – ttnphns
    Commented Dec 16, 2018 at 9:57

1 Answer 1

Reset to default
1
$\begingroup$

Get the source code.

The cluster package is open source, so you can modify it and add support for weighting. I am convinced the authors will be happy to receive such an extension and make it available for future users!

But before, you may want to first check that you are on the right track:

  1. Try a sample. Use 10k rows, and check if the clusters work for you.
  2. Try unweighted. Maybe the duplicates don't need to be considered, and the result is good enough without (or even better?)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.