I'm trying to cluster sets by their similarity in terms of included elements. The group of possible elements is of size ~1 million. It is my understanding that in order to run k-means or a similar algorithm I'd have to create a dataframe that has a correspondingly large number of columns. This doesn't seem feasible.

Is there a clustering algorithm better suited for this problem? Will I be able to solve this problem as I've formulated it?

  • $\begingroup$ How many sets do you have? $\endgroup$ – user20160 Dec 7 '18 at 1:42
  • $\begingroup$ If size is an issue consider using Klara algorithm. $\endgroup$ – user2974951 Dec 7 '18 at 7:28
  • $\begingroup$ @user20160 roughly 500,000 sets $\endgroup$ – Patrick Connors Dec 7 '18 at 18:52

You can implement k-means with sparse data points. You just may need to do this yourself, rather than relying on someone else's code to be memory efficient enough for you.

However, k-means is likely not a good idea for other reasons: it requires continuous variables for it's least-squares and mean based approach to make sense.

On sets, the obvious alternative is to use an actual frequent itemset mining approach to identify common subsets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.