4
$\begingroup$

I've a got a data which I did a PCA on. I want to do a kmeans on the individuals coordinates on the 5 first principal components. Therefore I have a 200000 x 5 matrix of the coordinates. I'm looking to find a way to determine the optimal number of cluster so I can run a kmeans on my coordinates data using R. I found many methods to do that using R (here is a list : https://stackoverflow.com/questions/15376075/cluster-analysis-in-r-determine-the-optimal-number-of-clusters). None of those methods have worked for me because my data is too large. I get an error like : "negative length vectors are not allowed". I really need help on that because I shouldn't decide what number of cluster I should use, I have to let the statistic decide. Thank you very much.

$\endgroup$
  • $\begingroup$ Just out of curiosity, what's the goal of doing clusterization on individual features instead of all principal components? Maybe you need some statistics methods instead? $\endgroup$ – cyberj0g Aug 13 '15 at 9:35
  • $\begingroup$ Read about 'K-Means with splitting' algorithm... This variant continues to add more clusters to till such time there is at least one observation away from its centroid by more than a certain distance 'w' which you will have to define... as long as there is atleast one such observation, it will be made a new centroid and the 'K-means' algo is rerun... I dont think you will find much info on this variant of K-means on google... I doubt if there are any R packages that have this algo... you will have to code it yourself but its not that complicated... It has worked for me one many occasions $\endgroup$ – Gaurav Aug 13 '15 at 10:18
2
$\begingroup$

I actually solved my issue using the xmeans algorithm of ‘RWeka’ package. It's more relevant than kmeans, calculate automatically the number of clusters and run much faster than other methods. Here is a detailed mathematics description of the algorithm : https://www.cs.cmu.edu/~dpelleg/download/xmeans.pdf

And here is the package where you can find the xmeans algorithm : https://cran.r-project.org/web/packages/RWeka/RWeka.pdf

It took me a while to find such an efficient algorithm for my problem.

$\endgroup$
0
$\begingroup$

To solve your main issue with memory issues in R try Big Memory, which has been described here.

For a computationally efficient stopping rule for PCA try the broken-stick method. To learn more about the broken-stick method, read this.

$\endgroup$
  • $\begingroup$ This looks more as a comment than an answer. Will you consider adding descriptions of the options you link to, which might possibly make the answer informative and valuable? $\endgroup$ – ttnphns Aug 13 '15 at 12:06
  • $\begingroup$ @ttnphns I have updated the answer with two additional links. The OP cannot make a choice between methods, because the data is too big for R. I provide a link to a package that should resolve the type of error the poster is having. IMHO, the second part of the question has already been answered elsewhere - in the link that the OP provided. (I also point to an algorithm for PCA as this seemed more relevant when the questions was originally asked in a programming context at StackOverflow.) $\endgroup$ – noumenal Aug 13 '15 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.