I have a data set which is 3 columns and about 50,000 rows. I am trying to do some cluster analysis and would like to speed it up. Most examples online use data sets like
iris which are relatively tiny. Take the following code:
nb <- NbClust(d, distance = "euclidean", min.nc = 2, max.nc = 30, method = "complete", index ="all")
This has been running for 24 hours so far on a server with 32 cores and 100 gig of ram, although only one core will be being used.
There doesn't seem to be a way of doing any parallel processing unless I use a loop that tackles each index choice individually. That will speed up process considerably. However the real issue is size of the data. And 50,000 rows is hardly 'Big data'.
It seems using samples to speed up the process is a good idea. Like how
clara is much faster than
My question is, if I were to take samples and run
NbClust, what size should the samples be and how many samples are sufficient?
I'd like the answer as a proportion of
n - the number of total rows, such that this answer scales.
A final note would be that before doing any of this work, I would run
hopkins) to check the Hopkin's statistic to see if clustering even makes sense.
I have run:
clustend <- get_clust_tendency(scale(d), 5000)
But on large data sets this will also take a long time and I'm not sure if using samples like my above idea is wise or if you should really test the whole set to be sure. I'm also not sure if 5000 was the right number to choose, I'd seen examples where the number chosen was roughly 1/3 of the size of the data set, however these sets were under 300 rows and the choice hadn't been explained well.
set.seed(123) d <- data.frame(x = unlist(lapply(1:500, function(i) rnorm(100, runif(1)*2^2))), y = unlist(lapply(1:500, function(i) rnorm(100, runif(1)*2^2))), z = unlist(lapply(1:500, function(i) rnorm(100, runif(1)*2^2))))
Here are the results of nb clust, which is why I don't want to use less tests and also may be evidence why a sample may misrepresent the data
* Among all indices: * 6 proposed 2 as the best number of clusters * 5 proposed 3 as the best number of clusters * 2 proposed 5 as the best number of clusters * 4 proposed 8 as the best number of clusters * 2 proposed 10 as the best number of clusters * 1 proposed 13 as the best number of clusters * 1 proposed 23 as the best number of clusters * 1 proposed 24 as the best number of clusters * 1 proposed 27 as the best number of clusters