I have high dimensional ($m \approx 2k$), high sample (n=140,000) dataset in R that I load into memory run PCA on it (returns $m \approx 400$ components to cover 95% of variance) then I run k means clustering on this dataset. However, even with wildly different number of clusters (from 1 to 1000) I always get the same total sum of squares. Though the cluster assigned to the datapoints I have inspected seem reasonable at first sight (they change seemingly appropriately with the number of clusters.)
So codewise:
trans = preProcess(train, method=c('BoxCox','center','scale','pca'))
train_pc = predict(trans, train)
kNumbers = c(1,5,10,15,20,25,100,1000)
for (i in kNumbers) {
model = kmeans(train_pc, centers=i, nstart=10)
cat(model$totss + '\n')
}
Things I have tried:
- Playing with
iter_max
from 10 to 100 (then usually no warning messages about lack of convergence) - Increasing the
n_start
(1 to 10) - Preprocessing the data
BoxCox
,center
,scale
Ultimately, this is just a part of feature engineering so if there is a way to get some generic way of informing the subsequent algorithm of some sort of idea of "big picture" closeness, then I am all up for that as well.
Any ideas?
TSS returned by K means clustering is always the same
. Why should it be different? It is WSS what is expected to differ. $\endgroup$