I am using kmeans for clustering and if I read the topics around here and somewhere else it is always recommended to do a graphical check-up for the number of clusters.

The problem is my orginal dataframe has 80 features. So I did a pca in python

pca = PCA(n_components=2)
X_pca = pca.fit_transform(data)

Is it ok to use this as analysis now? The output looks like this (only two dimensions): enter image description here

Or do I miss something when I did the pca?


To determine the number of clusters is a hard problem. Let $K$ be the number of clusters.

Larger K means smaller "within cluster variation" (W), which is good.

Larger K means smaller "Between cluster variation" (B), which is bad.

One solution found in the literature is to use the ration between the two metrics and it is called the CH index=B/W. See for instance Tibshirani slides.

Use the CH-index in Sklearn.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.