I have a data set of 40,000 individuals which I clustered using k-means. I used 30 variables, each ordinal from 1=minimum to 5=maximum. I reduced these 30 variables to 10 factors and ran K-means on these new variables. I kept a clustering of 12 clusters.

I now have 7000 new observations, I want to classify these using a discriminant function. I built one in SPSS using the 30 previous variables. I get a good clasification on the original dataset (40,000), but when I run it on the new observations, one segment doesn't show at all. 0 cases are of this type. This makes no sense to me.

The original clusters seem healthy and apart, I did them in R with multiple different initial center methods and ran them many many times, and stuck with the best one. The discriminant function has no problem finding this "disappearing" cluster in the original dataset.

What could be causing this?

  • 1
    $\begingroup$ Could this "disappearing" cluster be very small, actually a outlier cluster? Could the finding be due to that you clustered the original set in 10 principal components but performed LDA on original 30 variables? $\endgroup$ – ttnphns Aug 25 '12 at 7:16
  • 2
    $\begingroup$ Are you sure your model is not overfitted? PCA sometimes can generate nice, but purely artificial clusters. $\endgroup$ – user88 Aug 25 '12 at 23:48
  • $\begingroup$ @mbq Could you by any chance point me to a source where I can read about this issue with PCA and clustering please? $\endgroup$ – JEquihua Mar 22 '14 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.