3
$\begingroup$

I have five (normalized) variables related to family history of insurance applicants. n = 59,831.

I performed a Principal Components Reduction, reduced it to 3 dimensions, put together explains 90.1% variance. When i plot the principal component vectors in 3-D, i get what i can see as 3 clear clusters, as below, which are split along the PC2 dimension (1 & 3 cross-section does not see as clear split) (plotted using scatterplot3d library in R):3 Apparent clusters along 3 Principal Component Directions

The question is how do i now assign the labels to these clusters? I kind of 'know' that there are 3 clusters, so this seems to add a supervised flavour to the problem. I tried k-means along just the PC2 dimension, and also along all 3 dimensions but even with a lot of nstarts (=1000) the clusters dont seem to come as expected: below is clustered along PC2 dimension using K-means = 3, I am guessing that K-means is trying to divide the clusters 'equally' (same number of n in each cluster?)

enter image description here

And below is clustered using kmeans=3 , along PC1 and PC2 dimensions (similar results for all 3 dimensions). The new split along PC1 suggests that adding more dimensions to the clustering exercise may not be a good idea.

enter image description here

So, using just PC2 as the clustering variable, how would i get the correct clusters? Thanks in advance...

UPDATE & EDIT - Thanks to good suggestions, I tried applying the GMM with k = 3. I am getting results like the below. These are much better than using k-means. However, the 'ends' of the clusters dont seem to be assimilated within their main distributions..I'm getting the feeling that all the 'outliers' are getting their own cluster here. It seems to me that none of the surrounding distributions wants to take 'responsibility' for these red points in between the green and black clusters. enter image description here

I used the mvnormalmixEM and the normalmixEM (for just PC2 dimension) to get the posterior probabilities after convergence, and then assigned the label where the posterior probability was the highest among the three GMM components. I didn't set any initial values of the prior responsibilities/weights [lambda parameter]. Would anyone have ideas on how to fine-tune? It feels to me that the red points arent falling within the distribution parameters that the EM algorithm has discovered, and that I might need to suggest to the algorithm to search for larger variance parameters, but I am unsure how to do this

Thanks in advance

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Isn't this just normal clustering? You did dimentionality reduction (which is not clustering), and now you want do clustering to assign points into clusters. I'm not seeing the supervised part here. $\endgroup$
    – Frames Catherine White
    Commented Jan 26, 2018 at 7:13
  • $\begingroup$ Hi Lyndon, yes, you are probably right given my inexperience..it just felt to me that PCA has 'given' me the clusters in the diagrams above, and now I want to add the label to them, which label I can check using the graphs above. If i can find the correct clustering algorithm (which i can then validate using my PCA data), then i might be able to re-apply that algorithm to my test data set, as a new feature? $\endgroup$
    – Nitin
    Commented Jan 26, 2018 at 7:23
  • $\begingroup$ You can use GMM with 3 clusters in this case: it will take care of difference of densities for each cluster $\endgroup$
    – ahstat
    Commented Jan 26, 2018 at 7:28
  • 1
    $\begingroup$ Don't use k-means, use mixture of Gaussians with k=3. $\endgroup$
    – amoeba
    Commented Jan 26, 2018 at 8:05
  • $\begingroup$ Thanks ahstat & amoeba for the direction - the GMM does seem to answer what I am looking for, leading to a 'soft' classification. I also found this useful resource: tinyheero.github.io/2015/10/13/mixture-model.html. Will try it and update $\endgroup$
    – Nitin
    Commented Jan 26, 2018 at 8:25

3 Answers 3

Reset to default
2
$\begingroup$

K means is very sensitive to feature scaling. So it's hard to get mathematically meaningful results when your axes have different units!

Furthermore, the PCA plot is likely misleading. It is probably showing some artifact of your data. There probably is some attribute with very high variance and just three levels (e.g. income 1000, 10000, 100000) that dominates this visualization in an undue way. The other attributes then just add a "gaussian blur" to this. So all you are doing is reverse engineer that attribute that already is in your data. My guess is that you can simply identify it by looking at the highest variance attribute.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Out of 5 original variables, one of them had 3 levels - 1,2,and 3. I did scale them to between 0 and 1 before doing PCA. The variance is not so different as the other variables though, which were previously normalized oto between 0 and 1. Interesting insight, and thanks $\endgroup$
    – Nitin
    Commented Feb 22, 2018 at 13:37
  • $\begingroup$ On second thought, I guess the variance between 0 and 0.5 and 1 is probably higher than variance between 0.0001 and 0.0002 and 0.0010 up to 0.9999... $\endgroup$
    – Nitin
    Commented Feb 22, 2018 at 14:51
1
$\begingroup$

Finally got the below using GaussianMixture in python's scikit-learn v0.19. This allowed me to specify that the covariance matrix should be shared across clusters (option = 'tied' in the covariance_type param) - something which doesn't seem available as an option in R.

I also perhaps, 'cheated' a little bit by roughly specifying the initial distribution weights and the initial rough centroid locations by visual inspection of the PCA scatterplot.

enter image description here

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Tied exists in Mclust in R. But are you sure you are not seeing (and clustering) and artefact from low resolution input data? $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Feb 21, 2018 at 8:23
0
$\begingroup$

It will be quite useful to look also at the loadings. Which is the variable that create the scatter plot you see ? I always do not trust only scatter plots. Changing axes you can find cluster almost wherever you would like to and for sure you can find a methodology that can give you 3 clusters.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The variables are just the pca scores along the 3 dimensions, for all 59,831 rows. The pca scores were got using the x vector of the result of prcomp() function. I also used rgl library to rotate the axes - the 1 & 3 cross section is quite uninteresting, a big blob, where the 1 & 2 and 2 & 3 cross sections seemed 'clustery' $\endgroup$
    – Nitin
    Commented Jan 26, 2018 at 8:29
  • $\begingroup$ Yes but how loadings relates your component to your original variable ? (as an example in 2d you can plot a biplot with loadings related to original variable superimposed) support.sas.com/documentation/cdl/en/statug/63347/HTML/default/… $\endgroup$
    – Jojostack
    Commented Jan 26, 2018 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.