4
$\begingroup$

I am trying to separate my dataset into meaningful clusters. I have tried k-means, hierarchical and EM clustering (fitting a gaussian mixture model using EM algorithm, using the EMCluster R package) on the output of PCA. Of these, EM clustering creates the clusters that seem "nicest" in the sense that they seem to be intuitively meaningful and they correspond well to the visual appearance of the 2D scatter plot on the first two principal components. I have read, however, that there is some theoretical correspondence between PCA and k-means clustering. Does that in any way constrain which clustering algorithm should be used on PC data?

$\endgroup$
3
  • $\begingroup$ What is "EM clustering"? Fitting a Gaussian mixture model via expectation-maximization algorithm? $\endgroup$ – amoeba Sep 4 '15 at 15:58
  • $\begingroup$ amoeba: Yes, I have updated to clarify. $\endgroup$ – fns Sep 4 '15 at 16:07
  • $\begingroup$ What is the dimensionality of your dataset? How many PCs do you retain to fit GMM? Do you apply k-means and hierarchical clustering to the full-dimensional data? $\endgroup$ – amoeba Sep 4 '15 at 19:24
2
$\begingroup$

It's interesting you note that EM performs nicely from a visual standpoint, because PCA is assuming a linear generative model, whereas clustering can accommodate non-linear.

PCA has a probabilistic interpretation: The hidden component (the principal components) distribution, conditioned on the observed variables, is multivariate normal: http://research.microsoft.com/pubs/67218/bishop-ppca-jrss.pdf.

Clustering with GMM's probabilistic interpretation: The hidden components are a mixture distribution, sum of several multivariate normal: https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/0802_Reynolds_Biometrics-GMM.pdf.

Why not apply clustering to the original dataset?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.