I am trying to separate my dataset into meaningful clusters. I have tried k-means, hierarchical and EM clustering (fitting a gaussian mixture model using EM algorithm, using the EMCluster R package) on the output of PCA. Of these, EM clustering creates the clusters that seem "nicest" in the sense that they seem to be intuitively meaningful and they correspond well to the visual appearance of the 2D scatter plot on the first two principal components. I have read, however, that there is some theoretical correspondence between PCA and k-means clustering. Does that in any way constrain which clustering algorithm should be used on PC data?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ What is "EM clustering"? Fitting a Gaussian mixture model via expectation-maximization algorithm? $\endgroup$– amoebaSep 4, 2015 at 15:58
-
$\begingroup$ amoeba: Yes, I have updated to clarify. $\endgroup$– fnsSep 4, 2015 at 16:07
-
$\begingroup$ What is the dimensionality of your dataset? How many PCs do you retain to fit GMM? Do you apply k-means and hierarchical clustering to the full-dimensional data? $\endgroup$– amoebaSep 4, 2015 at 19:24
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It's interesting you note that EM performs nicely from a visual standpoint, because PCA is assuming a linear generative model, whereas clustering can accommodate non-linear.
PCA has a probabilistic interpretation: The hidden component (the principal components) distribution, conditioned on the observed variables, is multivariate normal: http://research.microsoft.com/pubs/67218/bishop-ppca-jrss.pdf.
Clustering with GMM's probabilistic interpretation: The hidden components are a mixture distribution, sum of several multivariate normal: https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/0802_Reynolds_Biometrics-GMM.pdf.
Why not apply clustering to the original dataset?
