I am trying to separate my dataset into meaningful clusters. I have tried k-means, hierarchical and EM clustering (fitting a gaussian mixture model using EM algorithm, using the EMCluster R package) on the output of PCA. Of these, EM clustering creates the clusters that seem "nicest" in the sense that they seem to be intuitively meaningful and they correspond well to the visual appearance of the 2D scatter plot on the first two principal components. I have read, however, that there is some theoretical correspondence between PCA and k-means clustering. Does that in any way constrain which clustering algorithm should be used on PC data?
It's interesting you note that EM performs nicely from a visual standpoint, because PCA is assuming a linear generative model, whereas clustering can accommodate non-linear.
PCA has a probabilistic interpretation: The hidden component (the principal components) distribution, conditioned on the observed variables, is multivariate normal: http://research.microsoft.com/pubs/67218/bishop-ppca-jrss.pdf.
Clustering with GMM's probabilistic interpretation: The hidden components are a mixture distribution, sum of several multivariate normal: https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/0802_Reynolds_Biometrics-GMM.pdf.
Why not apply clustering to the original dataset?