7
$\begingroup$

I have a specific problem which I'm surprised I don't find answers on-line and I hope somebody here has a good suggestion for me. I'm working with a large data set which I'm clustering into specific groups using custom cluster densities. So the original space is quite heterogeneous and has a large number of feature. During and after model optimization, the responsibility vector of each data point is projection of the original features into k dimensions (k = number of clusters). Each such vector also sums to one, which makes them a (discrete) distribution over the space of clusters. Using these soft assignments, I can always create a heatmap visualization of the corresponding confusion matrix (when I have categorical labels). Typically, these assignments are quite sharp, meaning that many vectors are of the form (0,0,1,0,0...)

Now, I would like to see a (2d) representation of the points grouped by the information in the responsibility matrix. This way, I could visualize each step in the optimization procedure without knowing actual labels. My first thoughts were PCA, MDS and graph layout algorithms. However, although the heatmap suggests a clear grouping, the PCA does look quite dense with points aligned in lines. As MDS and graphs use distances, I thought about calculating pairwise Hellinger or earth mover distances between the responsibility vectors to apply any of the MDS or graph layout algorithms. However, MDS wasn't successful so far.

Has anybody done something like this before? Ideally, I would like to see an animation of the clustering procedure as data points being grouped together.

$\endgroup$
  • 2
    $\begingroup$ It may be helpful if you can post some graphs here so that statements like 'PCA does look quite dense' are better understood. $\endgroup$ – rnso Apr 16 '15 at 12:22
1
$\begingroup$

Try visualization methods, such as surface plots and other techniques for high-dimensional data visualization, described in the paper "mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation" by Chris Fraley, Adrian E. Raftery, T. Brendan Murphy and Luca Scrucca (http://www.stat.washington.edu/research/reports/2012/tr597.pdf). Specifically, check section 8 (pp. 35-43) and plotting functions summary on p. 52.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.