I have a specific problem which I'm surprised I don't find answers on-line and I hope somebody here has a good suggestion for me. I'm working with a large data set which I'm clustering into specific groups using custom cluster densities. So the original space is quite heterogeneous and has a large number of feature. During and after model optimization, the responsibility vector of each data point is projection of the original features into k dimensions (k = number of clusters). Each such vector also sums to one, which makes them a (discrete) distribution over the space of clusters. Using these soft assignments, I can always create a heatmap visualization of the corresponding confusion matrix (when I have categorical labels). Typically, these assignments are quite sharp, meaning that many vectors are of the form (0,0,1,0,0...)
Now, I would like to see a (2d) representation of the points grouped by the information in the responsibility matrix. This way, I could visualize each step in the optimization procedure without knowing actual labels. My first thoughts were PCA, MDS and graph layout algorithms. However, although the heatmap suggests a clear grouping, the PCA does look quite dense with points aligned in lines. As MDS and graphs use distances, I thought about calculating pairwise Hellinger or earth mover distances between the responsibility vectors to apply any of the MDS or graph layout algorithms. However, MDS wasn't successful so far.
Has anybody done something like this before? Ideally, I would like to see an animation of the clustering procedure as data points being grouped together.
