I have a large data set: over 100,000 data points, each with 60 dimensions. I want to display the data in 2D to visibly maximize the separation between classes, which I know for each point. I asked a similar question before, and Linear Discriminant Analysis (LDA) was suggested -- a solid answer.

Now, besides knowing that each data point falls into a certain class (say 1/2/3), each of these data points also has a second type of classification (say A/B/C). I would like to visualize how well these points cluster by one class, and then see how similar the substructures of the clusters are when each cluster is broken up based on the other class. For example, if we choose the numerical class for the "outer" clustering and the alphabetical class for the "inner" clustering, I would want to find the 2D representation of the data which makes it look maximally like this:

A sample classification diagram

Bonus points if we can find a way to guarantee that the A/B/C subclasses are all oriented the same way with respect to each other, for each main class. Note, however, that I would also expect to be able to perform the clustering in the opposite order (with the alphabetical class forming the "outer" clusters and the numerical class forming the "inner" clusters) because the classifications are independent.

The problem with LDA in this context is that, when considering only one class at a time, it maximizes the between-class separation at the expense of potentially blurring the boundaries of the other class in the resulting visualization, because it doesn't know about that the other class exists. So, when I visualize LDA on one classification, the distinctions for the second classification are not as clear as they could be. However, if I do a 9-way LDA (with each class being a combination of the two classifications, like 1A), I see 9 clusters, but they are not grouped together in any easily visualizable way; this is expected, however, because the classifications are independent (not to mention the fact that there are now 8 LD dimensions).

I think about it this way. Say I start at center of the figure above. If I know which instance of the numerical class I want, I know which direction to move in: down for 1, up-left for 2, up-right for 3. The goal is to show that, no matter which numerical cluster I went to, I have the same directional options if I want to then move from that cluster's center to an alphabetical cluster. In each case, it's up-left for A, down for B, and up-right for C. And of course I would expect the same to be true if the classifications were performed in reverse (although this would require a different figure).

Are there any standard techniques for doing this kind of hierarchical cluster visualization? If not, any advice on how to get started?


The best solution I've found so far is to perform two separate LDAs: one for the alphabetical class and one for the numerical class. Then I can visualize the first dimension of each in the same graph, like this:

Best-so-far classification diagram

While this gives me something like the picture I was looking for, I'm not sure that it proves what I set out to prove, which is that the data represent these two types of classifications independently. While this graph makes it look like the data bear this out, to make the argument robust I'll have to show that the alphabetical LD and the numerical LD are uncorrelated.


1 Answer 1


I'm afraid that your expectance that the data cluster in space hierarchically, as classes and subclasses, is an illusion. If it had been true then the 9-class LDA could have not concealed the fact that the subclasses are closer to each other than the classes. The presumption must be that there are 9 classes, and if the analysis trying to distinguish them all renders the nested configuration, then we may say it exists. And this is probably not your case.

Did you check your data with some clustering criterion, such as Calinski-Harabasz? It could be that the 9-cluster solution is not worse or even better than the 3-cluster solution. If so, you can't speak of "3 subclasses within 3 classes" as natural structure of your data.

  • $\begingroup$ Sorry, I must have been unclear in my question. The idea is not that the subclass clusters need to be closer, in the data, than the main class clusters. In fact, these are two independent classifications; the main/sub relationship only applies to the view I'm trying to construct. I could just as easily make a view where the A/B/C classes are the main clusters and the 1/2/3 classes form the subclusters. There is no reason to expect that this is a "natural" structure for my data; I just want to see the extent to which it can be forced into such a structure. I'll clarify this in my question. $\endgroup$
    – dmonner
    Oct 15, 2011 at 17:33
  • $\begingroup$ Then, sorry, LDA is probably not for you. Like PCA, LDA does not force data to follow user-preferred visual structure, it simply projects data the best way to visualize the separateness of classes. And it is for one-way classification only. $\endgroup$
    – ttnphns
    Oct 15, 2011 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.