I have a data set consisting of 15K labeled samples (of 10 groups). I want to apply dimensionality reduction into 2 dimensions, that would take into consideration the knowledge of the labels.

When I use "standard" unsupervised dimensionality reduction techniques such as PCA, the scatter plot seems to have nothing to do with the known labels.

Does what I am looking for have a name? I would like to read some references of solutions.

  • 3
    If you are looking for linear methods, then linear discriminant analysis (LDA) is what you should be using. – amoeba Jul 14 '15 at 11:57
  • @amoeba: Thanks. I used it and it performed much better! – Roy Jul 14 '15 at 13:18
  • Glad that it helped. I provided a brief answer with some further references. – amoeba Jul 14 '15 at 14:17
  • 1
    One possibility would be to first reduce to the nine-dimensional space spanning the class centroids, and then use PCA to further reduce to two dimensions. – A. Donda Jul 25 '15 at 15:46
  • Related: stats.stackexchange.com/questions/16305 (possibly duplicate, though perhaps the other way round. I will come back to this after I update my answer below.) – amoeba Jun 5 at 10:56
up vote 25 down vote accepted

The most standard linear method of supervised dimensionality reduction is called linear discriminant analysis (LDA). It is designed to find low-dimensional projection that maximizes class separation. You can find a lot of information about it under our tag, and in any machine learning textbook such as e.g. freely available The Elements of Statistical Learning.

Here is a picture that I found here with a quick google search; it shows one-dimensional PCA and LDA projections when there are two classes in the dataset (origin added by me):

PCA vs LDA

Another approach is called partial least squares (PLS). LDA can be interpreted as looking for projections having highest correlation with the dummy variables encoding group labels (in this sense LDA can be seen as a special case of canonical correlation analysis, CCA). In contrast, PLS looks for projections having highest covariance with group labels. Whereas LDA only yields 1 axis for the case of two groups (like on the picture above), PLS will find many axes ordered by the decreasing covariance. Note that when there are more than two groups present in the dataset, there are different "flavours" of PLS that will produce somewhat different results.

Update (2018)

I should find time to expand this answer; this thread seems to be popular but my original answer above is very short and not detailed enough.

In the meantime, I will mention Neighbourhood Components Analysis -- a linear method that finds the projection maximizing $k$-nearest-neighbours classification accuracy. There is a nonlinear generalization using neural networks, see Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure. One can also use neural network classifiers with a bottleneck, see Deep Bottleneck Classifiers in Supervised Dimension Reduction.

  • 1
    nice graphic, explains much – Titou Feb 2 '17 at 9:14

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.