5
$\begingroup$

I am working with dimensionality reduction algorithms. Linear Discriminant Analysis (LDA) is a supervised algorithm that takes into account the class label (which is not the case of PCA for example). I am using Python to do a comparative study between some algorithms.

Why with two classes (k = 2), regardless of the data dimensionality, LDA gives one dimension, i.e. the new subspace is composed of only one dimension? For example if I try to project my dataset (of 100 attributes and 2 classes) to 10-dimensional space by using LDA(n_components=10) I only obtain 1 dimension?

For k>2 I can obtain the number of discriminant vectors that I specified as n_components. How can I get n_components dimensions for k=2?

$\endgroup$
  • 2
    $\begingroup$ I don't understand the difference between Q1 and Q2 and I think Q3 is much too vague and does not belong here. Your Q1 is a good question though. $\endgroup$ – amoeba says Reinstate Monica Sep 25 '18 at 9:28
  • $\begingroup$ It seems it was asked before a few times but stayed unanswered... stats.stackexchange.com/questions/252616 stats.stackexchange.com/questions/191908 $\endgroup$ – amoeba says Reinstate Monica Sep 25 '18 at 9:30
  • $\begingroup$ You are right! the 2nd and 3rd questions are drawn from the 1st one. Q3 is in the case that with 2 classes I get only one dimension by applying LDA, what should I change to get the number of dimensions that I precise, if it is possible. $\endgroup$ – mbahri Sep 25 '18 at 9:31
  • $\begingroup$ stats.stackexchange.com/a/190821/3277 answers the title $\endgroup$ – ttnphns Sep 25 '18 at 9:53
  • 3
    $\begingroup$ Maybe I am missing something obvious here, but the goal of LDA is to discriminate among the classes; with 2 classes you have 1 "discirminator' function, right? $\endgroup$ – Peter Flom - Reinstate Monica Sep 25 '18 at 11:01
4
$\begingroup$

Like @amoeba, I don't understand your difference between Q1 and Q2: in any case, LDA obtains at most $k-1$ dimensions. For $k=2$ that's one dimension, for $k=8$ it would be 7. And of course, if the input space has lower dimensionality that that's the limiting factor.


One idea/heuristic behind LDA is that separating classes is easy if they are all spheres of the same size: all you then need is distance to the class means. Alternatively, you can say that the connection between class means are normal vectors for suitable separation planes.
In that narrative, you can look at LDA as a projection that in the first place does singular value decomposition (i.e. something very similar to PCA) of the pooled covariance matrix in order to get a space with such spherical classes. Let's call this our primary score space.

Depending on the customs of your field, this is either the PCA score space of the pooled covariance matrix = the PCA score space of the data matrix centered to the respective class means, or the PCA score space squeezed according to the eigenvalues/SVD diagonal matrix (some fields by default put this scaling for PCA into the loadings, others [mine] into the scores).

The primary score space still has the dimensionality corresponding to the rank of the pooled covariance matrix/data matrix centered to the respective class means. Pooled covariance matrix in this space is a unit sphere, i.e. each class is considered as a unit-sphere centered at the respective class mean, the means also projected into score space.

(At this point, we could derive other classification algorithms that use the SVD projection as heuristic but don't exploit/rely on the assumed unit-sized spherical shape of the classes.)

Now, as we have unit-spherical class shapes, the only thing remaining to care about are the class means. I.e., $k$ points. Even if our primary score space has higher dimensionality, the $k$ points will form a $k-1$ dimensional shape (simplex). Thus without loss of anything, we can further rotate our primary score space so that our $k-1$-simplex of class means lies in the first $k-1$ dimensions.
For the postulted spherical classes, all further dimensions cannot not help with the distinction: the classes have exactly the same size and position in these further dimensions. Thus, we throw them away.

(Again, you may derive another classifier that uses the first two projection steps but then keeps [some of] those further dimensions as the classes in practice may not be spherical.)


One classifer to look into when comparing LDA and PCA is SIMCA which can be seen as a one-class classifier analogous to LDA.

In SIMCA, you'll find the notion of in-model space and out-of-model space: in in-model space you detect changes of the same type of the usual variation within your class (but possibly of unusual magnitude). In out-of-model space you detect variation of a type that does not usually occur within any of the classes.

For our description of LDA, the $k-1$ dimensions we keep would be the in-model space, whereas the remaining dimensions are the out-of-model space. Interpretation will be slightly different compared to SIMCA, but you should have a start for your thoughts with this.

$\endgroup$
  • $\begingroup$ Thank you for your answer. So I understand that with LDA, the number of input dimension cannot be greater than (K-1) because it is a limiting factor. Is that right? $\endgroup$ – mbahri Sep 26 '18 at 11:14
  • 1
    $\begingroup$ @mbahri no, the number of output dimensions are limited by the assumptions that: 1. all classes fit in spheres of equal radius in each dimension, and 2. the radius of those spheres is $1$ (homoscedasticity). In that case, having more than $k-1$ dimensions is basically useless... $\endgroup$ – Matthieu Oct 30 '18 at 15:59
2
$\begingroup$

From the documentation:

discriminant_analysis.LinearDiscriminantAnalysis can be used to perform supervised dimensionality reduction, by projecting the input data to a linear subspace consisting of the directions which maximize the separation between classes (in a precise sense discussed in the mathematics section below). The dimension of the output is necessarily less than the number of classes, so this is, in general, a rather strong dimensionality reduction, and only makes sense in a multiclass setting.

http://scikit-learn.org/stable/modules/lda_qda.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.