Can someone explain to a newbie the concepts of linear discriminant analysis? I am not looking for a technical implementation like this. I wish to understand it conceptually. I understand logistic regression and a little bit about naive Bayes classification, but cannot make any sense of LDA.

What problem does LDA solve that is not suited for Naive Bayes or logistic regression? More than two output categories?


2 Answers 2


1. LDA or logistic regression?

LDA and logistic regression can both be used to 'predict' the class of a subject, both can handle the case of more than two classes.

They both differ in the way of solving the classification problem and therefore they make different assumptions: logistic regression assumes the well-known S-shape, while LDA assumes that in each class your data are (1) multivariate normal and (2) with the same var-covar matrix in each class.

If the assumptions of multivariate normality and same var-covar are fulfilled then, in general, LDA will perform better.

2. Intuition behind LDA

You have 'subjects' that are characterized by features $x_1, x_2, \dots x_n$. The goal is to decide on the class of the subject, knowing the value of its features.

As said, LDA assumes that, in each class '$c$', your features have a multivariate distribution with a mean that depends on the class, so $\mu_c$ (note that this is a vector) and var-covar $\Sigma$ ( the same for all the classes), so for each class we know the multivariate density $\Phi_c(\mu_c,\Sigma)$ that allows us to calculate the probabilities.

Now, given the features, we can compute the $\Phi_c$ for the featurs $x_i$ and we will put the subject in that class $c$ where this yields the highest value (i.e. Where the 'probability' is heighest).

3. Why is LDA a dimension reduction technique?

LDA assumes multivariate normality in each class with the same var-covar. Therefore the classes are all the 'same' except for their mean. So you can 'feel' that the number of means will be important.

If you have $n$ features, then, in the end, the solution will depend on $C$ means, $C$ being the number of classes. In fact it can be show mathematically that LDA 'solves' the classification problem in a 'subspace' of the $n$-dimensional feature space, and that this subspace has dimension that is lower than $C$.

To make it more concrete, assume that you have subjects with $25$ features and you want to classify them in two classes, then you can 'solve' the problem in a one-dimesnional space (thus on a line). This is why LDA is said to be a dimension reduction technique, in this case it reduces the dimension from $25$ to one .

  • $\begingroup$ +1. However, there is a point in your answer section 3 which appeals for clarification from you. Consider two identical in shape, size, and orientation ellipsoids (that is, two data clouds with equal covariance matrices) differing only in some relative shift in their centroids in the p-dimensional space (p>=2) they lie in. (Let the ellipsoids not superpose, be clear.) Then, do always just the single discriminant line suffice to decide correctly which cloud every their point belong to? $\endgroup$
    – ttnphns
    Sep 18, 2015 at 10:13
  • $\begingroup$ (cont.) Or, to put the question other way, can ever a classification (such as Bayes) of such data performed based on all the p dimensions be better than the classification performed on that single discriminant line? $\endgroup$
    – ttnphns
    Sep 18, 2015 at 10:13
  • $\begingroup$ (cont) Or, put this same problem yet again in other words. Given the two-class data I described and let p=2: can the line perpendicular to the discriminant line bear some remnant classification-relevant information, that is, the information not captured by the discriminant line? $\endgroup$
    – ttnphns
    Sep 18, 2015 at 10:34
  • $\begingroup$ @ttnpns: If your two ellipsoids are perfectly seperable, then you could e.g. connect the two centers and project everything on that line, or do I miss something here ? $\endgroup$
    – user83346
    Sep 18, 2015 at 13:52
  • $\begingroup$ @ttnphs, I have a question regarding LDA/FDA. Let say, original dataset: 100 features/dimensions; 2 classes; 50 samples/examples. We start with n dimensions and end with k dimensions, where k < n. and is not restricted to only 2 output value. Is that correct? $\endgroup$
    – aan
    May 6, 2020 at 19:43

This site has a pretty good answer: https://medium.com/@QuarizmiAdTech/a-full-introduction-to-the-linear-fisher-discriminant-analysis-848530dce336

However, I think that understanding Principal Component Analysis before diving into LDA is a good idea

Imagine trying to classify images of animals - lets say Cats and Dogs only. What features in the images are useful to identify the fruit? (Technically, identify = classify)

Imagine that the information dataset for above case has 20 attributes. Do you need all 20? Only 4?

You would use LDA to reduce these attributes. It finds if 2 variables are correlated. And if they are, then you can remove one of the correlated attributes. As a different example, if a dataset shows that height and weight are very strongly linked, then you can use either height weight as a classification attribute - you don't need both.

The weakness of LDA: The attributes have to be somehow comparable! For example, the LDA for identifying attributes to compare Apples vs Bananas would not work for Cats Vs Dogs. Mathematically, this implies that you there is a linear relationship of the type y = ax + b between x and y. (Add more for multiple attributes).

Obviously this will fail for quite a few cases - but the number of cases where LDA can work is surprisingly high - and it is also very useful in driving down computation costs: 5 attributes out of 20 may be enough to gather 98% confidence; and o get to 99.99% you might need 16. It is obvious that we would prefer to be able to work with 5 attributes, because the computation cost jump could be from 5$ to 70/80 $, not 5 to 10

  • $\begingroup$ I have a question regarding LDA/FDA. Let say, original dataset: 100 features/dimensions; 2 classes; 50 samples/examples. We start with n dimensions and end with k dimensions, where k < n. and is not restricted to only 2 output value. Is that correct? $\endgroup$
    – aan
    May 6, 2020 at 19:44
  • $\begingroup$ Sorry - missed this for a long while. Yes you are correct - LDA is not restricted to using only 2 dimensions as the final selection. Overall, LDA has 2 intents: 1) Remove correlated attributes / dimensions - ex - height and weight as above. If you have one, the other is redundant 2) Remove attributes / dimensions that have very little impact on the final prediction. So if 4 dimensions yield 98% confidence, and the next 16 increase the confidence level to 98.5%, then you will likely only work with the first 4 dimensions $\endgroup$
    – A. Kumar
    Jun 18, 2020 at 20:27
  • $\begingroup$ thanks. Is Small sample size is a problem for LDA? $\endgroup$
    – aan
    Jun 19, 2020 at 13:31
  • $\begingroup$ Sorry, what is the meaning of linear relationship in LDA? $\endgroup$
    – aan
    Jun 19, 2020 at 13:35
  • $\begingroup$ Yes, small sample size is a problem for LDA; especially when sample size is less than the number of dimensions. Linear: Say you have 4 dimensions: A, B, C, D. LDA can detect that A = 4B + 3C + 2D + 1 ==> All of A, B, C and D are limited to the power of one. Implying that if A = B^2 + C^2, LDA will not be able to detect it. However this last case (A = B^2 + C^2) is not a problem. You might replace B^2 by E, and C^2 by F, thus again finding a linear relationship: A = E + F $\endgroup$
    – A. Kumar
    Jun 28, 2020 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.