Conceptual undersanding of linear discriminant analysis

Can someone explain to a newbie the concepts of linear discriminant analysis? I am not looking for a technical implementation like this. I wish to understand it conceptually. I understand logistic regression and a little bit about naive Bayes classification, but cannot make any sense of LDA.

What problem does LDA solve that is not suited for Naive Bayes or logistic regression? More than two output categories?

1. LDA or logistic regression?

LDA and logistic regression can both be used to 'predict' the class of a subject, both can handle the case of more than two classes.

They both differ in the way of solving the classification problem and therefore they make different assumptions: logistic regression assumes the well-known S-shape, while LDA assumes that in each class your data are (1) multivariate normal and (2) with the same var-covar matrix in each class.

If the assumptions of multivariate normality and same var-covar are fulfilled then, in general, LDA will perform better.

2. Intuition behind LDA

You have 'subjects' that are characterized by features $x_1, x_2, \dots x_n$. The goal is to decide on the class of the subject, knowing the value of its features.

As said, LDA assumes that, in each class '$c$', your features have a multivariate distribution with a mean that depends on the class, so $\mu_c$ (note that this is a vector) and var-covar $\Sigma$ ( the same for all the classes), so for each class we know the multivariate density $\Phi_c(\mu_c,\Sigma)$ that allows us to calculate the probabilities.

Now, given the features, we can compute the $\Phi_c$ for the featurs $x_i$ and we will put the subject in that class $c$ where this yields the highest value (i.e. Where the 'probability' is heighest).

3. Why is LDA a dimension reduction technique?

LDA assumes multivariate normality in each class with the same var-covar. Therefore the classes are all the 'same' except for their mean. So you can 'feel' that the number of means will be important.

If you have $n$ features, then, in the end, the solution will depend on $C$ means, $C$ being the number of classes. In fact it can be show mathematically that LDA 'solves' the classification problem in a 'subspace' of the $n$-dimensional feature space, and that this subspace has dimension that is lower than $C$.

To make it more concrete, assume that you have subjects with $25$ features and you want to classify them in two classes, then you can 'solve' the problem in a one-dimesnional space (thus on a line). This is why LDA is said to be a dimension reduction technique, in this case it reduces the dimension from $25$ to one .

• +1. However, there is a point in your answer section 3 which appeals for clarification from you. Consider two identical in shape, size, and orientation ellipsoids (that is, two data clouds with equal covariance matrices) differing only in some relative shift in their centroids in the p-dimensional space (p>=2) they lie in. (Let the ellipsoids not superpose, be clear.) Then, do always just the single discriminant line suffice to decide correctly which cloud every their point belong to? – ttnphns Sep 18 '15 at 10:13
• (cont.) Or, to put the question other way, can ever a classification (such as Bayes) of such data performed based on all the p dimensions be better than the classification performed on that single discriminant line? – ttnphns Sep 18 '15 at 10:13
• (cont) Or, put this same problem yet again in other words. Given the two-class data I described and let p=2: can the line perpendicular to the discriminant line bear some remnant classification-relevant information, that is, the information not captured by the discriminant line? – ttnphns Sep 18 '15 at 10:34
• @ttnpns: If your two ellipsoids are perfectly seperable, then you could e.g. connect the two centers and project everything on that line, or do I miss something here ? – user83346 Sep 18 '15 at 13:52

This site has a pretty good answer: https://medium.com/@QuarizmiAdTech/a-full-introduction-to-the-linear-fisher-discriminant-analysis-848530dce336

However, I think that understanding Principal Component Analysis before diving into LDA is a good idea

Imagine trying to classify images of animals - lets say Cats and Dogs only. What features in the images are useful to identify the fruit? (Technically, identify = classify)

Imagine that the information dataset for above case has 20 attributes. Do you need all 20? Only 4?

You would use LDA to reduce these attributes. It finds if 2 variables are correlated. And if they are, then you can remove one of the correlated attributes. As a different example, if a dataset shows that height and weight are very strongly linked, then you can use either height weight as a classification attribute - you don't need both.

The weakness of LDA: The attributes have to be somehow comparable! For example, the LDA for identifying attributes to compare Apples vs Bananas would not work for Cats Vs Dogs. Mathematically, this implies that you there is a linear relationship of the type y = ax + b between x and y. (Add more for multiple attributes).

Obviously this will fail for quite a few cases - but the number of cases where LDA can work is surprisingly high - and it is also very useful in driving down computation costs: 5 attributes out of 20 may be enough to gather 98% confidence; and o get to 99.99% you might need 16. It is obvious that we would prefer to be able to work with 5 attributes, because the computation cost jump could be from 5$$to 70/80$$, not 5 to 10