Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Can someone explain to a newbie the concepts of linear discriminant analysis? I am not looking for a technical implementation like this. I wish to understand it conceptually. I understand logistic regression and a little bit about naive Bayes classification, but cannot make any sense of LDA.
What problem does LDA solve that is not suited for Naive Bayes or logistic regression? More than two output categories?
LDA and logistic regression can both be used to 'predict' the class of a subject, both can handle the case of more than two classes.
They both differ in the way of solving the classification problem and therefore they make different assumptions: logistic regression assumes the well-known S-shape, while LDA assumes that in each class your data are (1) multivariate normal and (2) with the same var-covar matrix in each class.
If the assumptions of multivariate normality and same var-covar are fulfilled then, in general, LDA will perform better.
You have 'subjects' that are characterized by features $x_1, x_2, \dots x_n$. The goal is to decide on the class of the subject, knowing the value of its features.
As said, LDA assumes that, in each class '$c$', your features have a multivariate distribution with a mean that depends on the class, so $\mu_c$ (note that this is a vector) and var-covar $\Sigma$ ( the same for all the classes), so for each class we know the multivariate density $\Phi_c(\mu_c,\Sigma)$ that allows us to calculate the probabilities.
Now, given the features, we can compute the $\Phi_c$ for the featurs $x_i$ and we will put the subject in that class $c$ where this yields the highest value (i.e. Where the 'probability' is heighest).
LDA assumes multivariate normality in each class with the same var-covar. Therefore the classes are all the 'same' except for their mean. So you can 'feel' that the number of means will be important.
If you have $n$ features, then, in the end, the solution will depend on $C$ means, $C$ being the number of classes. In fact it can be show mathematically that LDA 'solves' the classification problem in a 'subspace' of the $n$-dimensional feature space, and that this subspace has dimension that is lower than $C$.
To make it more concrete, assume that you have subjects with $25$ features and you want to classify them in two classes, then you can 'solve' the problem in a one-dimesnional space (thus on a line). This is why LDA is said to be a dimension reduction technique, in this case it reduces the dimension from $25$ to one .
We start with n dimensions and end with k dimensions, where k < n. and is not restricted to only 2 output value. Is that correct?
$\endgroup$
This site has a pretty good answer: https://medium.com/@QuarizmiAdTech/a-full-introduction-to-the-linear-fisher-discriminant-analysis-848530dce336
However, I think that understanding Principal Component Analysis before diving into LDA is a good idea
Imagine trying to classify images of animals - lets say Cats and Dogs only. What features in the images are useful to identify the fruit? (Technically, identify = classify)
Imagine that the information dataset for above case has 20 attributes. Do you need all 20? Only 4?
You would use LDA to reduce these attributes. It finds if 2 variables are correlated. And if they are, then you can remove one of the correlated attributes. As a different example, if a dataset shows that height and weight are very strongly linked, then you can use either height weight as a classification attribute - you don't need both.
The weakness of LDA: The attributes have to be somehow comparable! For example, the LDA for identifying attributes to compare Apples vs Bananas would not work for Cats Vs Dogs. Mathematically, this implies that you there is a linear relationship of the type y = ax + b between x and y. (Add more for multiple attributes).
Obviously this will fail for quite a few cases - but the number of cases where LDA can work is surprisingly high - and it is also very useful in driving down computation costs: 5 attributes out of 20 may be enough to gather 98% confidence; and o get to 99.99% you might need 16. It is obvious that we would prefer to be able to work with 5 attributes, because the computation cost jump could be from 5$ to 70/80 $, not 5 to 10
We start with n dimensions and end with k dimensions, where k < n. and is not restricted to only 2 output value. Is that correct?
$\endgroup$
linear relationship in LDA?
$\endgroup$