Can anybody explain differences and give specific examples how to use these three analyses?

  • LDA - Linear Discriminant Analysis
  • FDA - Fisher's Discriminant Analysis
  • QDA - Quadratic Discriminant Analysis

I searched everywhere, but couldn't find real examples with real values to see how these analyses are used and data calculated, only lots of formulas which are hard to understand without any real examples. As I tried to understand it was hard to distinguish which equations/formulas belonged to LDA and which to FDA.

For example let's say there is such data:

x1 x2 class
1  2  a
1  3  a
2  3  a
3  3  a
1  0  b
2  1  b
2  2  b

And let's say some testing data:

x1 x2
2  4
3  5
3  6

So how to use such data with all these three approaches? It would be best to see how to calculate everything by hand, not using some math package which calculates everything behind the scenes.

P.S. I only found this tutorial: It shows how to use LDA.

up vote 21 down vote accepted

"Fisher's Discriminant Analysis" is simply LDA in a situation of 2 classes. When there is only 2 classes computations by hand are feasible and the analysis is directly related to Multiple Regression. LDA is the direct extension of Fisher's idea on situation of any number of classes and uses matrix algebra devices (such as eigendecomposition) to compute it. So, the term "Fisher's Discriminant Analysis" can be seen as obsolete today. "Linear Discriminant analysis" should be used instead. See also. Discriminant analysis with 2+ classes (multi-class) is canonical by its algorithm (extracts dicriminants as canonical variates); rare term "Canonical Discriminant Analysis" usually stands simply for (multiclass) LDA therefore (or for LDA + QDA, omnibusly).

Fisher used what was then called "Fisher classification functions" to classify objects after the discriminant function has been computed. Nowadays, a more general Bayes' approach is used within LDA procedure to classify objects.

To your request for explanations of LDA I may send you to these my answers: extraction in LDA, classification in LDA, LDA among related procedures. Also this, this, this questions and answers.

Just like ANOVA requires an assumption of equal variances, LDA requires an assumption of equal variance-covariance matrices (between the input variables) of the classes. This assumption is important for classification stage of the analysis. If the matrices substantially differ, observations will tend to be assigned to the class where variability is greater. To overcome the problem, QDA was invented. QDA is a modification of LDA which allows for the above heterogeneity of classes' covariance matrices.

If you have the heterogeneity (as detected for example by Box's M test) and you don't have QDA at hand, you may still use LDA in the regime of using individual covariance matrices (rather than the pooled matrix) of the discriminants at classification. This partly solves the problem, though less effectively than in QDA, because - as just pointed - these are the matrices between the discriminants and not between the original variables (which matrices differed).

Let me leave analyzing your example data for yourself.

  • I think I remember learning in my machine learning course that gaussian 2-class LDA assumed gaussian densities and uses Bayesian MPE rule whereas Fisher's LDA doesn't make the gaussian assumption and changes the optimality criterion to maximizing SNR. Is this congruent with your answer? – Austin Dec 18 '17 at 12:31
  • 1
    @Jake Interested about this as well, in this answer:…, it is stated that the result is the same. Comments? – Dole May 3 at 3:04

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.