26
$\begingroup$

Can anybody explain differences and give specific examples how to use these three analyses?

  • LDA - Linear Discriminant Analysis
  • FDA - Fisher's Discriminant Analysis
  • QDA - Quadratic Discriminant Analysis

I searched everywhere, but couldn't find real examples with real values to see how these analyses are used and data calculated, only lots of formulas which are hard to understand without any real examples. As I tried to understand it was hard to distinguish which equations/formulas belonged to LDA and which to FDA.

For example let's say there is such data:

x1 x2 class
1  2  a
1  3  a
2  3  a
3  3  a
1  0  b
2  1  b
2  2  b

And let's say some testing data:

x1 x2
2  4
3  5
3  6

So how to use such data with all these three approaches? It would be best to see how to calculate everything by hand, not using some math package which calculates everything behind the scenes.

P.S. I only found this tutorial: http://people.revoledu.com/kardi/tutorial/LDA/LDA.html#LDA. It shows how to use LDA.

$\endgroup$
23
$\begingroup$

"Fisher's Discriminant Analysis" is simply LDA in a situation of 2 classes. When there is only 2 classes computations by hand are feasible and the analysis is directly related to Multiple Regression. LDA is the direct extension of Fisher's idea on situation of any number of classes and uses matrix algebra devices (such as eigendecomposition) to compute it. So, the term "Fisher's Discriminant Analysis" can be seen as obsolete today. "Linear Discriminant analysis" should be used instead. See also. Discriminant analysis with 2+ classes (multi-class) is canonical by its algorithm (extracts dicriminants as canonical variates); rare term "Canonical Discriminant Analysis" usually stands simply for (multiclass) LDA therefore (or for LDA + QDA, omnibusly).

Fisher used what was then called "Fisher classification functions" to classify objects after the discriminant function has been computed. Nowadays, a more general Bayes' approach is used within LDA procedure to classify objects.

To your request for explanations of LDA I may send you to these my answers: extraction in LDA, classification in LDA, LDA among related procedures. Also this, this, this questions and answers.

Just like ANOVA requires an assumption of equal variances, LDA requires an assumption of equal variance-covariance matrices (between the input variables) of the classes. This assumption is important for classification stage of the analysis. If the matrices substantially differ, observations will tend to be assigned to the class where variability is greater. To overcome the problem, QDA was invented. QDA is a modification of LDA which allows for the above heterogeneity of classes' covariance matrices.

If you have the heterogeneity (as detected for example by Box's M test) and you don't have QDA at hand, you may still use LDA in the regime of using individual covariance matrices (rather than the pooled matrix) of the discriminants at classification. This partly solves the problem, though less effectively than in QDA, because - as just pointed - these are the matrices between the discriminants and not between the original variables (which matrices differed).

Let me leave analyzing your example data for yourself.


Reply to @zyxue's answer and comments

LDA is what you defined FDA is in your answer. LDA first extracts linear constructs (called discriminants) that maximize the between to within separation, and then uses those to perform (gaussian) classification. If (as you say) LDA were not tied with the task to extract the discriminants LDA would appear to be just a gaussian classifier, no name "LDA" would be needed at all.

It is that classification stage where LDA assumes both normality and variance-covariance homogeneity of classes. The extraction or "dimensionality reduction" stage of LDA assumes linearity and variance-covariance homogeneity, the two assumptions together make "linear separability" feasible. (We use single pooled $S_w$ matrix to produce discriminants which therefore have identity pooled within-class covariance matrix, that give us the right to apply the same set of discriminants to classify to all the classes. If all $S_w$s are same the said within-class covariances are all same, identity; that right to use them becomes absolute.)

Gaussian classifier (the second stage of LDA) uses Bayes rule to assign observations to classes by the discriminants. The same result can be accomplished via so called Fisher linear classification functions which utilizes original features directly. However, Bayes' approach based on discriminants is a little bit general in that it will allow to use separate class discriminant covariance matrices too, in addition to the default way to use one, the pooled one. Also, it will allow to base classification on a subset of discriminants.

When there are only two classes, both stages of LDA can be described together in a single pass because "latents extraction" and "observations classification" reduce then to the same task.

$\endgroup$
  • $\begingroup$ I think I remember learning in my machine learning course that gaussian 2-class LDA assumed gaussian densities and uses Bayesian MPE rule whereas Fisher's LDA doesn't make the gaussian assumption and changes the optimality criterion to maximizing SNR. Is this congruent with your answer? $\endgroup$ – Austin Dec 18 '17 at 12:31
  • 1
    $\begingroup$ @Jake Interested about this as well, in this answer: stats.stackexchange.com/questions/87975/…, it is stated that the result is the same. Comments? $\endgroup$ – Dole May 3 '18 at 3:04
  • $\begingroup$ Are you sure ""Fisher's Discriminant Analysis" is simply LDA in a situation of 2 classes"? $\endgroup$ – zyxue Jan 8 at 18:34
  • $\begingroup$ @zyxue, 95% sure, but anyway I find the term obsolete. Please see my footnote in stats.stackexchange.com/q/190806/3277. $\endgroup$ – ttnphns Jan 9 at 6:46
  • $\begingroup$ @ttnphns,fyi, according to youtu.be/hGKt0yy9q_E?t=3010, FDA and LDA are often mixed up in the literature. One way to distinguish the two, FDA is a feature extraction method while LDA and QDA are classification technique. $\endgroup$ – zyxue Jan 24 at 23:05
1
$\begingroup$

I find it hard to agree that FDA is LDA for two-classes as @ttnphns suggested.

I recommend two very informative and beautiful lectures on this topic by Professor Ali Ghodsi:

  1. LDA & QDA. In addition, page 108 of the book The Elements of Statistical Learning (pdf) has a description of LDA consistent with the lecture.
  2. FDA

To me, LDA and QDA are similar as they are both classification techniques with Gaussian assumptions. A major difference between the two is that LDA assumes the feature covariance matrices of both classes are the same, which results in a linear decision boundary. In contrast, QDA is less strict and allows different feature covariance matrices for different classes, which leads to a quadratic decision boundary. See the following figure from scikit-learn for an idea how the quadratic decision boundary looks.

Some comments on the subplots:

  • Top row: when the covariance matrices are indeed the same in the data, LDA and QDA lead to the same decision boundaries.
  • Bottom row: when the covariance matrices are different, LDA leads to bad performance as its assumption becomes invalid, while QDA performs classification much better.

On the other hand, FDA is a very different species, having nothing to do with Gaussion assumption. What FDA tries to do is to find a linear transformation to maximize between-class mean distance while minimizing within-class variance. The 2nd lecture explains this idea beautifully. In contrast to LDA/QDA, FDA doesn't do classification, although the features obtained after transformation found by FDA could be used for classification, e.g. using LDA/QDA, or SVM or others.

$\endgroup$
  • 2
    $\begingroup$ Please see my reply to your answer in my answer. I haven't watched the video lesson you link to, so can't say if I can agree with it. I don't agree with the interpretation/definition (of LDA vs FDA) you are giving in the answer. But that - definition of the two terms - is not very important topic to me. It is more important to understand how it all works. $\endgroup$ – ttnphns Jan 25 at 9:14
  • 2
    $\begingroup$ If in your understanding FDA doesn't do classification, although the features obtained after transformation found by FDA could be used for classification then I would say it is what I call "extraction phase of LDA". Of course, these extracted features (the discriminant functions) - you may use them as you like. In standard LDA classification, they are used as gaussian classifiers. $\endgroup$ – ttnphns Jan 25 at 9:30
  • $\begingroup$ I am curious where did you read that "LDA first extracts linear constructs (called discriminants)"? I thought it's called linear discriminant because the decision boundary is linear, which is a result of assuming that the feature covariance matrix is the same for different classes. Similarly, QDA has a quadratic decision boundary. These are also shown in the embedded figures. Besides the above videos, I am referencing The Elements of Statistical Learning (pdf. At page 108, it starts describing LDA with no mention of feature extraction. $\endgroup$ – zyxue Jan 25 at 16:45
  • $\begingroup$ Afterall, I think we are talking about the same thing, but it's just a way of naming things. You think LDA = feature_extraction + classification, but based on my references, LDA is just about classification. The feature_extraction part here is called FDA. Importantly, there is nothing to do here with the number of classes involved here. Both of LDA and FDA can deal with more than two classes. $\endgroup$ – zyxue Jan 25 at 16:47
  • $\begingroup$ We may stick to the definitions we are used too. Anyway, one should note that "linear constructs" and "linear decision boundaries" are related, in the context of DA they are about the same thing. Look at your picture with the decision border as straight line. When you transform the too classes, ellipses to be spherical the discriminant function will be precisely perpendicular to the border. Actually, the "primary" thing here is the discriminant function, a variable, a dimension, while the decision boundary is the border in space dependent on the direction of it. Dec. boundary is "secondary". $\endgroup$ – ttnphns Jan 25 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.