8
$\begingroup$

The problem of designing a multi-class classifier using LDA can be expressed as a 2 class problem(one vs everything else) or a multi-class problem.

Why is it that in certain cases Multi-class LDA classifier out-performs 2 class LDA (one vs everything else) or vice-versa.

$\endgroup$
  • $\begingroup$ Could you add details to your quite reserved question. Examples. And do you consider here both stages of LDA - extraction and classification, - or classification only? $\endgroup$ – ttnphns Jun 8 '12 at 16:07
  • $\begingroup$ well, I am trying to project a 27 dimension vector to lower dimensions and comparing the vectors. The motive is to design a simple classification technique to classify as good as possible. $\endgroup$ – garak Jun 9 '12 at 13:43
  • 1
    $\begingroup$ Occasionally you might encounter data when a two-class classification is more accurate (for example, when one class is far apart, "outlier", from the rest ones, close to each other. But, as a rule, the k-class classification should turn out to be better. First, k classes allow for more discriminant axes. Second, a clump of k-1 classes is generally not expected to follow multivariate normal distribution on which the LDA classification stage relies on. See JohnSmith's answer. $\endgroup$ – ttnphns Jun 9 '12 at 14:55
12
$\begingroup$

I think that Multi-class LDA classifier always (well, in most practical tasks) out-performs 2 class LDA. And I will try to describe why.

Have a look at the example dataset: sample dataset with three classes

You have three classes here. And let's say you want to build one-vs-other classifier with LDA for the blue class.

The estimated mean for class "blue" is zero, but the estimated mean for class "other" is zero as well. And the covariance is the same from the definition of LDA. That means that LDA will respond with the label that has more elements. And it will never return class "blue" at all!

For Multi-class LDA it will manage to find the right classes perfectly.

The background about this is that the mixture of Gaussians is not a Gaussian any more in most cases. So this assumption of LDA fails. And I must say it is really difficult to come up with an example of a dataset where every class is Gaussian, and they still Gaussian after we join them.

That's why I would highly recommend to use Multi-class LDA. Hope it will help!

$\endgroup$
  • 1
    $\begingroup$ I think what John has shown very nicely here is that LDA for two class uses a single line to separate the classes. But in the example good classification requires two lines which is done in a three class problem. $\endgroup$ – Michael Chernick Jun 8 '12 at 20:09
  • $\begingroup$ @MichaelChernick, yes, exactly, that is an explanation from the other points of view, thanks for your comment! $\endgroup$ – Dmitry Laptev Jun 8 '12 at 23:37
  • $\begingroup$ thanks for the quick answer guys! However I have encountered a few days back, a case where a Multi-Class LDA (accuracy of 60%) is performing far less efficient than a 2 class LDA (accuracy of over 80%) in a classification problem of 10 classes. $\endgroup$ – garak Jun 9 '12 at 13:41
  • $\begingroup$ @MichaelChernick but if you use LDA as a projection method you could decide to keep two dimensions (the two eigen vectors of biggest eigen values) and get the separation you're looking for (using a kNN classification method instead of separating planes), or am I missing something obvious? $\endgroup$ – Matthieu Apr 28 '16 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.