1
$\begingroup$

Since some weeks, I pursue the question "Under which circumstances will LDA achieve a higher classification accuracy than QDA using the same training and test set as well as the same prior probabilities?".

I have found these links to this issue: link1 link2 link3 link4 link5

Unfortunately, the answers do not satisfy me completely.

Therefore, I would like to ask this question again in the hope of comprehensive responses consisting of descriptive examples.

$\endgroup$
5
  • $\begingroup$ Have you considered a case where the data generating process corresponds to LDA? The true model should beat any competitors in large samples and it should beat more complex competitors (such as QDA) in small samples, too. $\endgroup$ – Richard Hardy Oct 6 '19 at 19:14
  • $\begingroup$ I have, and the QDA accuracy was always higher than the LDA accuracy. $\endgroup$ – Daniel Oct 6 '19 at 19:33
  • $\begingroup$ That is weird... $\endgroup$ – Richard Hardy Oct 7 '19 at 6:14
  • $\begingroup$ @RichardHardy do you maybe have any other idea? $\endgroup$ – Daniel Nov 16 '19 at 21:07
  • $\begingroup$ Thank you for your answer! I will check your answer and notify as soon as I have finished writing my paper. $\endgroup$ – Daniel Dec 23 '19 at 13:35
1
$\begingroup$

Consider a case where the data generating process (DGP) corresponds to LDA or can be closely approximated by it. LDA should beat any competitors in large samples and it should beat more complex competitors (such as QDA) in small samples, too. The added flexibility of QDA would result in higher estimation variance due to the need for estimating some additional parameters. In relatively small samples, this cost will outweigh the lower model bias due to QDA offering a better approximation to the DGP than LDA. James et al. "An Introduction to Statistical Learning" (2013) p. 152-153 offer three examples where LDA beats QDA:

Scenario 1: There were 20 training observations in each of two classes. The observations within each class were uncorrelated random normal variables with a different mean in each class. The left-hand panel of Figure 4.10 shows that LDA performed well in this setting, as one would expect since this is the model assumed by LDA. KNN performed poorly because it paid a price in terms of variance that was not offset by a reduction in bias. QDA also performed worse than LDA, since it fit a more flexible classifier than necessary. Since logistic regression assumes a linear decision boundary, its results were only slightly inferior to those of LDA.

Scenario 2: Details are as in Scenario 1, except that within each class, the two predictors had a correlation of $−0.5$. The center panel of Figure 4.10 indicates little change in the relative performances of the methods as compared to the previous scenario.

Scenario 3: We generated $X_1$ and $X_2$ from the $t$-distribution, with 50 observations per class. The $t$-distribution has a similar shape to the normal distribution, but it has a tendency to yield more extreme points — that is, more points that are far from the mean. In this setting, the decision boundary was still linear, and so fit into the logistic regression framework. The set-up violated the assumptions of LDA, since the observations were not drawn from a normal distribution. The right-hand panel of Figure 4.10 shows that logistic regression outperformed LDA, though both methods were superior to the other approaches. In particular, the QDA results deteriorated considerably as a consequence of non-normality.

(Emphasis is mine.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.