5
$\begingroup$

The naive Bayes classifier assumes the regressors to be mutually independent, while linear discriminant analysis (LDA) allows them to be correlated. James et al. "An Introduction to Statistical Learning" (2nd edition, 2021) section 4.5 (bottom of p. 159) claim that LDA is in fact a special case of the naive Bayes classifier (admitting that the fact is not at all obvious -- with which I agree, and hence my question). What is the intuition?

$\endgroup$
6
+50
$\begingroup$

Here's my intuition:

The LDA classifier assumes that across all classes, the $p$ predictors $\boldsymbol{X}_k$ (for $k=1, \dots,p$) all share some covariance matrix ${\boldsymbol \Sigma}$, but may have different means $\boldsymbol{\mu}_k$. Thus, if you define the alternate set of predictors $\boldsymbol{Z}$ to be $p$ independent normal random variables with variance 1 and means $\boldsymbol{\Sigma}^{-1/2} \boldsymbol{\mu}$, then $\boldsymbol{X} = \boldsymbol{\Sigma}^{1/2} \boldsymbol{Z}$. This is a linear transformation, so a linear classifier on the $\boldsymbol{Z}$ variables would be linear on the $\boldsymbol{X}$ variables, too.

Note that the naive Bayes classifier is linear on its predictors (this is shown on page 159 of your reference), and it clearly applies on the $\boldsymbol{Z}$ predictors since they are independent by definition. So LDA is the same as some naive Bayes classifier. But as mentioned in your reference (also page 159), the same is true of any linear classifier.

$\endgroup$
2
  • $\begingroup$ Interesting. This explains in which sense LDA is a special case of naive Bayes (nB). For any given problem (a given set of $X$s) that satisfies the assumptions of LDA, LDA is not a special case of nB unless the $X$s happen to be uncorrelated. However, every such problem can be reformulated so that nB becomes applicable, at the expense of making LDA no longer applicable due to assumption violations that are introduced. Did I get that right? $\endgroup$ Sep 28 at 5:11
  • $\begingroup$ That's not quite how I see it. My perspective is that there is a one-to-one correspondence between the two. So it's not that LDA is invalid on the transformed version - you could do LDA there, too, just as well. Instead, the linear transformation just gives you two distinct ways of looking at the exact same classifier. From one perspective it is called LDA, from the other is is called naive Bayes. But it is the exact same classifier. I have edited my answer to clarify this point. $\endgroup$
    – Wesley
    Sep 28 at 16:17
2
$\begingroup$

LDA is a special case of a n̶a̶i̶v̶e̶ Bayes classifier.

  • It is assuming Gaussian distributions
  • For different classes, the distributions have the same variance (the same covariance matrix for their distribution with respect to the variables $X$).

Gaussian Naive Bayes classifier is a special case of LDA

If you consider the naive Bayes classifier with the assumption of Gaussian distributions and the same variance for different groups/classes, then you could see this as a special case of LDA. It is like LDA with the restriction that the covariance matrix $\Sigma$ is diagonal.

Other point of view

You might also see the LDA as a pre-treatment step giving you as result one or more components. Then afterward you apply naive Bayes on the components. So LDA can be seen as a special case of naive Bayes in the sense that it is naive Bayes with a pretreatment extracting first LDA components.

$\endgroup$
3
  • 1
    $\begingroup$ I think I do have a basic understanding of LDA and Bayes classifier. I am wondering about a specific relationship between them, one that seems counterintuitive. Does your answer address that? Your first paragraph seems to overlook the property of naive Bayes that the regressors are assumed to be independent. $\endgroup$ Sep 28 at 5:05
  • $\begingroup$ @RichardHardy you are right. LDA is a Bayes classifier, but not a naive Bayes classifier. The naive Bayes classifier is like LDA with the restriction that the covariance matrix $\Sigma$ is a diagonal matrix. On the other hand, LDA is like naive Bayes classifier with restrictions that the distribution is assumed to be Gaussian and similar in all classes. So neither is a special case of the other. $\endgroup$ Sep 28 at 8:03
  • 1
    $\begingroup$ Right... Which brings us back to my question: why do they state otherwise in the new edition of ISL? I like the answer by Wesley which has been deleted unexpectedly. My comment under it shows how much I understand. If you can see the deleted answer and my comment, do you agree with them? $\endgroup$ Sep 28 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.