Intuition for why LDA is a special case of naive Bayes

Question

The naive Bayes classifier assumes the regressors to be mutually independent, while linear discriminant analysis (LDA) allows them to be correlated. James et al. "An Introduction to Statistical Learning" (2nd edition, 2021) section 4.5 (bottom of p. 159) claim that LDA is in fact a special case of the naive Bayes classifier (admitting that the fact is not at all obvious -- with which I agree, and hence my question). What is the intuition?

Answer 1

Here's my intuition:

The LDA classifier assumes that across all classes, the $p$ predictors $\boldsymbol{X}_k$ (for $k=1, \dots,p$) all share some covariance matrix ${\boldsymbol \Sigma}$, but may have different means $\boldsymbol{\mu}_k$. Thus, if you define the alternate set of predictors $\boldsymbol{Z}$ to be $p$ independent normal random variables with variance 1 and means $\boldsymbol{\Sigma}^{-1/2} \boldsymbol{\mu}$, then $\boldsymbol{X} = \boldsymbol{\Sigma}^{1/2} \boldsymbol{Z}$. This is a linear transformation, so a linear classifier on the $\boldsymbol{Z}$ variables would be linear on the $\boldsymbol{X}$ variables, too.

Note that the naive Bayes classifier is linear on its predictors (this is shown on page 159 of your reference), and it clearly applies on the $\boldsymbol{Z}$ predictors since they are independent by definition. So LDA is the same as some naive Bayes classifier. But as mentioned in your reference (also page 159), the same is true of any linear classifier.

Answer 2

LDA is a special case of a n̶a̶i̶v̶e̶ Bayes classifier.

• It is assuming Gaussian distributions
• For different classes, the distributions have the same variance (the same covariance matrix for their distribution with respect to the variables $X$).

Gaussian Naive Bayes classifier is a special case of LDA

If you consider the naive Bayes classifier with the assumption of Gaussian distributions and the same variance for different groups/classes, then you could see this as a special case of LDA. It is like LDA with the restriction that the covariance matrix $\Sigma$ is diagonal.

Other point of view

You might also see the LDA as a pre-treatment step giving you as result one or more components. Then afterward you apply naive Bayes on the components. So LDA can be seen as a special case of naive Bayes in the sense that it is naive Bayes with a pretreatment extracting first LDA components.