Murphy writes that a multivariate Gaussian used in a generative classifier ("Gaussian discriminant analysis"), i.e., $p(\mathbf x|y=c,\mathbf\theta) = \mathcal{N}(\mathbf x|\mathbf y_c,\mathbf \Sigma_c)$, is equivalent to Gaussian Naive Bayes if $\mathbf \Sigma_c$ is diagonal. This makes sense to me since GNB assumes independent features, each following a normal distribution.

However, I've also read in multiple papers, for example this one, about an alleged equivalence of Naive Bayes and a mixture of Gaussians with diagonal covariance matrices. How can this be the case when a mixture can approximate any distribution given enough Gaussian base components? It seems to me that the GMM is more powerful than both Naive Bayes and Gaussian discriminant analysis in that it can provide better likelihoods.

  • 1
    $\begingroup$ I haven't read the papers you refer to, but I would guess they mean that the mixture in question constitutes the different classes. Ie, if you had 2 classes, you would have a mixture of 2 (possibly multivariate) Gaussians. So although a sufficient number of Gaussians can approximate any distribution, that isn't what they mean. $\endgroup$ – gung - Reinstate Monica Jun 28 '14 at 18:16
  • $\begingroup$ What @gung said is how I first learned Naive Bayes. They definitely mean a mixture of two Gaussians, one for each class. $\endgroup$ – shadowtalker Jun 29 '14 at 6:00
  • $\begingroup$ I've only seen mixtures of Gaussians being used to model a single class, multiple mixtures for multiple classes. One Gaussian per class didn't constitute a "mixture" for me. Thanks a lot for clearing that up! $\endgroup$ – fschulze Jun 29 '14 at 7:05
  • $\begingroup$ @ssdecontrol, it sounds like you are pretty familiar w/ these issues. Why don't you make that an official answer? $\endgroup$ – gung - Reinstate Monica Jun 29 '14 at 15:21
  • $\begingroup$ Because I was reading the paper he linked to and now I'm not so sure. @fschulze I think I could be wrong. They might mean that the entire $p(c|x_1,\dots,x_I)$ is a mixture of Gaussians. $\endgroup$ – shadowtalker Jun 29 '14 at 21:06

In the second paper you linked to:

Naive Bayes represents a distribution as a mixture of components, where within each component all variables are assumed independent of each other. Given enough components, it can approximate an arbitrary distribution arbitrarily closely. ... When learned from data, naive Bayes models never contain more components than there are examples in the data ...

and then later, the quote you mentioned,

Naive Bayes models can be viewed as Bayesian networks in which each $X_i$ has $C$ as the sole parent and $C$ has no parents. A naive Bayes model with Gaussian $P(X_i |C )$ is equivalent to a mixture of Gaussians with diagonal covariance matrices (Dempster et al., 1977).

Then they go on to show that, in the discrete case, $P(X=x)=\sum_{c=1}^k\left(P(c)\prod_{i=1}^{|X|}P(x_i|c)\right)$.

So consider $X\sim N(\mu,\Sigma^\text{diag})$. It's easy to see that: $$ P(X=x)=\sum_{c=1}^kP(c)\prod_{i=1}^{|X|}P(x_i|c)=\sum_{c=1}^kP(c)\cdot P(x_1,\dots,x_{|X|}|c) $$ since the joint distribution of $X$ is just the product of the distributions of its independent components. This, of course, is a mixture of $k$ Gaussians, each with weight $P(c)$. So unless they're talking about something else, I think I was right in the comments. What makes me less than 100% confident is that a) they talk about "enough components" which makes me think it should be more than $k$, and b) I have no idea where in Dempster (1977) this is or why they cite that paper of all things. I tried sifting through Dempster but it's on a completely different subject (E-M algorithm for ML with missing data) and I don't think they even had the term "Naive Bayes" back then. APA really needs to start requiring page numbers somehow.

I think the reason they point this out is that they develop Naive Bayes from a graphical / network perspective, so this provides a different intuition.

  • $\begingroup$ Thanks! I encountered the same problem when skimming Dempster's paper. $\endgroup$ – fschulze Jun 30 '14 at 16:08

(Related): Quoting Greg Heath from a Google Groups discussion,

In general, the two concepts (Gaussian Mixture Model and Naive Bayes Network) are not related.

The Gaussian Mixture Model approximates the probability density with a sum of Gaussians. Each gaussian is, in general, characterized by a full covariance matrix. The variables are independent if, and only if, the matrix is diagonal.

The Naive Bayes Assumption assumes that the variables are independent. Therefore, the probability density can be characterized by a product of univariate densities. If the univariate densities are Gaussian, the probability is characterized by a Gaussian Mixture Model with a diagonal covariance matrix.

Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.