I know that Bayes classifier assigns the new data point $\pmb{x}$ to the class $\omega_j, \ j=1,\dots,M$, when

$p(\omega_j \mid \pmb{x}) = \max_{q=1,\dots,M}p(\omega_q \mid \pmb{x})$,


$p(\omega_j\mid \pmb{x}) = \frac{p(\pmb{x}\mid \omega_j)p(\omega_j)}{p(\pmb{x})} = \frac{p(\pmb{x}\mid \omega_j)p(\omega_j)}{\sum_j p(\pmb{x} \mid \omega_j)p(\omega_j)}$.

The difference from the Naive Bayes classifier is that Naive Bayes assumes statistical independent features,

$p(\pmb{x}|\omega_j) = \prod_{k=1}^{l}p_k(x_k|\omega_j), \ \ \ \ \ j=1,\dots,M$

where $l$ is the number of features.

Why there are only Naive Bayes classifier implementations and there are not the full Bayes ones?


1 Answer 1


Specifically, Naive Bayes assumes conditional independence of features (conditioned on classes), not statistical independence. And, informally speaking, despite strong assumptions, it works pretty ok in practice. An obvious reason of using naive assumption is that the number of parameters needed in the model is linear with increasing number of features. Imagine that $p(\mathbf{x}|w_i)$ is Gaussian and you need the covariance matrix to characterize the Bayes classifier, and $\mathbf{x}$ is high dimensional, like $R^{10000}$, like in spam classification problem, e.g. the size of the vocabulary set. Then, the covariance matrix would have $10^8$ entries to be learnt. This isn't scalable. Or using non-parametric methods (like KDE) for joint PDF estimation would need immense amount of data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.