1
$\begingroup$

It is told that

the most important disadvantage of Naive Bayes is that it has strong feature independence assumptions.

Can someone please explain this more elaborately?

$\endgroup$
1
  • $\begingroup$ The Naive Bayes classifier assumes that all variables are conditionally independent given the outcome. This assumption rarely holds in practice. $\endgroup$
    – George
    Nov 23, 2015 at 1:17

1 Answer 1

3
$\begingroup$

$\newcommand{\vect}[1]{\mathbf{#1}}$

The Naive Bayesian classifier explores the idea of maximizing posterior probability that given tuple $\vect{X} = (x_1, x_2, \dots, x_N)$ belongs to the class $C_{i}$, i.e. maximizing $P(C_{i}|\vect{X})$.

By Bayes' theorem $$ P(C_{i}|\vect{X}) = \frac{P(\vect{X}|C_{i})P(C_{i})}{P(\vect{X})} $$

$P(\vect{X})$ is constant for the classes, and if we don't have any prior for $P(C_{i})$, we assume $P(C_{i}) = P(C_{j})$.

So we have only $P(\vect{X}|C_{i})$ to compute for all the training data. And this is there the "Naive" assumption is made: we assume that there is no dependence relationships between attributes.

That means $$ P(\vect{X}|C_{i}) = \prod_{k=1}^{N} P(x_k|C_{i}) $$

So now we can estimate the probabilities $P(x_1|C_{i})$, $\dots$, which is very easy compared to estimating $P(\vect{X}|C_{i})$.

The price paid for this easiness is the class-condition independence assumption made above, which is not always a true.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.