2
$\begingroup$

I am looking for a toy example to understand this behavior. Preferebly a text classification one

I read the following from http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf at page 7.

Furthermore, even when naive Bayes has good classification accuracy, its probability estimates tend to be poor. To understand why, imagine training naive Bayes on a data set in which all the features are repeated, that is, x = (x1,x1,x2,x2,...,xK,xK). This will increase the confidence of the naive Bayes probability estimates, even though no new information has been added to the data.

I am able to produce any toy example for which $$p(y) * \sum_{i=1}^k( p(x_i|y))$$ changes by duplicating the features.

$\endgroup$
  • $\begingroup$ Please write more explicit titles. $\endgroup$ – Franck Dernoncourt Dec 26 '16 at 3:33
2
$\begingroup$

I'm not sure what kind of example you are looking for. But to understand this behavior you simply need to consider this: For duplicated variables you have $$p(x_1=k, x_2=k|y)=p(x_1=k|y)=p(x_2=k|y)$$ Yet naive bayes models this as $$p_{NaiveBayes}(x_1=k,x_2=k|y)=p(x_1=k|y)p(x_2=k|y)=p(x_1=k,x_2=k|y)^2$$

$\endgroup$
  • $\begingroup$ "This will increase the confidence of the naive Bayes probability estimates" I am looking for an example which increases probability estimated by naïve Bayes. $\endgroup$ – Kumaran Dec 9 '16 at 21:17
  • $\begingroup$ the (normalized) squared probability will be closer to 0 or 1 always. That's exactly what increases the confidence. Say you have a true probability of $p(C | x_1, x_2) = 0.9$ and a uniform prior. The naive bayes probability will be around 0.988. $\endgroup$ – DaVinci Dec 9 '16 at 21:57
0
$\begingroup$

If you are looking for proof, here is one.

PROOF

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.