# Why does training naive Bayes on a data set in which all the features are repeated increase the conﬁdence of the naive Bayes probability estimates?

I am looking for a toy example to understand this behavior. Preferebly a text classification one

I read the following from http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf at page 7.

Furthermore, even when naive Bayes has good classiﬁcation accuracy, its probability estimates tend to be poor. To understand why, imagine training naive Bayes on a data set in which all the features are repeated, that is, x = (x1,x1,x2,x2,...,xK,xK). This will increase the conﬁdence of the naive Bayes probability estimates, even though no new information has been added to the data.

I am able to produce any toy example for which $$p(y) * \sum_{i=1}^k( p(x_i|y))$$ changes by duplicating the features.

• Please write more explicit titles. – Franck Dernoncourt Dec 26 '16 at 3:33

I'm not sure what kind of example you are looking for. But to understand this behavior you simply need to consider this: For duplicated variables you have $$p(x_1=k, x_2=k|y)=p(x_1=k|y)=p(x_2=k|y)$$ Yet naive bayes models this as $$p_{NaiveBayes}(x_1=k,x_2=k|y)=p(x_1=k|y)p(x_2=k|y)=p(x_1=k,x_2=k|y)^2$$
• the (normalized) squared probability will be closer to 0 or 1 always. That's exactly what increases the confidence. Say you have a true probability of $p(C | x_1, x_2) = 0.9$ and a uniform prior. The naive bayes probability will be around 0.988. – DaVinci Dec 9 '16 at 21:57