We know that according to Naive Bayes assumption input features are assumed to be independent of each others given the target variable $y$.
Now, If we intentionally add a duplicate (exact copy of one of our features) to the data set, what will happen for the classifier result? Will it still remain robust? Can we deduce that Naive Bayes is algorithm is not robust when the features are dependent to each other (contradicting Naive Bayes assumption)?
Let's start with an experiment. I am just duplicating the first column again and again in my data set.
data(HouseVotes84, package = "mlbench")
errors <- NULL
for(i in 1:50)
{
HouseVotes84[,ncol(HouseVotes84)+1] <- HouseVotes84$V1
model <- naiveBayes(Class ~ ., data = HouseVotes84[1:299,])
error <- sum(predict(model, HouseVotes84[300:400,])!=HouseVotes84[300:400,]$Class)
errors <- c(errors,error)
}
plot(errors,type='l',xlab='Number of duplications of V1',ylab='Error on the test set')
For information, the data set looks like:
Class V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16
1 republican n y n y y y n n n y <NA> y y y n y
2 republican n y n y y y n n n n n y y y n <NA>
3 democrat <NA> y y <NA> y y n n n n y n y y n n
4 democrat n y y n <NA> y n n n n y n y n n y
Indeed, the error rate increases as the first column gets duplicated. It seems to saturate at 32. Note that, keeping the first two columns only:
model <- naiveBayes(Class ~ ., data = HouseVotes84[1:299,1:2])
error <- sum(predict(model, HouseVotes84[300:400,])!=HouseVotes84[300:400,]$Class)
The error is 31.
What actually went on?
It all boils down to the construction of the Naive Bayes. Keeping Wikipedia's notations (https://en.wikipedia.org/wiki/Naive_Bayes_classifier):
$$p(C_k \vert x_1, \dots, x_n) = \frac{1}{Z} p(C_k) \prod_{i=1}^n p(x_i \vert C_k)$$
Where $C_k$ is the event "the target belongs to class $k$", and $x_i$ is the value of the $i$-th variable and $Z$ is a constant.
Classifying is just looking for the max of the above expression.
$$k = \arg \max_l p(C_l|x) $$
Looking at the logarithm and replicating $M$ times the first variable (calling $\tilde x_M$ the new vector created), we observe that:
$$\log(p(C_k|\tilde x_M))= \log(p(C_k|x)) + M \log(p(x_1 \vert C_k))$$
And we observe that clustering is done according to the first variable only, for $M$ large enough.
Another view of "robustness" is try to see if model is easy to get over fitting, i.e., works work on some (training) data set, but not working well on data have dramatic change.
Naive Bayes is a linear model and it is a generative model. Comparing to Neural Network, Support Vector Machine, it is less likely to over fit a Naive Bayes. In that sense, it is robust.
