We know that according to Naive Bayes assumption input features are assumed to be independent of each others given the target variable $y$.

Now, If we intentionally add a duplicate (exact copy of one of our features) to the data set, what will happen for the classifier result? Will it still remain robust? Can we deduce that Naive Bayes is algorithm is not robust when the features are dependent to each other (contradicting Naive Bayes assumption)?

  • $\begingroup$ Any idea please? $\endgroup$ Feb 12 '15 at 16:22
  • $\begingroup$ Though old, this is an interesting question. However, I don't think "robust" is the best way to describe your question. From wikipedia "One motivation is to produce statistical methods that are not unduly affected by outliers." As NB deals with classification, outliers are out of scope... $\endgroup$
    – RUser4512
    Sep 29 '15 at 14:54

Let's start with an experiment. I am just duplicating the first column again and again in my data set.

data(HouseVotes84, package = "mlbench")

errors <- NULL
for(i in 1:50)
  HouseVotes84[,ncol(HouseVotes84)+1] <- HouseVotes84$V1

  model <- naiveBayes(Class ~ ., data = HouseVotes84[1:299,])
  error <- sum(predict(model, HouseVotes84[300:400,])!=HouseVotes84[300:400,]$Class)

  errors <- c(errors,error)

plot(errors,type='l',xlab='Number of duplications of V1',ylab='Error on the test set')

Evolution of the error as the first column gets duplicated

For information, the data set looks like:

       Class   V1 V2 V3   V4   V5 V6 V7 V8 V9 V10  V11  V12 V13 V14 V15  V16
1 republican    n  y  n    y    y  y  n  n  n   y <NA>    y   y   y   n    y
2 republican    n  y  n    y    y  y  n  n  n   n    n    y   y   y   n <NA>
3   democrat <NA>  y  y <NA>    y  y  n  n  n   n    y    n   y   y   n    n
4   democrat    n  y  y    n <NA>  y  n  n  n   n    y    n   y   n   n    y

Indeed, the error rate increases as the first column gets duplicated. It seems to saturate at 32. Note that, keeping the first two columns only:

  model <- naiveBayes(Class ~ ., data = HouseVotes84[1:299,1:2])
  error <- sum(predict(model, HouseVotes84[300:400,])!=HouseVotes84[300:400,]$Class)

The error is 31.

What actually went on?

It all boils down to the construction of the Naive Bayes. Keeping Wikipedia's notations (https://en.wikipedia.org/wiki/Naive_Bayes_classifier):

$$p(C_k \vert x_1, \dots, x_n) = \frac{1}{Z} p(C_k) \prod_{i=1}^n p(x_i \vert C_k)$$

Where $C_k$ is the event "the target belongs to class $k$", and $x_i$ is the value of the $i$-th variable and $Z$ is a constant.

Classifying is just looking for the max of the above expression.

$$k = \arg \max_l p(C_l|x) $$

Looking at the logarithm and replicating $M$ times the first variable (calling $\tilde x_M$ the new vector created), we observe that:

$$\log(p(C_k|\tilde x_M))= \log(p(C_k|x)) + M \log(p(x_1 \vert C_k))$$

And we observe that clustering is done according to the first variable only, for $M$ large enough.

  • $\begingroup$ My graph shows 32 as the largest error, however, the model with a single variable has an error of 31... Should I assume it is "due to numerical reasons"? Or is there something inherently wrong in my arguments ? $\endgroup$
    – RUser4512
    Sep 29 '15 at 14:10

Another view of "robustness" is try to see if model is easy to get over fitting, i.e., works work on some (training) data set, but not working well on data have dramatic change.

Naive Bayes is a linear model and it is a generative model. Comparing to Neural Network, Support Vector Machine, it is less likely to over fit a Naive Bayes. In that sense, it is robust.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.