2
$\begingroup$

In the training set for naive bayes, there are some duplicate samples. Should we train the naive bayes with duplicate samples, or should we eliminate all the duplicates and then train the naive bayes.

I have points for both for and against eliminating the duplicates.

For:
Since the duplicate sample does not add any new knowledge to the system, we should eliminate it. If there are large number of duplicate samples, the model we build will be biased towards the duplicated sample. Hence duplicates should be avoided.

Against:
If a sample is occurring multiple times, it is natural to have a bias to this sample. Hence duplicates should not be filtered.

Please suggest which way is the right way, if at all a right way exists.

$\endgroup$
5
+25
$\begingroup$

I think you've mostly answered the question. If the duplicates are part of the natural sampling process, and you're going to see them when you actually apply the classifier at test time, then you should include them. In this case they do add information, since they are telling the classifier about the distribution of the inputs that it is going to encounter.

If the duplicates are due to some reason that's not going to appear at test time (e.g. they come from overlapping data samples that were combined together) then you may have more serious issues in general with your training set, since it's not reflecting the true distribution of samples.

$\endgroup$
  • $\begingroup$ [Note: I'm not the original asker of this question, but I posted the bounty.] I agree with your reasoning for the most part, but the information added by including the duplicates seems to be of a different nature, to me. I'm wondering if there's a better way to incorporate it. Also (for the bounty) do you have any credible and/or official sources? Most things I've read start with the assumption that duplicates have been removed, but that's because they're almost always dealing with the second case you describe (i.e. duplicates = mistakes.) $\endgroup$ – Bill Clark Feb 25 '15 at 22:13
0
$\begingroup$

I would not agree that a "duplicate sample does not add any new knowledge to the system". Let me try to make my point with a linear regression, with which I am more familiar.

Try

reps <- 2

X1 <- rep(3,reps)
X2 <- rep(4,reps)
X3 <- rep(5,reps)
X <- c(X1,X2,X3)

y1 <- rep(6,reps)
y2 <- rep(7,reps)
y3 <- rep(9,reps)
y <- c(y1,y2,y3)

plot(X,y)
reg <- lm(y~X)
abline(reg)
summary(reg)

While the point estimates remain the same when you set reps <- 1 the standard errors won't and hence the precision of your predictions should suffer.

If you do not have duplicates everywhere as below, even point estimates will change:

reps <- 2

X1 <- 3
X2 <- rep(4,reps)
X3 <- rep(5,reps)
X <- c(X1,X2,X3)

y1 <- 6
y2 <- rep(7,reps)
y3 <- rep(9,reps)
y <- c(y1,y2,y3)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.