7
$\begingroup$

I am new to machine learning and have a problem.

I have a dataset with 1191 samples with 10 features which belong to 5 different classes. I have trained a neural network with this dataset and obtained a good accuracy of about 0.9. I noticed that about 350 samples are duplicated. All random selections of data for testing the network, contain about 180 samples which also have been in training data.

My question is should I remove the duplicates from the dataset? Do they contribute to this accuracy?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ Hi, welcome. Yes, these duplicates add weight to the fit of those specific observations (cases). Wether this effect is big or small is hard to tell from the information you have provided. $\endgroup$
    – Jim
    Commented Sep 22, 2019 at 9:14

2 Answers 2

Reset to default
10
$\begingroup$

You should probably remove them. Duplicates are an extreme case of nonrandom sampling, and they bias your fitted model. Including them will essentially lead to the model overfitting this subset of points.

I say probably because you should (1) be sure they are not real data that coincidentally have values that are identical, and (2) try to figure why you have duplicates in your data. For example, sometimes people intentionally 'oversample' rare categories in training data, though why this is done is not clear, as it probably helps only in rare cases.

As a side note, it's worth reading this thread: Why is accuracy not the best measure for assessing classification models?

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ +1. I answered a similar question a few months ago here: stats.stackexchange.com/questions/602110/… (I just flagged it as a duplicate of this one). Now, I'd be a bit wary of saying we should probably remove them, because it may be critical to determine if the identical observations are erroneous duplicates, or different observations whose values happen to coincide (using the term "probably" could encourage to not look hard into the issue). $\endgroup$
    – J-J-J
    Commented Jan 8 at 10:17
  • 1
    $\begingroup$ @J-J-J Fair point. If I wrote this answer today, I would probably emphasise the need for checking more. I will edit this later to do that. And +1 to your answer, I'd be happy for the duplicate to go the other way as your answer seems more comprehensive. $\endgroup$
    – mkt
    Commented Jan 8 at 13:20
  • 1
    $\begingroup$ ah, too late, the other question has been closed! Anyway, no big deal I guess; I realized that's a FAQ, as searching for "removing duplicates" returns a few similar results (stats.stackexchange.com/search?q=removing+duplicates). Other relevant answers are probably scattered all around the website. $\endgroup$
    – J-J-J
    Commented Jan 8 at 16:02
  • 1
    $\begingroup$ @J-J-J Duplicate direction reversed now. But we can see if there's a canonical answer about this out there and direct all threads towards it. $\endgroup$
    – mkt
    Commented Feb 7 at 15:40
  • $\begingroup$ I tried to provide a relatively comprehensive answer, however it might be that another thread could be more adequate as a canonical answer. I prefer to let other people decide that. I also suspect that a few of these questions might have more to do with the "unbalanced dataset problem", and there are excellent canonical answers to this kind of question. Another situation that my answer does not cover is the case of bootstrapping (I vaguely remember reading a couple of comments about bootstrapping in a discussion about removing duplicates, but I can't find it again). $\endgroup$
    – J-J-J
    Commented Feb 8 at 7:01
1
$\begingroup$

You should also make sure the samples in the test data are not the same as the samples in the training data, as that is then overfitting the model in that it will think it is predicting the data in the test set perfectly when it might just be due to the model having 'seen' that data in the training already.

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ Because this implies an unusual meaning of "overfitting" (and doesn't seem to be necessarily true), could you please explain your remark? $\endgroup$
    – whuber ♦
    Commented Jan 5 at 16:13
  • 1
    $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Jan 5 at 17:27
  • $\begingroup$ Consider a very simple model where you'd have just one independent binary variable (gender: woman/man), and one dependent binary variable (exam results: passed/failed). With regard to your answer, can you clarify how you would identify whether an observation is the same as another one? Or, in other words, what is your definition of "the same"? $\endgroup$
    – J-J-J
    Commented Jan 8 at 17:43
  • $\begingroup$ Ah I see, I was thinking more in terms of a neural network model on text data but for numeric data this wouldn't apply I guess. In the case of text classification sometimes it can be good to have duplicates to give more weight to certain classes (e.g. under-represented classes) so it depends on the data and the use case. $\endgroup$
    – userK
    Commented Jan 10 at 16:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.